# Better Compute

> An embeddable TypeScript control plane for GPU, CPU and sandbox capacity: nodes, reservations, bare-metal provisioning, and isolated VMs, authorized by Better IAM. Created by Sean Filimon.

# bcd commands (/docs/daemon/commands)

> Every bcd subcommand with its flags, what it does and its exit codes, plus the full list of bcd doctor checks with their statuses and fixes.



`bcd` is one binary with one subcommand per mode or local task. This page lists every subcommand the code
registers in `daemon/cmd/bcd/main.go`, its flags exactly as defined there, what it does, and how it exits. The
[overview](/docs/daemon#modes) explains what each mode is for, [installing bcd](/docs/daemon/install) covers the
install script that runs most of these for you, and [the config.toml reference](/docs/daemon/configuration) covers
the file most of them read. When something goes wrong, start with `bcd doctor` (below) and
[troubleshooting](/docs/daemon/troubleshooting).

> **What has run where.** 
  Every command is covered by Go unit tests, mostly against fakes (fake sysfs trees, a fake `nvidia-smi`, a fake
  `virsh`). `bcd join`, `bcd node`, `bcd helper`, `bcd sim` and `bcd update` run in `tests/e2e/bcd.test.ts` as a
  fakehw build against a real gateway. On real hardware, `bcd hardware`, `bcd doctor`'s GPU checks and an
  unprivileged `bcd node` have run on one RTX 5090 under WSL 2 (`tests/e2e/bcd-host-real.test.ts`; 0915ab7,
  f84f1f9, da60bbd). No release has been tagged, so no installed host has run any of them from a release binary.

## Usage and exit codes [#usage-and-exit-codes]

```text
bcd <command> [flags]
bcd <command> -h    flags of one command
bcd help            the command list (also -h, -help, --help)
bcd --version       same as bcd version (also -version)
```

Every command uses Go's `flag` package: flags are `-name value` or `--name=value`, booleans are `--name` or
`--name=false`. No command takes positional arguments. bcd never echoes an unexpected argument, so a token pasted
on the command line does not reach a log:

```bash
$ bcd join bcjt_jt_01...
bcd join: takes no positional arguments
bcd join reads the join token from BCD_JOIN_TOKEN, stdin or --token-file, never from arguments
```

An unknown command name is echoed only when it looks like one (lower-case letters and `-`, at most 32 characters);
anything else prints `bcd: unknown command` alone.

| Exit code | Meaning                                                                                                             |
| --------- | ------------------------------------------------------------------------------------------------------------------- |
| `0`       | Success, `-h`, or an agent mode stopped by SIGINT or SIGTERM                                                        |
| `1`       | Failure: a configuration that does not load, a refused join, a doctor check that failed, an agent that cannot start |
| `2`       | Usage error: an unknown command or flag, a positional argument, a missing token, a command this build does not have |

| Environment variable                                             | Read by                  | Meaning                                                                                                    |
| ---------------------------------------------------------------- | ------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `BCD_JOIN_TOKEN`                                                 | `bcd join`, `bcd sim`    | The join token; unset by bcd as soon as it is read. Redacted from support bundles                          |
| `NODE_NAME`                                                      | `bcd node` on Kubernetes | The Kubernetes Node bcd runs on, as a DaemonSet sets it (see [bcd on Kubernetes](/docs/daemon/kubernetes)) |
| `BCD_RENEW_AFTER`                                                | fakehw builds only       | A Go duration: renew this soon after start (end-to-end tests)                                              |
| `BCD_FAKEHW_SEED`, `BCD_FAKEHW_TPM_KEYS`, `BCD_FAKEHW_TPM_EK_CA` | fakehw builds only       | The fake machine's seed and simulated TPM                                                                  |

## The commands [#the-commands]

| Command                     | Runs as                         | Reads `config.toml`         | Long-running |
| --------------------------- | ------------------------------- | --------------------------- | ------------ |
| [`node`](#bcd-node)         | root, or the `[helper] user`    | yes                         | yes          |
| [`site`](#bcd-site)         | root (its own unit)             | yes                         | yes          |
| [`ramdisk`](#bcd-ramdisk)   | root, inside the PXE initramfs  | no (`bc.*` boot parameters) | yes          |
| [`helper`](#bcd-helper)     | root                            | yes                         | yes          |
| [`sim`](#bcd-sim)           | anyone; fakehw builds only      | yes                         | yes          |
| [`join`](#bcd-join)         | root                            | yes, or writes it           | no           |
| [`status`](#bcd-status)     | whoever owns the status socket  | no                          | no           |
| [`doctor`](#bcd-doctor)     | root for the full set of checks | yes                         | no           |
| [`hardware`](#bcd-hardware) | root for DMI serials            | no                          | no           |
| [`update`](#bcd-update)     | root on a helper node           | yes                         | no           |
| [`version`](#bcd-version)   | anyone                          | no                          | no           |

Two names exist that are not for operators: `__bcx-exec` is bcd re-executing itself as the
[plugin sandbox's shim](/docs/extending/drivers#the-plugin-sandbox), and `helper-call` exists only in builds with
the `helpere2e` tag, where the root helper end-to-end test sends one JSON request from standard input to
`[helper] socket`. A release build has neither in its command list.

## bcd node [#bcd-node]

Runs the node agent: the session to the gateway, the journal, inventory, NodeSpec convergence, commands, tunnels,
telemetry and self-update.

| Flag       | Default                | Meaning                |
| ---------- | ---------------------- | ---------------------- |
| `--config` | `/etc/bcd/config.toml` | The configuration file |

At start it loads the configuration, refuses `dev_insecure_h2c` in a host build, loads the identity from
`data_dir` (`identity.json` and `node.key`), opens the journal, starts the [bcx plugins](/docs/extending/drivers)
of `[plugins] dir`, checks `[kubernetes]`, prepares self-update, switches to the WebSocket fallback when
`[websocket] url` is set, serves `bcd status` on `socket_path`, and with `[imds] listen` serves the instance
metadata service. With `[helper] socket` it sends its privileged work to `bcd helper` and reads the root-only
inventory (DMI serials, DIMMs, NVMe health) through it; see [privilege
separation](/docs/daemon#privilege-separation). A host build running as an ordinary user without `[helper]`
starts anyway, but claims no GPU binding capability and fails every GPU hold at once as `node-unprivileged`
(da60bbd).

On an EC2 instance (a `*.metal` VPS host included), `169.254.169.254` and `fd00:ec2::254` belong to AWS's
instance metadata service, which answers for the host: its identity document and its instance-profile
credentials. `bcd node` then:

* refuses to start with `[imds] listen` on either address (`[imds] listen "169.254.169.254:80" is EC2's own
  instance metadata service on this EC2 host`). Bound there, bcd's own service would make the address local, so
  the host's `cloud` section, spot interruption notices and `cloud-iid` join would read bcd's service instead of
  AWS's. Serve the instances' metadata on another address, such as the host's address on the instances' bridge.
* drops every instance tap's packets to both addresses, whatever the instance's network policy, before any policy
  rule accepts them.

Also launch the instance with `HttpTokens=required` and an `HttpPutResponseHopLimit` of 1 (2 when bcd runs in a
container without host networking): IMDSv1 answers any packet the host forwards, and IMDSv2 answers a token
request only within the hop limit.

It logs JSON lines to standard error (`[log]`), and runs until SIGINT or SIGTERM. Exits:

* `0` on a signal, or after a [self-update](/docs/daemon/install#self-update) swapped the binary slots, in which
  case `main` execs the entrypoint (the new binary) with the same arguments.
* `1` when it cannot start (`bcd node: agent: this node has not joined (run bcd join)`, a journal of another node,
  an invalid `[kubernetes]` value, `[imds] listen` on AWS's metadata address on EC2) or when the certificate is revoked (`agent: this node's certificate was
  revoked`).

```bash
bcd node --config /etc/bcd/config.toml
```

## bcd site [#bcd-site]

Runs the [site agent](/docs/daemon/install#a-site-agent): the same agent with role `site` (BMC operations,
`site.httpProxy`, boot entries) and, with `[boot] listen` set, the boot server (proxy DHCP, TFTP, iPXE, the image
cache).

| Flag       | Default                | Meaning                |
| ---------- | ---------------------- | ---------------------- |
| `--config` | `/etc/bcd/config.toml` | The configuration file |

It refuses a data directory whose identity is not a site's ("this data directory holds a node identity; join with
a site token (bcd join --role site)", exit 1). With `[boot] listen`, `join-encryption.jwk` must sit next to the
configuration file (exit 1 otherwise), and `proxy_dhcp` or `tftp` need `[boot] server_ip` (IPv4) and `[boot] url`
(exit 2). Without `[boot] listen` it logs a warning and serves no PXE machine. `[helper]` is ignored.

```bash
BCD_JOIN_TOKEN=bcjt_... bcd join --role site --config /etc/bcd/config.toml
bcd site --config /etc/bcd/config.toml
```

## bcd ramdisk [#bcd-ramdisk]

Runs inside the PXE ramdisk a site agent boots. It has no `config.toml`: it reads the `bc.*` parameters from the
kernel command line (listed in [the ramdisk](/docs/daemon/install#the-ramdisk)), joins with the `discovery`
attestor as role `ramdisk` unless the data directory already holds an identity, then runs the agent.

| Flag         | Default         | Meaning                                                                         |
| ------------ | --------------- | ------------------------------------------------------------------------------- |
| `--cmdline`  | `/proc/cmdline` | Where to read the `bc.*` boot parameters                                        |
| `--data-dir` | `/run/bcd`      | Key, identity and journal, on tmpfs; the status socket is `<data-dir>/bcd.sock` |

A missing or malformed boot parameter, or an `http://` gateway in a host build, exits 2. A `bc.jwk` that does not
match its pin exits 1 ("bc.jwk does not match the join encryption pin"). Logs are always JSON. The OS a
`deploy.image` installs gets a `config.toml` rendered from the same parameters (with the default `data_dir`) and the
same `join-encryption.jwk`.

## bcd helper [#bcd-helper]

Runs as root beside an unprivileged `bcd node` and serves its typed privileged operations on `[helper] socket`.
See [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged) for the units and
[privilege separation](/docs/daemon#privilege-separation) for the operations.

| Flag       | Default                | Meaning                                                   |
| ---------- | ---------------------- | --------------------------------------------------------- |
| `--config` | `/etc/bcd/config.toml` | The configuration file; it must have a `[helper]` section |

Before it listens, it refuses to start (exit 1) when:

| Message                                                                                          | Cause                                                                                                                                                                  |
| ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `[helper] socket and user are not configured`                                                    | No `[helper] socket`                                                                                                                                                   |
| `runs as root`                                                                                   | Not root (fakehw builds excepted)                                                                                                                                      |
| `helper.user: user: unknown user bcd`                                                            | `[helper] user` names no local account                                                                                                                                 |
| `<file> must be a regular file root owns that no one else can write`                             | `config.toml` is a symlink, not root's, or group- or world-writable, or a directory above it is (3fa9021)                                                              |
| `<dir> must be a directory root owns that no one else can write (the helper socket lives there)` | A directory from `/` down to the slots directory, the socket's directory or the entrypoint's directory is not root's or is group- or world-writable (c139b9c, 9e71b1f) |
| `[kubernetes] allowed_clusters: ... is not https://host[:port]#sha256:<64 hex>`                  | A malformed allow-list entry                                                                                                                                           |
| `command_root_sha256 must be 64 lowercase hex digits per root, comma-separated`                  | A malformed root pin                                                                                                                                                   |

It then creates the socket's directory if needed, binds the socket (mode 0660, group the node account's primary
group) in a private directory and renames it into place, re-arms the self-update watchdog for a swap still waiting
for confirmation, and serves until a signal. Each refused connection is logged as `helper: connection refused` with
the reason; see [troubleshooting](/docs/daemon/troubleshooting#helper-socket-refusals).

```bash
sudo bcd helper --config /etc/bcd/config.toml
```

## bcd sim [#bcd-sim]

Runs `--nodes` real node agents in one process against the configuration's gateway, each on a fake machine. It
exists only in fakehw builds (`go build -tags fakehw`); a host build prints `bcd sim: needs a fakehw build (go build
-tags fakehw)` and exits 2. See [bcd sim](/docs/daemon#bcd-sim) for what each simulated machine does.

| Flag       | Default                | Meaning                                                                                 |
| ---------- | ---------------------- | --------------------------------------------------------------------------------------- |
| `--config` | `/etc/bcd/config.toml` | Gateway URL, pins, journal limits and logging; machines live under `<data_dir>/sim/<i>` |
| `--nodes`  | `1`                    | How many machines; below 1 exits 2                                                      |
| `--seed`   | `1`                    | The fakehw seed of machine 0; machine `i` uses `seed + i`                               |

When any machine has not joined, it reads one token from `BCD_JOIN_TOKEN` or standard input (exit 2 without one)
and needs `join-encryption.jwk` next to the configuration (exit 1 without it). At most 16 joins run at once.

```bash
printf '%s\n' "$SIM_TOKEN" | bcd sim --config ./sim.toml --nodes 5 --seed 100
```

## bcd join [#bcd-join]

Joins this machine to the fleet and writes `node.key` and `identity.json` under `data_dir`. The token comes from
`BCD_JOIN_TOKEN`, else the first line of standard input (at most 4 KiB, a leading byte-order mark dropped), else
`--token-file`; never from an argument.

| Flag                    | Default                | Meaning                                                                                                                                                                                                                                                                                                                                                                                                    |
| ----------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--config`              | `/etc/bcd/config.toml` | The configuration file; with `--server`, the file to write when it does not exist                                                                                                                                                                                                                                                                                                                          |
| `--role`                | `node`                 | `node` or `site`; anything else exits 2. The token decides what is allowed                                                                                                                                                                                                                                                                                                                                 |
| `--token-file`          |                        | Read the token from this file: a regular file, mode 0600 or 0400 (others may not read it), at most 4 KiB. Removed once the join succeeds or the machine had already joined; a read-only mount only logs a warning                                                                                                                                                                                          |
| `--node-id`             |                        | The node an installed OS must join as; implies `--attestor provisioned`. A different identity exits 1                                                                                                                                                                                                                                                                                                      |
| `--server`              |                        | The control plane URL (`https://`): its `/.well-known/better-compute` supplies the join key when `join-encryption.jwk` is absent, and the gateway when `config.toml` is                                                                                                                                                                                                                                    |
| `--attestor`            | `join-token`           | `tpm` joins with the host's TPM (`/dev/tpmrm0`) and no token; `cloud-iid` with an EC2 instance's signed identity document and no token (exits 1 off EC2, before any metadata request; EC2 is the DMI `sys_vendor` `Amazon EC2`, or `Xen` with EC2's BIOS version or UUID on previous-generation types such as t2 and m4); `provisioned` with `--node-id`. `tpm` or `cloud-iid` with `--token-file` exits 2 |
| `--serving-ca-pin`      |                        | `sha256/<base64>`; required with `--server` and no `config.toml` (host builds). With a `config.toml` it must equal `serving_ca_spki_pin`                                                                                                                                                                                                                                                                   |
| `--join-signing-pin`    |                        | Same, for `join_signing_key_pin`                                                                                                                                                                                                                                                                                                                                                                           |
| `--join-encryption-pin` |                        | Same, for `join_encryption_key_pin`                                                                                                                                                                                                                                                                                                                                                                        |
| `--data-dir`            | `/var/lib/bcd`         | The `data_dir` a configuration written by `bcd join` names                                                                                                                                                                                                                                                                                                                                                 |
| `--reset`               | `false`                | Move this node's identity and journal aside first, so it joins again; root only                                                                                                                                                                                                                                                                                                                            |
| `--yes`                 | `false`                | Confirm `--reset`                                                                                                                                                                                                                                                                                                                                                                                          |

What it does, in order:

1. Loads `config.toml`, or, when the file does not exist and `--server` is given, builds one from the well-known
   document checked against the three pin flags ([joining from the well-known
   document](/docs/daemon/install#joining-from-the-well-known-document)).
2. With `--reset`: as root, renames `identity.json`, `node.key`, `node.key.next` and `journal.db` to
   `<name>.reset-<UTC time>`. Without `--yes` it prints what it would move and exits 2. See [joining again with
   \--reset](/docs/daemon/install#joining-again-with---reset).
3. If an identity loads, prints `already joined as <node> (site <site>, role <role>)` and exits 0, without reading
   a token. Under `[helper]` it hands `identity.json` and `node.key` to the `[helper] user` first.
4. Otherwise reads the credential, seals the token to the pinned JoinService key, and calls `Join` with the hardware
   claims. Retryable failures (`UNAVAILABLE`, `RESOURCE_EXHAUSTED`, deadlines) retry with jittered back-off up to 20
   attempts, or exactly when the gateway's `grpc-retry-pushback-ms` says; a pending approval waits and asks again;
   a `tpm` join answers up to 3 challenge rounds. A `cloud-iid` join reads the identity document and its signature from
   IMDSv2 once and sends them as the attestation, with no token. The control plane admits each instance once, so a
   `Join refused` after a retried attempt says the evidence is spent: the lost attempt may have admitted the instance
   (check for the node, or re-join with `--reset` and a join token).
   In a container on EC2 (an EKS DaemonSet without host networking), IMDSv2 answers the token request only when the
   instance's metadata hop limit (`HttpPutResponseHopLimit`) is at least 2; otherwise the read times out with that
   hint, and `bcd node` keeps retrying the cloud section with back-off (5 s doubling to 5 min).
5. Verifies the signed `JoinResponse`, stores the key and identity, writes a bootstrapped `config.toml` (0600) and
   `join-encryption.jwk` (0644), removes the token file, and prints
   `joined as <node> (site <site>, role <role>); certificate valid until <time>`.

| Exit | When                                                                                                                                                                                                                                                                             |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`  | Joined, or already joined                                                                                                                                                                                                                                                        |
| `1`  | `--attestor cloud-iid` on a host whose DMI is not EC2's, configuration or key errors, a refused join (`Join refused`), a rejected `JoinResponse`, a pin flag that does not match `config.toml`, a joined node other than `--node-id`, `--reset` not as root                      |
| `2`  | A positional argument, a bad `--role`, no token (`no join token: set BCD_JOIN_TOKEN or pass the token on standard input`), something that is not a token (`that is not a join token (bcjt_...)`), `--attestor tpm` or `cloud-iid` with `--token-file`, `--reset` without `--yes` |

```bash
# the token on standard input, as the install script does it
printf '%s\n' "$TOKEN" | sudo bcd join --config /etc/bcd/config.toml
# a site agent
sudo BCD_JOIN_TOKEN=bcjt_... bcd join --role site
# with the TPM, no token
sudo bcd join --attestor tpm < /dev/null
# an EC2 instance the control plane created or attached into a bcd pool, no token
sudo bcd join --attestor cloud-iid < /dev/null
# an OS a deploy installed, with no config.toml yet
sudo bcd join --server https://compute.example.com --token-file /etc/bcd/join-token --node-id node_01k5w... \
  --serving-ca-pin sha256/... --join-signing-pin sha256/... --join-encryption-pin sha256/...
```

The join flow, the attestors and the server's refusals are covered in [joining
nodes](/docs/guides/nodes/joining) and [attestation](/docs/guides/nodes/attestation).

## bcd status [#bcd-status]

Asks a running agent (node, site or ramdisk) for its state over its status socket. The socket is created mode 0600
in a 0700 directory, so only the account the agent runs as (or root) can ask.

| Flag       | Default             | Meaning                                                  |
| ---------- | ------------------- | -------------------------------------------------------- |
| `--socket` | `/run/bcd/bcd.sock` | The agent's unix socket (`socket_path` in `config.toml`) |
| `--json`   | `false`             | Print the full status as indented JSON                   |

```text
node       node_01k5w... (site fsn1, role node)
bcd        0.1.0, boot 6f0c...
session    connected (epoch 14, clock offset -3 ms)
spec       applied 42, observed 42
  devices    CONVERGED
  host       PROGRESSING Retrying: helper: not allowed
commands   0 running, 0 waiting for a spec
events     0 unacked, 0 waiting, 0 metrics dropped
identity   certificate until 2026-10-06T02:11:00Z, node key in the TPM, trust bundle v3, signed intent enforced
```

The GPU, `kubernetes` and `telemetry` lines that follow are described in [bcd status](/docs/daemon#bcd-status).
The JSON object has `nodeId`, `siteId`, `role`, `version`, `bootId`, `incarnation`, `journalId`, `connected`,
`epoch`, `clockOffsetMs`, `appliedGeneration`, `observedGeneration`, `sections` (`section`, `state`, `reason`,
`message`), `running`, `deferred`, `pendingEvents`, `droppedMetrics`, `waitingEvents`, `certificateNotAfter`,
`nodeKey` (`tpm` or `file`), `trustVersion`, `commandSigning`, and, when present, `gpus`, `kubernetes` and
`telemetry`. It exits 1 with `the node agent is not running (no socket at <path>)` when nothing listens.

## bcd doctor [#bcd-doctor]

Checks the host and prints one line per check: status (`ok`, `warn` or `fail`), check name, detail.

| Flag       | Default                | Meaning                |
| ---------- | ---------------------- | ---------------------- |
| `--config` | `/etc/bcd/config.toml` | The configuration file |

It exits 1 when any check fails and 0 otherwise; warnings never fail it. A configuration that does not load fails
`config` and skips every check that needs it (Kubernetes, helper, gateway, identity, update). Run it as root:
the `helper` directory checks and the DMI finding only run or read correctly as root. The install script runs it
before the join and only warns on failure.

```bash
sudo bcd doctor --config /etc/bcd/config.toml
```

An excerpt (a GPU host has many more lines):

```text
ok    config    /etc/bcd/config.toml: gateway https://gw.eu.example.com, site fsn1
ok    nvml      driver …, 8 GPUs
ok    hardware  host: 8 GPUs, … inventory sections hashed
ok    kvm       /dev/kvm present
ok    iommu     16 IOMMU units
ok    gateway   gw.eu.example.com:443 reachable in 4ms
ok    identity  node node_01k5w… (site fsn1, role node), certificate valid for 14h12m0s
warn  update    no [update] release_pins: self-update is refused (reinstall to upgrade)
ok    clock     synchronized (estimated error … µs)
```

The real-host preflight (`systemd`, `delegation`, the KVM cause, `vfio`, `iommu-groups`, `clock` and the
inventory findings; e2a0cf0) runs only in host builds. The order is: configuration, NVML, hardware, host checks,
GPU checks, Kubernetes and helper checks, gateway, identity, configuration checks, then the preflight.

### Doctor checks [#doctor-checks]

| Check                                          | Status               | Means                                                                                                                                                                                                                                                                                               | Fix                                                                                                                                  |
| ---------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `config`                                       | `fail`               | The file does not load; the detail names the line and key                                                                                                                                                                                                                                           | Fix the key; see [config.toml](/docs/daemon/configuration)                                                                           |
| `config`                                       | `warn`               | `<file> has mode 0644; the install script writes 0600 (0640 root:bcd under [helper])`: others have any access, or the group can write                                                                                                                                                               | `chmod 0600`, or 0640 with group `bcd` on a node installed with `--unprivileged`. Group read alone (0640) is not a finding           |
| `nvml`                                         | `ok`, `warn`, `fail` | `nvidia-smi` answers with a driver version and GPU count; `warn` when it is missing or the build has `-tags nonvml`; `fail` for any other error                                                                                                                                                     | Install the driver ([nvidia-driver](/docs/daemon/host-config#nvidia-driver)), or see `gpus`                                          |
| `hardware`                                     | `warn`, `fail`       | The inventory could not be collected (`warn`) or hashed (`fail`)                                                                                                                                                                                                                                    | Run `bcd hardware` for the detail                                                                                                    |
| `kvm`                                          | `warn`               | `no /dev/kvm`; on host builds a second line says why: module not loaded, a VM without nested virtualization, or VT-x/AMD-V off in firmware                                                                                                                                                          | `modprobe kvm_intel` or `kvm_amd`; enable nested virtualization or the firmware setting                                              |
| `iommu`                                        | `warn`               | No `/sys/class/iommu` units: no GPU passthrough                                                                                                                                                                                                                                                     | `intel_iommu=on` or `amd_iommu=on` and VT-d/AMD-Vi ([iommu](/docs/daemon/host-config#iommu))                                         |
| `libvirt`                                      | `warn`               | Neither `virsh` nor `cloud-hypervisor`: no VPS instances                                                                                                                                                                                                                                            | Install a runtime if the host should run instances                                                                                   |
| `libvirt`                                      | `fail`               | Under `[helper]`: the node user is in the root-equivalent `libvirt` group; `qemu.conf` lacks `dynamic_ownership = 0`; QEMU's user is not in the node's group                                                                                                                                        | `gpasswd -d bcd libvirt`; set `dynamic_ownership = 0`; `usermod -aG bcd libvirt-qemu`. Rerunning the install script does all three   |
| `cloud-hypervisor`                             | `warn`, `fail`       | Firmware `/usr/share/cloud-hypervisor/CLOUDHV.fd` missing (`warn`); no `systemd-run` (`fail`); under `[helper]`, empty `instance_bridges` (`warn`)                                                                                                                                                  | Install the firmware or systemd; set `[helper] instance_bridges`                                                                     |
| `qemu-img`, `seed`, `nftables`                 | `fail`               | With a runtime installed: no `qemu-img` (no disks), neither `genisoimage` nor `xorriso` (no user data), no `nft` (no anti-spoofing, metadata service or instance dials)                                                                                                                             | Install the package                                                                                                                  |
| `cgroups`                                      | `fail`, `warn`       | No cgroup v2, or no `cpuset`, `cpu` or `memory` controller (`fail`); no `hugetlb` (`warn`)                                                                                                                                                                                                          | Boot with the unified hierarchy; enable the controllers                                                                              |
| `hugepages`                                    | `ok`                 | The free and total 2 MiB and 1 GiB pages, or that there is no pool                                                                                                                                                                                                                                  | Set the [hugepages](/docs/daemon/host-config#hugepages) unit if flavors need them                                                    |
| `vfio`                                         | `warn`               | An IOMMU but `vfio-pci` not loaded                                                                                                                                                                                                                                                                  | `modprobe vfio-pci`                                                                                                                  |
| `systemd`                                      | `fail`, `warn`       | Not the init system (host builds); with `cloud-hypervisor`, older than 247 (`fail`) or unknown (`warn`)                                                                                                                                                                                             | Hold slices and VMM units need systemd                                                                                               |
| `delegation`                                   | `warn`               | Controllers the kernel offers are not in the cgroup root's `subtree_control`                                                                                                                                                                                                                        | Usually settles once bcd's first slice exists                                                                                        |
| `gpu`                                          | `ok`, `warn`, `fail` | One line per GPU: load, temperature, power, memory. `warn` for a pending row remap or MIG mode change; `fail` for failed row remapping (RMA) or uncorrectable ECC errors since the driver loaded                                                                                                    | `gpu.reset` for pending changes; replace a GPU whose remapping failed                                                                |
| `gpus`                                         | `fail`               | `nvidia-smi` failed; the detail ends with a hint (see [troubleshooting](/docs/daemon/troubleshooting#nvidia-smi-errors))                                                                                                                                                                            | Follow the hint                                                                                                                      |
| `gpu-host`                                     | `warn`               | A WSL 2 paravirtual GPU: CUDA containers work, VFIO, `gpu.reset`, MIG and DCGM do not                                                                                                                                                                                                               | Expected on WSL 2                                                                                                                    |
| `nouveau`                                      | `fail`               | The `nouveau` module is loaded                                                                                                                                                                                                                                                                      | Blacklist it, rebuild the initramfs, reboot                                                                                          |
| `gpu-bus`                                      | `warn`, `fail`       | NVIDIA GPUs on the PCI bus with no driver answering or unbound (`warn`); more GPUs bound to `nvidia` than `nvidia-smi` sees, a GPU that fell off the bus (`fail`)                                                                                                                                   | Install the driver; `gpu.reset` or a reboot                                                                                          |
| `persistence`                                  | `warn`               | Persistence mode off on some GPUs                                                                                                                                                                                                                                                                   | Enable `nvidia-persistenced`                                                                                                         |
| `fabric-manager`                               | `fail`               | NVSwitches present but `nvidia-fabricmanager` not active                                                                                                                                                                                                                                            | Start it ([fabric-manager](/docs/daemon/host-config#fabric-manager))                                                                 |
| `gpu-binding`                                  | `warn`               | GPUs, but `bcd node` would run as an ordinary user without `[helper]`: no GPU reservation is placed here                                                                                                                                                                                            | Run `bcd node` as root or configure `[helper]`                                                                                       |
| `kubernetes`                                   | `ok`, `fail`         | The number of allowed clusters; `fail` for a malformed `[kubernetes] allowed_clusters`, an unreadable join state, or a node bcd joined to a cluster with no allow-list                                                                                                                              | List the cluster ([k8s-join](/docs/daemon/host-config#k8s-join))                                                                     |
| `gpu-isolation`                                | `fail`               | Another path hands out GPUs: the container toolkit lets unprivileged pods pick GPUs, another DRA driver offers them, or (with `[host] units = false`) the runtime configuration cannot be read                                                                                                      | See [GPU isolation](/docs/daemon/kubernetes#gpu-isolation)                                                                           |
| `kubelet`                                      | `ok`, `fail`         | On a node bcd joined: whether the kubelet runs                                                                                                                                                                                                                                                      | `systemctl start kubelet`                                                                                                            |
| `device-plugin`                                | `warn`               | `/var/lib/kubelet/device-plugins` missing                                                                                                                                                                                                                                                           | Check the kubelet's root directory                                                                                                   |
| `tpm`                                          | `fail`, `warn`       | Under `[helper]`: the node user is in the `tss` group (`fail`), or membership could not be read (`warn`)                                                                                                                                                                                            | `gpasswd -d bcd tss`, restart `bcd.service`                                                                                          |
| `gateway`                                      | `ok`, `fail`         | A TCP connection to the gateway within 5 seconds (no TLS)                                                                                                                                                                                                                                           | Firewall, DNS, `gateway_url`                                                                                                         |
| `node key`                                     | `ok`, `warn`         | Key in the TPM, or a file key on a host without one; `warn` for a file key on a host with a TPM (the next renewal moves it)                                                                                                                                                                         | Nothing; wait for the renewal                                                                                                        |
| `identity`                                     | `ok`, `warn`, `fail` | `warn`: not joined yet. `fail`: an identity that does not load, or an expired certificate; the detail ends with `bcd join --reset --yes, then bcd join`                                                                                                                                             | [Join again](/docs/daemon/install#joining-again-with---reset)                                                                        |
| `update`                                       | `ok`, `warn`         | Release pins and threshold; `warn` without `[update] release_pins`, for a pending update, and for each failed version                                                                                                                                                                               | Add pins; `bcd update --clear-failed`                                                                                                |
| `slots`                                        | `ok`, `fail`         | The slots are in use but `[update] entrypoint` does not link to the slots' `current`                                                                                                                                                                                                                | Relink, or rerun the install script                                                                                                  |
| `helper`                                       | `ok`, `warn`, `fail` | As root under `[helper]`: a directory to the slots, socket or entrypoint not root's alone (`fail`); what host units bind to: root's signed join record (`ok`), trust on first use (`warn`), a record of another node or a bad signature (`fail`); no command-signing roots pinned anywhere (`fail`) | See [troubleshooting](/docs/daemon/troubleshooting#helper-socket-refusals) and [signed specs](/docs/daemon/host-config#signed-specs) |
| `bmc`                                          | `warn`               | A site agent without `[site] bmc_networks`                                                                                                                                                                                                                                                          | Set the BMC networks                                                                                                                 |
| `iommu-groups`                                 | `ok`, `warn`         | A GPU shares its IOMMU group with a NIC or disk, has no group, or no NVIDIA GPU is on the PCI bus                                                                                                                                                                                                   | Move the GPU to a CPU-attached slot or enable ACS (6cf5122)                                                                          |
| `clock`                                        | `ok`, `warn`, `fail` | `adjtimex` says the kernel clock is synchronized (`ok`, with the estimated error), unsynchronized (`fail`) or cannot be read (`warn`)                                                                                                                                                               | Run `systemd-timesyncd` or `chrony`                                                                                                  |
| `dmi`, `numa`, `dimms`, `disks`, `bmc`, `nics` | `warn`               | Inventory findings: a placeholder or root-only product UUID, no NUMA nodes, non-ECC DIMMs, a worn or failing disk, a BMC that does not answer in-band, no physical NIC                                                                                                                              | See [bcd hardware](#bcd-hardware)                                                                                                    |
| `cpu`, `memory`, `inventory`                   | `fail`, `warn`       | `/proc/cpuinfo` or `/proc/meminfo` unreadable, EDAC uncorrectable errors (`warn`), nothing collected                                                                                                                                                                                                | Check the host                                                                                                                       |

The preflight drops findings whose status is `ok`, and the `kvm`, `iommu` and `cgroup` findings, which other
checks already report.

## bcd hardware [#bcd-hardware]

Prints what bcd sees of this machine and the bring-up findings, without a configuration file (e82d3a8). Use it on a
new server model before it joins.

| Flag        | Default | Meaning                                                                                                                                                                                              |
| ----------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--json`    | `false` | Print the full `NodeInventory` as JSON instead of the summary                                                                                                                                        |
| `--capture` |         | Copy every procfs and sysfs file the collector read into this directory, a fixture `inventory.Collector{Root: dir}` replays; a path ending in `.tar.gz` writes an archive instead (host builds only) |
| `--redact`  | `false` | With `--capture`: replace serials, UUIDs, MACs, WWNs and the hostname with stable stand-ins                                                                                                          |

The summary lists the system, CPU and NUMA nodes, memory and DIMMs, disks, NICs, GPUs, the BMC and the OS, then a
`Findings` block with the same checks doctor's preflight prints. As an ordinary user the product UUID and serials
read empty (they are root-only in sysfs) and the `dmi` finding says so. It exits 1 when a finding fails, and 2 for
`--redact` without `--capture` or `--capture` in a fakehw build.

```bash
sudo bcd hardware
sudo bcd hardware --capture ./r760-h100.tar.gz --redact
```

## bcd update [#bcd-update]

Shows the self-update state: the running version, the release floor, the pin count and threshold, the slots, a
pending update and the versions that failed to start.

| Flag             | Default                | Meaning                                                                                      |
| ---------------- | ---------------------- | -------------------------------------------------------------------------------------------- |
| `--config`       | `/etc/bcd/config.toml` | The configuration file                                                                       |
| `--clear-failed` | `false`                | Forget the failed versions, so a rollout may try them again (9b56c7d); a local decision only |

It reads the records from `[update] slots_dir`, else `/usr/local/lib/bcd` under `[helper]`, else `<data_dir>/bin`
with the records in `data_dir`. On a helper node those are root's, so run it as root.

```text
version 0.9.1, release floor 12, 2 release pins (threshold 2)
slots: current a (0.9.1), other b (0.9.0)
failed: 0.9.2 (bcd 0.9.2 did not start)
```

See [self-update](/docs/daemon/install#self-update) for the slots and records.

## bcd version [#bcd-version]

Prints `bcd <version> <os>/<arch> <go version>`, plus ` tags=<tags>` for a build made with `-tags` (a fakehw build
says `tags=fakehw`). The version is set at link time with `-ldflags "-X main.version=<version>"`; an unset one is
`0.0.0-dev`.

```bash
$ go build -tags fakehw -ldflags "-X main.version=0.1.0" ./cmd/bcd && ./bcd version
bcd 0.1.0 linux/amd64 go1.26.0 tags=fakehw
```

The module requires Go 1.26 (`daemon/go.mod`); the Go version printed is the toolchain's that built the binary.


# config.toml reference (/docs/daemon/configuration)

> Every key and section of bcd's /etc/bcd/config.toml with its type, default, validation and the modes that read it, plus the files bcd keeps under data_dir and elsewhere.



`bcd node`, `bcd site`, `bcd helper`, `bcd join`, `bcd sim`, `bcd doctor` and `bcd update` read one file,
`/etc/bcd/config.toml` unless `--config` names another. The [install script](/docs/daemon/install) writes it with
mode 0600 (0640 `root:bcd` on a node installed with `--unprivileged`); `bcd join --server` writes it when a
machine has none, and a ramdisk writes one into the OS it deploys. This page is the reference for every key
`daemon/internal/config` accepts. The [commands](/docs/daemon/commands) page covers the flags, and
[troubleshooting](/docs/daemon/troubleshooting) covers what goes wrong.

## Format [#format]

The parser accepts a documented subset of TOML 1.0, and refuses everything else with a line number, so a typo
fails loudly instead of being ignored:

| Accepted                                                                                             | Refused                                                 |
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| Comments with `#`, on their own line or after a value                                                | Arrays, inline tables, floats, dates and times          |
| `[section]` headers with a bare name (`A-Z a-z 0-9 _ -`)                                             | Dotted section names and `[[arrays of tables]]`         |
| `key = value` with a bare key                                                                        | Dotted or quoted keys                                   |
| Basic strings `"..."` (escapes `\b \t \n \f \r \" \\ \uXXXX \UXXXXXXXX`) and literal strings `'...'` | Multi-line strings                                      |
| Decimal integers with an optional sign and `_` between digits                                        | Unknown keys and unknown sections                       |
| `true` and `false`                                                                                   | A key or section given twice, a value of the wrong type |

A file over 64 KiB or not valid UTF-8 is refused. Top-level keys must come before the first `[section]`: as in
TOML, a key after a header belongs to that section. Errors read like this:

```text
config /etc/bcd/config.toml: line 9: unknown key plugins.directory
config /etc/bcd/config.toml: line 4: log.json must be true or false
config /etc/bcd/config.toml: site is required
```

Every command that loads the file stops with exit 1 on such an error, and `bcd doctor` fails its `config` check
with the same text. Lists (pins, bridges, CIDRs, clusters, plugin names) are comma-separated strings, since arrays
are not accepted.

## A complete example [#a-complete-example]

```toml title="/etc/bcd/config.toml"
# Written by the better-compute install script. bcd refuses unknown keys.
gateway_url = "https://gw.eu.example.com"
site = "fsn1"
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."
approver_set_sha256 = "<64 lower-case hex digits>"
command_root_sha256 = "<64 hex>,<64 hex>"
# data_dir = "/var/lib/bcd"
# socket_path = "/run/bcd/bcd.sock"

[journal]
max_bytes = 67_108_864
max_events = 10_000

[log]
level = "info"
json = true

[update]
release_pins = "sha256/...,sha256/..."
release_threshold = 2
release_sequence = 12

[helper]
socket = "/run/bcd-helper/helper.sock"
user = "bcd"
instance_bridges = "br0"

[imds]
listen = "169.254.169.254:80"

[kubernetes]
allowed_clusters = "https://10.0.0.10:6443#sha256:<64 hex>"
```

No node needs every section. The install script writes the top-level keys and only the sections its flags ask for;
see [generate the script](/docs/daemon/install#generate-the-script).

## Who reads what [#who-reads-what]

| Section                 | `node` | `site`         | `helper`                                | `join`                       | `sim` | `doctor`, `update` |
| ----------------------- | ------ | -------------- | --------------------------------------- | ---------------------------- | ----- | ------------------ |
| top-level keys          | yes    | yes            | pins, `command_root_sha256`, `data_dir` | yes                          | yes   | yes                |
| `[journal]`, `[log]`    | yes    | yes            | `[log]`                                 | `[log]`                      | yes   |                    |
| `[plugins]`             | yes    | yes            |                                         |                              |       |                    |
| `[update]`              | yes    | yes            | yes                                     |                              |       | yes                |
| `[helper]`              | yes    | ignored        | yes                                     | hands the identity to `user` |       | yes                |
| `[websocket]`           | yes    | yes            |                                         |                              |       |                    |
| `[site]`                |        | yes            |                                         |                              |       | `doctor`           |
| `[imds]`, `[instances]` | yes    |                |                                         |                              |       |                    |
| `[kubernetes]`          | yes    | validated only | `allowed_clusters`                      |                              |       | `doctor`           |
| `[host]`                | yes    |                | yes                                     |                              |       | `doctor`           |
| `[boot]`                |        | yes            |                                         |                              |       |                    |

`bcd ramdisk` reads no file: it builds the same structure from its `bc.*` boot parameters (see
[the ramdisk](/docs/daemon/install#the-ramdisk)). `bcd hardware`, `bcd status` and `bcd version` read none.

## Top-level keys [#top-level-keys]

| Key                       | Type   | Required | Default             | Validation and meaning                                                                                                                                                                                                              |
| ------------------------- | ------ | -------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gateway_url`             | string | yes      |                     | `https://` with a host and no user info, else "gateway\_url must be an https\:// URL without credentials". The regional gateway bcd dials; port 443 unless given. `http://` only with `dev_insecure_h2c`                            |
| `site`                    | string | yes      |                     | Non-empty, no whitespace. The site id; a configuration `bcd join --server` writes holds the site the join answered                                                                                                                  |
| `serving_ca_spki_pin`     | string | yes      |                     | A pin (below). Pins the gateway's serving CA: bcd trusts no system root for the gateway, and the chain the gateway sends must contain a CA with this SPKI                                                                           |
| `join_signing_key_pin`    | string | yes      |                     | A pin. The JoinService key that signs `JoinResponse` and the `SignedTrustBundle` `bcd helper` takes roots from                                                                                                                      |
| `join_encryption_key_pin` | string | yes      |                     | A pin. The JoinService key the token is sealed to; `join-encryption.jwk` (or the well-known key) is used only when it matches                                                                                                       |
| `approver_set_sha256`     | string | no       |                     | 64 lower-case hex digits. The only approver set a daemon without approver keys takes first; until it has that set it runs no destructive command. See [pinning the first set](/docs/operations/pki#pinning-the-first-set)           |
| `command_root_sha256`     | string | no       |                     | 1 to 16 comma-separated lower-case hex SHA-256 digests of command-signing root certificates (DER). `bcd helper` applies a `host.unit` only from a spec signed under them; see [signed specs](/docs/daemon/host-config#signed-specs) |
| `data_dir`                | string | no       | `/var/lib/bcd`      | Absolute path. Identity, journal and every other file in [the data directory](#the-data-directory)                                                                                                                                  |
| `socket_path`             | string | no       | `/run/bcd/bcd.sock` | Absolute path of the status socket that `bcd status` reads (its `--socket` flag does not read this key; pass the same path)                                                                                                         |
| `gateway_server_name`     | string | no       |                     | The name the gateway certificate must carry instead of `gateway_url`'s host: a gateway reached by IP address, behind an L4 balancer, or from WSL. Also the default `[websocket] audience`                                           |
| `dev_insecure_h2c`        | bool   | no       | `false`             | Allows `http://` (cleartext HTTP/2) for `gateway_url`, and `ws://` for `[websocket] url`. A host build refuses to start with it ("dev\_insecure\_h2c is only accepted by fakehw development builds")                                |

A **pin** is `sha256/` followed by the standard base64 of the 32-byte SHA-256 of a SubjectPublicKeyInfo, the form
HPKP and `curl --pinnedpubkey` use. Anything else is "a pin must be sha256/ followed by the standard base64 of a
32-byte digest". The three pins come from the sealed PKI bundle (`bootstrapPins` in `better-compute/pki`, or
`daemon install-script`); they are the node's trust anchors, never taken from the network. A `bcd join` pin flag
that differs from the file is refused ("--serving-ca-pin does not match /etc/bcd/config.toml").

## \[journal] [#journal]

| Key          | Type    | Default             | Validation and meaning                                                                                             |
| ------------ | ------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `max_bytes`  | integer | `67108864` (64 MiB) | 1 to 64 MiB: may lower the cap, never raise it. The bbolt journal's size bound                                     |
| `max_events` | integer | `10000`             | 1 to 10,000. Unacked events kept; when full, metric events are dropped oldest first, results and transitions never |

See [the journal](/docs/daemon/delivery#the-journal).

## \[log] [#log]

| Key     | Type   | Default | Validation and meaning                                                         |
| ------- | ------ | ------- | ------------------------------------------------------------------------------ |
| `level` | string | `info`  | `debug`, `info`, `warn` or `error`                                             |
| `json`  | bool   | `true`  | JSON lines on standard error; `false` for text. `bcd ramdisk` always logs JSON |

## \[update] [#update]

Self-update ([self-update](/docs/daemon/install#self-update)). Without release pins bcd takes no new binary:
the control plane alone can never hand a node one.

| Key                 | Type    | Default                                                 | Validation and meaning                                                                                                                                                                                          |
| ------------------- | ------- | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `release_pins`      | string  |                                                         | Comma-separated pins of the release-signing keys; a pin listed twice counts once                                                                                                                                |
| `release_threshold` | integer | `1`                                                     | At least 1, at most 16, and at most the number of distinct pins when pins are set. How many distinct pinned keys must have signed a release manifest                                                            |
| `release_sequence`  | integer | `0`                                                     | Not negative. The release the node was installed at: no manifest with a lower sequence is taken. The floor is the highest of this, the sequence linked into the binary, and the recorded floor in `update.json` |
| `entrypoint`        | string  | `/usr/local/bin/bcd`                                    | Absolute path; the binary `bcd.service` starts. The first update makes it a symlink to the slots' `current`. Under `[helper]` it may not be inside `data_dir`                                                   |
| `slots_dir`         | string  | `<data_dir>/bin`; under `[helper]` `/usr/local/lib/bcd` | Absolute path for `bcd-a`, `bcd-b`, `current`, `previous`, `slots.json` and, beside them, `upgrade.json` and `update.json`. Under `[helper]` it may neither be inside `data_dir` nor contain it                 |

> **Why the slots stay out of data_dir under the helper.** 
  Under `[helper]` the unprivileged node owns `data_dir`. Slots or an entrypoint there would let it plant the binary
  root adopts and runs, so the configuration is refused ("under bcd helper, update.slots\_dir may not be inside
  data\_dir"; c139b9c, 9e71b1f), and `bcd helper` also refuses to start unless every directory down to the slots and
  the entrypoint is root's alone.

Without `slots_dir` and without `[helper]`, `upgrade.json` and `update.json` stay in `data_dir`.

## \[helper] [#helper]

Privilege separation: `bcd node` runs as `user` and sends its privileged work to `bcd helper`, as root, on
`socket`. See [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged).

| Key                | Type   | Default | Validation and meaning                                                                                                                                                                                                                                                                            |
| ------------------ | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `socket`           | string |         | Absolute path. Empty: no helper, and `bcd node` does its privileged work itself (as root). The install script uses `/run/bcd-helper/helper.sock`, outside the node's `/run/bcd`                                                                                                                   |
| `user`             | string |         | Required with `socket`, never `root` ("helper.socket needs an absolute path and helper.user, the unprivileged account bcd node runs as"). The account `bcd node` runs as and the only non-root caller the helper serves                                                                           |
| `instance_bridges` | string |         | Comma-separated interface names (1 to 15 of `A-Z a-z 0-9 _ . -`), else "helper.instance\_bridges: ... is not an interface name". The bridges an instance NIC may join: the helper refuses any libvirt domain or `cloud-hypervisor` tap on another bridge, and with none set defines no NIC at all |

Security notes:

* The helper reads its own copy of this file as root and refuses to start unless it is a regular file root owns,
  writable by no one else, in directories only root can write. `[host] units`, `command_root_sha256` and the join
  pin in it decide what root does for the node, so never hand the file to the `bcd` account or make it
  group-writable.
* The socket's directory must be root's and closed to group and other writes; the helper creates the socket there
  with mode 0660, group the node user's primary group.
* The helper serves a peer only when its uid and gid are `user`'s (or root) and its `/proc/<pid>/exe` is the
  entrypoint, a slot binary or the helper's own binary.
* `bcd join` running as root with this section hands `identity.json` and `node.key` to `user` when it finishes, and
  keeps root's signed join record that the helper binds host units to.

## \[plugins] [#plugins]

bcx driver plugins (see [driver plugins in bcd](/docs/extending/drivers#driver-plugins-in-bcd) and [the plugin
sandbox](/docs/extending/drivers#the-plugin-sandbox)). Read by `bcd node` and `bcd site`.

| Key          | Type    | Default    | Validation and meaning                                                                                                                                                                                                                                                           |
| ------------ | ------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dir`        | string  |            | Directory holding the plugin binaries and their `pins` file (sha256sum format). Empty: no plugin runs. A missing `pins` file logs "no plugin pins; no plugin runs"                                                                                                               |
| `user`       | string  |            | Who plugins run as when bcd is root. `root` or `0` is refused. Unset, each plugin gets its own uid and gid from `0x7FFE0000` to `0x7FFEFFFF`, assigned in name order                                                                                                             |
| `privileged` | string  |            | Comma-separated plugin names that run as root with no shim and no seccomp filter, such as a Kata runtime; each logs "privileged plugin: root, no seccomp filter"                                                                                                                 |
| `sandbox`    | bool    | `true`     | Run other plugins through bcd's `__bcx-exec` shim with `no_new_privs` and a seccomp filter. Linux only: elsewhere set `false` or no plugin starts                                                                                                                                |
| `cgroup`     | string  |            | An absolute path ("plugins.cgroup must be an absolute path"), meant to be a directory under `/sys/fs/cgroup` in bcd's delegated cgroup; each plugin runs in a child `bcx-<name>`. If it cannot be prepared, no plugin runs ("the plugins' cgroup is not usable; no plugin runs") |
| `memory_max` | integer | `0` (none) | Not negative. Each plugin cgroup's `memory.max` in bytes                                                                                                                                                                                                                         |
| `pids_max`   | integer | `0` (none) | Not negative. Each plugin cgroup's `pids.max`                                                                                                                                                                                                                                    |

A node with `[plugins]` needs the systemd drop-in that gives bcd `CAP_SETUID`, `CAP_SETGID` and its delegated
cgroup; see [the plugin drop-in](/docs/daemon/install#the-plugin-drop-in). `--unprivileged` cannot host plugins.

## \[websocket] [#websocket]

The [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04) for networks whose TLS-inspecting proxy
strips client certificates. Read by `bcd node` and `bcd site`; setting `url` turns it on.

| Key             | Type   | Default                                    | Validation and meaning                                                                                                                                                                                                                      |
| --------------- | ------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`           | string |                                            | `wss://` with a host and no user info (`ws://` only with `dev_insecure_h2c`), else "websocket.url must be a wss\:// URL without credentials". The gateway's `.../v1/session` endpoint                                                       |
| `proxy_ca_file` | string |                                            | Absolute path of a PEM file holding the proxy's CA ("websocket.proxy\_ca\_file: ... holds no PEM certificate" otherwise). Trusted for the TLS connection only, never for the gateway's own proof, which must chain to the pinned serving CA |
| `audience`      | string | `gateway_server_name`, else the URL's host | The gateway's public host the node's proof names                                                                                                                                                                                            |

Over the fallback bcd takes only signed commands and specs and opens no tunnels; a node whose trust bundle holds no
command-signing roots refuses to dial ("the WebSocket fallback needs command-signing roots (every command and spec
signed)"). Renewals go to `POST /v1/renew` on the same endpoint.

## \[site] [#site]

| Key            | Type   | Default | Validation and meaning                                                                                                                                                                                |
| -------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bmc_networks` | string |         | Comma-separated CIDRs ("site.bmc\_networks: ... is not a CIDR" otherwise). A site agent never sends `site.httpProxy` into them, whatever the control plane asks; doctor warns a site agent without it |

## \[boot] [#boot]

A site agent's boot server, read only by `bcd site`. See [a site agent](/docs/daemon/install#a-site-agent) for an
example and [sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site) for its behaviour.

| Key                 | Type    | Default         | Meaning                                                                                                                        |
| ------------------- | ------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `listen`            | string  |                 | HTTP address of the boot server, e.g. `0.0.0.0:8080`; empty turns it off. Needs `join-encryption.jwk` beside the configuration |
| `url`               | string  |                 | How machines reach the boot server; ramdisks it boots get `<url>/cache` as their image cache                                   |
| `kernel`, `initrd`  | string  |                 | URLs of the site's ramdisk kernel and initramfs                                                                                |
| `cmdline`           | string  |                 | Appended to every ramdisk's kernel command line                                                                                |
| `server_ip`         | string  |                 | The agent's IPv4 address on the provisioning network                                                                           |
| `proxy_dhcp`        | bool    | `false`         | Answer PXE clients on UDP 67 and 4011                                                                                          |
| `tftp`, `tftp_root` | string  |                 | TFTP's UDP address (e.g. `:69`) and the directory of iPXE builds                                                               |
| `cache_max_bytes`   | integer | `0` (unbounded) | The image cache's budget; least recently used images go first, never one `cache_images` still names                            |

`proxy_dhcp` and `tftp` need `server_ip` (IPv4) and `url`, or `bcd site` exits 2. These keys are checked when
`bcd site` starts, not when the file is parsed.

## \[imds] [#imds]

| Key      | Type   | Default | Meaning                                                                                                                                                                                                                    |
| -------- | ------ | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `listen` | string |         | Where `bcd node` on a VPS host serves its instances' metadata service, for example `169.254.169.254:80` on the instances' bridge. Empty turns it off. A failure to serve is logged ("metadata service stopped"), not fatal |

## \[instances] [#instances]

| Key                     | Type    | Default | Validation and meaning                                                                                                                                                                                   |
| ----------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `balloon_floor_percent` | integer | `50`    | 10 to 100, else "instances.balloon\_floor\_percent must be between 10 and 100". The share of its memory a ballooned guest always keeps; see [memory ballooning](/docs/daemon/delivery#memory-ballooning) |

## \[kubernetes] [#kubernetes]

| Key                | Type   | Default         | Validation and meaning                                                                                                                                                                                                                                                                                            |
| ------------------ | ------ | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allowed_clusters` | string |                 | Comma-separated `https://host[:port]#sha256:<64 hex>` (the API server and its CA certificate's SHA-256; no path). The only clusters the `k8s-join` unit may join this machine to. Empty refuses every join, as root and under the helper alike. A malformed entry stops `bcd node` and `bcd helper` from starting |
| `gpu_api`          | string | `device-plugin` | `device-plugin` or `dra`. How a node bcd did not join itself (a DaemonSet) offers bound GPUs to the kubelet; any other value stops the agent with `[kubernetes] gpu_api must be "device-plugin" or "dra"`                                                                                                         |

Joining a cluster hands it root on the machine (privileged pods), which is why root, not the control plane, names
the clusters. See [k8s-join](/docs/daemon/host-config#k8s-join) and [GPUs for pods](/docs/daemon/kubernetes#gpus-for-pods).
Both keys are validated by the agent at start (`bcd node`, `bcd site`) and `allowed_clusters` also by `bcd helper`
and `bcd doctor`, not by the parser, so a bad entry does not stop `bcd join` or `bcd update`.

## \[host] [#host]

| Key     | Type | Default | Meaning                                                                                                                                                                                                                                                                                                                                     |
| ------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `units` | bool | `true`  | `false` when something else manages the host's configuration, such as the NVIDIA GPU operator beside [bcd as a DaemonSet](/docs/daemon/kubernetes): bcd applies no `NodeSpec.host` unit and reports the section `UNSUPPORTED` with reason `HostManagedExternally`. `bcd helper` honours its own root-owned copy, refusing every `host.unit` |

## The data directory [#the-data-directory]

`data_dir` (default `/var/lib/bcd`, mode 0700) is bcd's own. Under `[helper]` it belongs to the `bcd` account, so
nothing root trusts lives there.

| Path under `data_dir`                             | Written by                      | Holds                                                                                                                  |
| ------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `node.key`                                        | `bcd join`, renewals            | The node key (0600): PKCS#8 PEM, or a `BCD TPM NODE KEY` record of a key in the TPM                                    |
| `node.key.next`                                   | renewals                        | The next key until its certificate is stored; promoted or discarded on the next load                                   |
| `identity.json`                                   | `bcd join`, renewals, `Welcome` | Node id, site, role, certificate chain, expiry, trust bundle and the server time mark (0600)                           |
| `journal.db`                                      | the agent                       | The bbolt journal: commands, results, events, specs, spent exec-ticket ids                                             |
| `journal.db.corrupt-<unix>`                       | the agent                       | A journal that did not open, moved aside; a new one starts                                                             |
| `journal.db.<old node>-<unix>`                    | `bcd join`                      | The journal of an earlier identity, moved aside by a join as another node                                              |
| `<name>.reset-<UTC time>`                         | `bcd join --reset --yes`        | The identity, key, next key and journal moved aside; never deleted by bcd                                              |
| `faults.json`                                     | the agent                       | This boot's XID conditions, so a restart in the same boot keeps them                                                   |
| `gpus-seen.json`                                  | `bcd node` (host builds)        | GPUs the driver had, so one passed through to a VM keeps its inventory entry and PCI address                           |
| `host-intent.json`                                | `bcd node`                      | The signed spec that last set the host section (0600), which `bcd helper` checks units against                         |
| `releasing.json`                                  | `bcd node` on Kubernetes        | Pending Kubernetes releases                                                                                            |
| `instances-rebound.json`                          | `bcd node`                      | Instances whose identity was rebound after a memory restore                                                            |
| `batch-leaves.json`, `unbatched-destructive.json` | quorum nodes                    | Batch leaves already run; the rolling-hour count of unbatched destructive commands                                     |
| `bmc-hosts.json`, `boot-entries.json`             | `bcd site`                      | BMCs the agent has driven; persisted boot entries                                                                      |
| `cache/`                                          | `bcd site`                      | The image cache, `cache/<sha256>`                                                                                      |
| `snapshots/`                                      | `bcd node`                      | Snapshots being uploaded or restored                                                                                   |
| `staging/`                                        | `bcd node`                      | Self-update downloads, `staging/bcd-<version>`                                                                         |
| `bin/`                                            | self-update                     | The slots (`bcd-a`, `bcd-b`, `current`, `previous`, `slots.json`) unless `[update] slots_dir` or `[helper]` moves them |
| `upgrade.json`, `update.json`                     | self-update                     | A swap awaiting its `Welcome`; the release floor and failed versions (here only without `slots_dir` and `[helper]`)    |
| `libvirt/`, `cloud-hypervisor/`                   | the runtimes                    | Instance disks, seeds, base images and records (`instances/<id>/vm.json`)                                              |
| `volumes/`                                        | the runtimes                    | Persistent volumes `<id>.raw` and `<id>.json` (0600)                                                                   |
| `sim/<i>/`                                        | `bcd sim`                       | One simulated machine's whole data directory                                                                           |

Outside `data_dir`:

| Path                                                                             | Holds                                                                                            |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `/etc/bcd/config.toml`, `/etc/bcd/join-encryption.jwk`                           | This file and the JoinService encryption key (public)                                            |
| `socket_path` (`/run/bcd/bcd.sock`)                                              | The status socket, 0600                                                                          |
| `/etc/cdi/bcd-<allocation>.json`                                                 | CDI specs of GPU holds (root's, or the helper's)                                                 |
| `/run/bcd/allocations/<allocation>.env`                                          | Each hold's env file, rewritten when missing after a reboot                                      |
| `/run/bcd/instances.nft`                                                         | The instance firewall ruleset                                                                    |
| `/sys/fs/cgroup/bcd.slice/bcd-alloc.slice/bcd-alloc-<allocation>.slice`          | A hold's cgroup slice, created through `systemctl`                                               |
| `/var/lib/bcd-hostconfig/`                                                       | Root's host-unit records, `k8s-join.json`, `helper-intent.json` and the helper's libvirt staging |
| `[update] slots_dir`, `[update] entrypoint`                                      | The binary slots and the link `bcd.service` starts                                               |
| `/etc/systemd/system/bcd-rollback.service`, `bcd.service.d/50-bcd-rollback.conf` | Written by the first self-update under systemd                                                   |
| `[helper] socket`                                                                | The helper's socket, 0660                                                                        |

The install script's own files and units are listed in [files on the host](/docs/daemon/install#files-on-the-host).


# Delivery (/docs/daemon/delivery)

> How NodeSpec generations and one-shot commands reach bcd, the acceptance checks, lanes and credits, cancellation, signatures, and the journal that makes delivery safe.



The control plane tells a node what to do in two ways, and the difference matters. **Desired state** is a
standing fact: which devices are held for which allocation, which instances should run, which SSH CA to trust.
**Commands** are one-shot actions: reboot, run diagnostics, collect logs. The rule from PLAN.md §7.3:

> If losing it leaves the wrong steady state, it is desired state. If running it late could do harm, it is a
> command with `not_after`, plus `min_generation` when it presupposes a spec.

Both travel down the node's [session](/docs/daemon/sessions), and both are delivered at least once. The daemon's
journal turns at-least-once delivery into at-most-once effect. The concepts are introduced
in [desired state and commands](/docs/guides/concepts/desired-state-and-commands); this page is the wire-level
detail.

## Desired state [#desired-state]

### NodeSpec and its sections [#nodespec-and-its-sections]

A NodeSpec is split into sections. The daemon converges them in this order, each with an
idempotent `plan(desired, observed)` and `apply`:

| Section     | Holds                                                                                                                                                                                                                                                                | Phase |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `host`      | Config units (kernel modules, sysctl, huge pages, the NVIDIA driver, Fabric Manager, the MIG layout, GPU sharing, the container toolkit, a Kubernetes join), each a unit name and its settings as canonical JSON; see [host configuration](/docs/daemon/host-config) | v0.2  |
| `devices`   | Device assignments: allocation, reservation, binding, devices, extra env lines                                                                                                                                                                                       | v0.1  |
| `instances` | VPS instances with their runtime, size, image, NICs and GPUs                                                                                                                                                                                                         | v0.3  |
| `access`    | The exec-ticket JWKS, SSH CA keys and principals, revoked certificate serials; filled when `tunnels` is configured                                                                                                                                                   | v0.3  |
| `telemetry` | The OTLP endpoint, export interval, resource attributes and a secret reference for headers                                                                                                                                                                           | v0.1  |
| `daemon`    | The self-update target: version, URL, SHA-256, signature                                                                                                                                                                                                             | v0.4  |
| `site`      | The site agent's boot lease state and `cache_images`, the images its cache pre-warms (`url`, `sha256`, `format`; the agent re-verifies each digest before serving it); draft, site agents only                                                                       | v0.2  |

The control plane builds the spec in the NodeSpec reconciler: the base sections plus every module's `nodeSpec()`
contribution (a `NodeSpecPart`; the `site` section merges field by field across modules), stored in `nodes.spec`
as protobuf JSON, with `cache_images` sorted by URL. It raises `nodes.generation` only when the canonical JSON
(sorted keys) differs from the stored spec, so re-running the reconciler never produces a new generation for
nothing. Desired state applies once a node is adopted or deployed.

### Generations [#generations]

Three numbers describe where a node stands:

| Number                | Owner                                     | Meaning                                                               |
| --------------------- | ----------------------------------------- | --------------------------------------------------------------------- |
| `generation`          | control plane, `nodes.generation`         | The newest spec written                                               |
| `applied_generation`  | daemon, in `Hello`, heartbeats and status | The newest spec the daemon holds, converged or not                    |
| `observed_generation` | control plane, from status reports        | The newest spec at which every section converged or failed terminally |

`observed_generation` is written with `WHERE observed_generation < $g`, so it never moves backwards, whichever
session a late report arrives on. Controllers that wait for the node to act on a spec wait for it.

### How a spec reaches the node [#how-a-spec-reaches-the-node]

* **At Hello**, when `generation` is above `applied_generation`, the gateway sends a full `DesiredState` before
  replaying commands.
* **On change**, the transaction that wrote a new generation pokes the node's gateway, which pushes the spec to that
  node's session at once.
* **As a backstop**, every 10 seconds the gateway compares each open session with `nodes.generation` and sends a
  full spec to any node that is behind both what it holds and what was already sent on this session.
* **On request**, the daemon sends `SpecRequest{have_generation, reason}` and gets the full spec back.

| SpecRequest reason    | When                                                                               |
| --------------------- | ---------------------------------------------------------------------------------- |
| `PATCH_BASE_MISMATCH` | A patch's `base_generation` is not the applied generation                          |
| `MIN_GENERATION`      | A command's `min_generation` is above the applied generation                       |
| `NO_SPEC`             | The journal holds no usable spec (new journal, corruption, failed signature check) |

A `DesiredStateBody` carries the node id, the generation, a `spec_hash` and either a `full` NodeSpec or a `patch`.
A patch applies against `base_generation`: every section it sets replaces that section wholesale (an empty section
clears it), unset sections stay. The daemon ignores a body for another node and any generation not above the one
it holds. It never recomputes `spec_hash`; it echoes it back as `applied_spec_hash`. The FakeDaemon applies both
forms. The gateway in this release sends full specs only (`base_generation` 0). When the NodeSpec controller stored
the encoded body of the current generation (`nodes.spec_body`, and with a signing key `nodes.spec_signature`,
migration `core/0021_signed_specs`), the gateway relays those bytes verbatim and never re-encodes them; otherwise it
renders `nodes.spec` unsigned with pki's `desiredStateBodyFromJson`, the writer's own encoder, byte for byte.

The daemon stores the last full spec, its hash and its signature in the journal, and enforces it while offline.

### Convergence and status [#convergence-and-status]

`observed_generation` advances to a spec's generation only when every section has converged or failed terminally;
a retryable failure keeps the section `PROGRESSING` and it is retried on the next pass. A section the daemon has no
converger for converges trivially when empty and is `UNSUPPORTED` otherwise, which counts as terminal, with
condition `SectionUnsupported` (reason: the field number) so the rest of the spec still converges. bcd applies the
rule with `converge.Run`. It has convergers for `host` (the config units, 6c2343f; see
[host configuration](/docs/daemon/host-config)), `devices`, `access`, `instances` (below), `telemetry` (since 27c7ee2
bcd's OTLP export of its own metrics; see [node metrics](/docs/operations/observability#node-metrics-from-bcd)), `site` (the boot lease the boot server obeys, and the
`cache_images` the site agent pulls into its image cache: the section stays `PROGRESSING` while images warm and
`FAILED` for an image whose bytes do not match its digest) and, on `bcd node` and `bcd site`, `daemon`
([self-update](/docs/daemon/install#self-update)). A section number bcd does not know is `UNSUPPORTED`. With
`[host] units = false` bcd has no `host` converger: a `host` section with units is `UNSUPPORTED` with reason
`HostManagedExternally`, which raises no condition and does not hold back `observed_generation` (a193249). A non-empty `daemon` section is `UNSUPPORTED` in `bcd sim` and
`bcd ramdisk`. A `bcd node` under [`bcd helper`](/docs/daemon#privilege-separation) converges it too: it downloads
the release, and the helper verifies and swaps it (2d67b25). The FakeDaemon, in turn, converges `instances` and
`daemon` when its options enable them but has no `access` converger.

The daemon reports progress in `StatusReport`: observed and applied generation, the spec hash, one
`SectionStatus` per section (`CONVERGED`, `PROGRESSING`, `FAILED`, `UNSUPPORTED`, with the generation, a reason and
a message), and its full condition set. Conditions carry `age_ms`, an age on the daemon's monotonic clock, and the
control plane derives `since` from `compute_now()`: agents report ages, not timestamps. From v0.3 the report also
lists every VM or container the host runtimes know. Status is coalesced latest-wins: the gateway keeps the last
report per session and writes it about once a second, running the host's status handlers (XID health, VPS instance
status), each under its own savepoint. Every report runs the handlers, but since 318b396 the `node.status` event is
published only when the report says something new: `statusKey` hashes it without its gauges (host pressure and
volume storage), and a session skips a report whose key it already published (a new session publishes its first
report again). See [scaling](/docs/operations/scaling#gateway-sessions).

### Device assignments [#device-assignments]

The `devices` section is how a `hold` [reservation](/docs/guides/capacity/reservations) reaches a machine. For each
assignment the daemon writes a CDI spec `/etc/cdi/bcd-<allocation_id>.json`, an env file under
`/run/bcd/allocations/` and device ownership, then journals an attested `AllocationBound` event with the CDI device
names (`bcd.dev/gpu=<id>`). Removing an assignment releases it (`AllocationReleased`). An assignment the daemon
cannot honor (a device missing, a device already held by another allocation, an unsupported binding) produces an
`AllocationFailed` event with a reason such as `overlap` or `device-missing`, and the section fails terminally at
that generation. The daemon never drops an assignment on its own, not even while disconnected. bcd and the
FakeDaemon both implement this converger: bcd writes the CDI spec and env file in the FakeDaemon's shape, reads its
holds back from `/etc/cdi` on every pass, and signs `AllocationBound` and `AllocationReleased` with the node key.
`tests/e2e/bcd.test.ts` binds and releases a 2-GPU reservation through it.

What bcd holds (46a61db, d02ba4f):

| Device class | Atom id                 | Resolves to                                        | Held when                                                                                                                                 |
| ------------ | ----------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `gpu`        | the GPU's UUID          | itself                                             | the inventory has it                                                                                                                      |
| `gpu.mig`    | `<gpuUuid>/gi<g>/ci<c>` | the MIG instance's `MIG-…` UUID from the inventory | the instance exists; until the [`mig-layout` unit](/docs/daemon/host-config#mig-layout) has made it, the hold fails with `device-missing` |
| `gpu.slot`   | `<gpuUuid>/s<k>`        | its GPU                                            | the GPU exists                                                                                                                            |

* **Conflicts** are parent-aware: a whole GPU is refused (`overlap`) while part of it is held, a MIG instance or slot
  while its GPU is held whole, and two holds never share an atom. No hold is bound on a GPU the `instances` converger
  passes through to a VM, counting the domains still running after the last pass, and since 6b1480b a VM start is
  refused a GPU that a live hold has, whole, as a MIG instance or as a slot, since passthrough would take it off its
  host driver.
* **CDI names** cannot contain `/`, so each device's CDI name is its atom id with `/` as `_`, and the real id rides in
  the device annotation `bcd.dev/device-id` (CDI 0.6), which is how bcd reads its holds back. `AllocationBound`
  reports the names as `bcd.dev/gpu=<name>`.
* **Env.** Devices carry no env of their own. A hold's spec has one spec-level `NVIDIA_VISIBLE_DEVICES` listing what
  the driver knows (a MIG instance's UUID; a GPU once, however many of its slots are held), and so does the env file.
  Every workload also gets `BCD_ALLOCATION_ID`, and a slot hold gets `CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps` when
  that directory exists (the [`gpu-sharing` unit](/docs/daemon/host-config#gpu-sharing) runs MPS).
* **Bindings.** `BINDING_HOLD` and `BINDING_KUBERNETES` bind the same way; a Kubernetes hold's spec also carries the
  annotation `bcd.dev/binding: kubernetes` and no `NVIDIA_VISIBLE_DEVICES`, since the kubelet device plugin hands a
  pod only part of it and sets its own (see [Kubernetes workers](/docs/guides/capacity/kubernetes)). Any other binding
  fails with `unsupported-binding`, and any other device class with `unsupported-device`.

**Host resources.** A reservation that booked host CPU, memory or 1 GiB pages (see
[host resources](/docs/guides/capacity/reservations#host-resources)) reaches the node as `DeviceAssignment.limits`
(c5a419c): `cpu_millis` from `vcpu` (1000 per CPU), `memory_bytes` from `memoryGiB`, `hugepages_1gi`, and a cpuset on
the NUMA nodes of the hold's devices, `cpus` only when those nodes have at least `vcpu` CPUs and `mems` only when they
hold the booked memory. bcd then creates the hold's cgroup-v2 slice,
`bcd.slice/bcd-alloc.slice/bcd-alloc-<allocation>.slice`, before the CDI spec names it (260e33c):

| Slice `bcd-alloc-<allocation>.slice`             | From                                                                                                                      |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `AllowedCPUs`, `AllowedMemoryNodes`              | `cpus`, `mems`                                                                                                            |
| `CPUQuota`                                       | `cpu_millis`, as whole percents of one CPU                                                                                |
| `MemoryMax`, and `MemoryHigh` at 90% of it       | `memory_bytes`                                                                                                            |
| `hugetlb.2MB.max`, `hugetlb.1GB.max` in cgroupfs | `hugepages_2mi`, `hugepages_1gi` (systemd has no property; bcd enables the `hugetlb` controller on the slice's ancestors) |

The limits are runtime properties (`systemctl set-property --runtime`), which a reboot drops while the hold stays,
so a pass that finds a held hold's slice missing creates it again. The CDI spec names the slice in the annotation
`bcd.dev/cgroup-parent`, and every workload gets `BCD_CGROUP_PARENT=bcd-alloc-<allocation>.slice`: containerd, Slurm
or a wrapper places the workload there; bcd does not move processes. On release, bcd stops the slice only once
`cgroup.events` says it is empty, so a release never kills a workload. A slice still in use is kept, and since
e01d7ff every pass, the 10-minute drift pass included, lists the hold slices on the host and tries again to stop any
whose hold is neither wanted nor released in that pass, so it goes once its workload exits. A hold without limits
gets no slice, and nothing changes for it. Neither does a `BINDING_KUBERNETES` hold: its pods live in the kubelet's
own cgroups (`kubepods.slice`), where the kubelet enforces their requests and limits, so bcd creates no slice for it
and names no cgroup parent in its CDI spec or env (e01d7ff).

**Host admission of holds** (31555bf). A new hold whose limits carry a memory limit or huge pages is admitted
against the host the way an instance start is (see [host-side admission](#instances)): from `MemTotal` less the huge
page pool and 1 GiB of headroom come the running and paused domains' commitments and those of the holds already
bound (their `MemoryMax`, and the huge pages of their limits not yet faulted in, per NUMA node). A hold that does not
fit fails with `AllocationFailed{reason: resources-exhausted}`, and the control plane re-places it. VM starts count
the bound holds in turn, so VMs and holds never take what the other was promised. A Kubernetes hold's booked limits
count here too, although it has no slice. A host whose memory cannot be read binds without the check, with a warning.

**Capabilities** (409f783). An older bcd drops `DeviceAssignment.limits` and `BINDING_KUBERNETES` as unknown fields,
so a hold would run without its limits, or a Kubernetes reservation's GPUs would fail as `unsupported-binding` only
after placement. On a GPU host bcd's `Hello` therefore advertises `devices.limits` and `binding.kubernetes`, which the
gateway writes as `bc.dev/cap.<name>` system labels, and capacity places a reservation that books host resources only
on `devices.limits` nodes and a `binding: 'kubernetes'` one only on `binding.kubernetes` nodes (see
[capabilities](/docs/daemon/sessions#hello)). A plain GPU hold without limits still goes to older daemons.

Under [privilege separation](/docs/daemon#privilege-separation) (`[helper] socket` set, 28dc2ae) the converger does
the same work through `bcd helper`: the node asks for a `file.write` or `file.remove` of kind `cdi` or
`allocation-env` with the allocation id, and the root helper builds the path itself; a hold's slice is
`slice.apply` or `slice.remove`, which accept only a `bcd-alloc-<id>.slice` name and cpulists. A refused or failed operation is
an error of the action that asked for it (`helper: …`), exactly as a failed direct write would be: retryable, so the
section stays `PROGRESSING` until a later pass succeeds. The `access`
converger's sshd files and reload, the instances' nftables table and its 5-second check, `gpu.reset`,
`node.reboot` and the privileged half of self-update (the `daemon` section and `daemon.upgrade`: verify, swap,
confirm, roll back) go through the helper the same way. `tests/e2e/bcd.test.ts` binds and releases a GPU hold, and
delivers a signed update, with an unprivileged node and a real helper.

### Break-glass SSH and tunnel keys [#break-glass-ssh-and-tunnel-keys]

The `access` section carries what sshd and the tunnel code on the node trust. bcd's converger writes the CA keys,
this node's own principals (`<account>@<nodeId>` only) and a KRL of the revoked serials into `/etc/ssh`, plus a
drop-in that points sshd at them, and reloads sshd only when that drop-in changes; an empty section removes all of
it. It also keeps the exec-ticket JWKS, which every `OpenTunnel` is verified against, and takes a new one on every
pass, even when no file changes. Because the CA keys and principals become lines of sshd's files, bcd refuses rather
than strips anything that could add a line: a CA key or principal with a control character, a CA key that is not
exactly one OpenSSH public key, or a principal with whitespace makes the whole spec invalid, signed or not, and it is
not applied (the converger checks again and fails terminally). See
[break-glass SSH](/docs/guides/access/ssh#what-the-node-needs) and [tunnels](/docs/guides/access/tunnels#the-node-half).

### Instances [#instances]

The `instances` section is the whole set of VMs and containers a VPS host should run. bcd's converger (on node
agents, over the runtimes the build provides) follows the FakeDaemon's `FakeHypervisor` rules:

* A domain bcd created that no spec names is removed, and so is one whose desired state is `DELETED`. A domain bcd
  did not create is never touched, only reported, so the control plane can flag it as a leak.

* User data is never in the spec: bcd fetches it with a `SecretRequest` bound to the instance's allocation before it
  defines the domain. A refusal fails that instance; an unavailable broker keeps the section `PROGRESSING`.

* Desired states are `RUNNING`, `PAUSED` (a resume is not a boot) and `STOPPED`. After a host reboot the runtime's
  domains are stopped and the next pass starts the ones that should run.

* An unknown runtime, a bad image, refused user data or a start the runtime refuses for lack of capacity fails only
  that instance (`ERROR` with its message); the section is then `FAILED` with reason `InstanceFailed`. A start that
  bcd's own host admission refuses is not a failure (below); the `FakeHypervisor`, whose only capacity check is its
  vCPU and memory totals, reports a start beyond them that way since 6922553.

* **Placement** (a184ec2). `InstanceSpec.placement` (from a flavor's `placement`, chosen by vps; see
  [flavors](/docs/guides/vps/flavors#placement)) names host CPUs (`cpus`, a cpulist of at least as many CPUs as
  vCPUs), NUMA nodes for the memory (`mems`), a huge page size (2 MiB or 1 GiB, with the memory in whole pages) and a
  balloon (never with huge pages). bcd checks it before anything is created, and a placement the host cannot apply
  fails the instance. `libvirt-kvm` maps it to a vCPU `cpuset` (floating) or `vcpupin` (exactly one CPU per vCPU),
  `numatune` in strict mode, locked `memoryBacking` huge pages and an explicit `memballoon` (none unless asked);
  `cloud-hypervisor` to `cpus.affinity`, `memory.hugepages` and a balloon with free page reporting (see
  [memory ballooning](#memory-ballooning)), with
  `AllowedCPUs` and `AllowedMemoryNodes` on the VMM's unit, whose `MemoryMax` drops to the VMM's overhead when the
  guest's memory is in huge pages. A restore keeps bcd's pinning, never the snapshot's.

* **Host-side admission** (0dd122a). Before a runtime starts a domain that is not running (a new one, a stopped one;
  a resume from `PAUSED` is not a start), bcd checks that the host can give it its memory, whatever the control plane
  decided, so a drifted control plane never overcommits a node. It counts what is committed, not what is used, since
  a VM that has not touched its memory yet will:

  * normal memory: `MemTotal` less the huge page pool (`Hugetlb`) and 1 GiB of headroom for the host, against the
    memory plus VMM overhead (1/32 of the memory plus 256 MiB) of every running or paused domain bcd manages on
    normal pages, of the bound holds' memory limits (since 31555bf), and of the starts admitted earlier in the same
    pass;
  * huge pages, for an instance whose `placement` asks for them: free pages of that size on the placement's NUMA
    nodes, and since aeaa982 also no more than the host-wide count (free less reserved, less what takers without a
    NUMA node need), or free less reserved pages host-wide without one; its VMM overhead still counts against
    normal memory.

  A start that does not fit stays stopped, and the instance's status message reads `ResourcesExhausted: … MiB of
  memory needed, … MiB left on this host` (or the huge pages needed and free). It is not `ERROR`, so vps does not fail
  the instance or its sandbox on the spot; the section stays `PROGRESSING` and the start is tried again on the next
  pass. vps records the refusal as the instance condition `HostResourcesExhausted` and, since 564f543, once it has
  stood a minute, places an instance with nothing of its tenant's on the host yet (a new one, or a warm one no
  sandbox has seen run) on another host, dropping it from this host's spec; one that already ran here, or whose
  volumes live here, keeps waiting for this host (see
  [hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance)). A
  runtime that checks capacity itself (the fakehw runtime) is left alone, and a host whose `/proc/meminfo` cannot be
  read starts without the check, with a warning.

* On a host with a runtime, every status report lists everything the runtimes hold in `StatusReport.instances`,
  with `instances_complete`, and since b8e7271 the volume store's space in `StatusReport.volume_storage` (see
  [persistent volumes](#persistent-volumes)).

* `restore_from` boots a new instance from a snapshot instead of an image. bcd downloads it, over https only since a
  snapshot may hold a tenant's memory, to `data_dir/snapshots/<sha256>` and verifies it first; an `http://` URL, a
  missing digest, a mismatch, another runtime, or a memory snapshot from another CPU model fails that instance. The
  download runs in the background: the converger waits on it as on user data (the instance is reported waiting "for
  the user data or snapshot"), goes on deleting domains and loading firewall rules meanwhile, and is triggered again
  when it ends; a failed download is reported for a minute before it is tried again (6827df8). A downloaded snapshot
  is deleted once no spec restores from it (it goes with its instance), on every pass of the converger. The
  `instance.snapshot` command writes one: the runtime captures the disk (and with `include_memory` the memory state)
  into a staging file under `data_dir/snapshots` (the data disk, never a tmpfs), and bcd hashes it and uploads it to
  a presigned https URL. Uploads a crash left behind are deleted too. Both transfers follow redirects only to https,
  and end when a 2-minute window moves less than 1 MiB, or after 6 hours in all, so a stalled or trickling peer
  cannot pin a bulk worker or keep a restore alive.

* Before any domain starts, the instances' `network_policy` becomes host nftables rules on their taps (one
  `bcd_instances` table, replaced atomically): `PRIVATE` keeps a NIC to its IPAM subnets and the host, `NONE` to the
  host only, `EGRESS` adds nothing. A policy that cannot be expressed (a NIC without IPAM addresses or a valid MAC),
  or rules that fail to load, fail the instance and stop it if it runs. Taps are matched by the name every runtime
  gives them (`runtime.TapName`), so the rules exist before the tap does. Every instance tap (`bc*`), guarded or not, drops DHCP server replies (UDP source port 67 or 547), router
  advertisements and redirects, and may never send from or claim (ARP, neighbour adverts) the host's own addresses
  on the bridges its instances attach to. Whatever the policy, every tap whose NIC has a MAC and IPAM addresses is
  also anti-spoofed in the same table: frames must carry the NIC's MAC, IP packets its own addresses (plus `0.0.0.0`
  and `::` for DHCP, and its own EUI-64 link-local address, derived from the MAC, for neighbour discovery; guests
  that use stable-privacy link-local addresses lose IPv6 neighbour discovery), ARP its own IPv4 addresses; each
  guarded address, the link-local one included, is bound to its tap, so no other port may send from it or claim it
  (ARP, neighbour adverts), and a deleted instance keeps its guard until its domain is gone. The metadata service
  and instance tunnels, which name an instance by its address, trust only addresses the loaded rules bind, never one
  two instances claim. The same transaction loads backstop tables (933c51b, completed in b12c217): `inet bcd_taps`
  with `iifname "bc*" drop` on the input and forward hooks and `oifname "bc*" drop` on forward and output, and
  `arp bcd_taps` with the same drops on ARP input and output, which the inet hooks never see. A tap on its bridge
  reaches those hooks as the bridge, so only a tap off its bridge, which the bridge table cannot see, is dropped
  there. A NIC on a bridge whose name starts with `bc` (it would match every tap rule) fails its instance. Every 5
  seconds bcd lists the tables (`nft -j list table bridge bcd_instances`, then `inet` and `arp bcd_taps`) and counts their rules against the
  ruleset it loaded, so a reload, `flush ruleset`, or a flushed table or chain ends that trust at
  once; bcd then loads the last ruleset again itself, outside any converge pass, and if that fails the next pass
  retries (6827df8). A host whose nft cannot load the table keeps egress instances running, logs a warning and
  retries on the next pass. bcd advertises the `Hello` capability `instance.network-policy` on
  hosts with a runtime, which the gateway writes as the node label `bc.dev/cap.instance.network-policy=1`; vps
  places `private` and `none` instances only there.

* An instance with `ssh_from_host_only` (vps sets it on every sandbox instance) takes tcp/22 only from the host
  (474d51f, security review 13 L5), whatever its policy. For each of its taps the same table drops tcp/22 forwarded
  from any other bridge port (`oifname <tap> tcp dport 22 drop`) and tcp/22 the host sends into the tap unless its
  source is one of the host's own addresses on the instances' bridges, where the tunnel dial comes from; a family
  with no host address has its tcp/22 dropped outright. Both rules sit ahead of every accept, so an exec certificate
  is usable only through a tunnel, which the control plane closes on revocation, and no guest-side KRL is needed.

* **Capabilities.** On hosts with a runtime, bcd's `Hello` advertises what it enforces, and the gateway writes each as
  a `bc.dev/cap.<name>=1` label that vps places on (see [capabilities](/docs/daemon/sessions#hello)):
  `instance.network-policy` (`private` and `none` instances), `instance.ssh-host-only` (sandbox and warm instances),
  `instance.placement` (flavors with a `placement`) and `instance.volumes` (instances that attach volumes), the last
  three since 82f4f51. An older bcd would drop the field and run the guest without it.

* The rules match taps by name, so a bcx runtime plugin is told each NIC's tap name (`runtime.TapName`) in its spec
  (b8a1801). When the rules guard an instance (a `PRIVATE` or `NONE` policy, or SSH from the host only) and a running
  instance of a plugin runtime has no interface under one of those names, bcd stops it and fails it with `runtime <id>
  did not create taps …: its network rules cannot apply`: it fails closed rather than run a guest outside the rules.

#### The runtimes [#the-runtimes]

A host build has two built-in runtimes, and runs each one whose tools are on its `PATH`. There is no configuration
key: `virsh` turns on `libvirt-kvm` (`runtime.Libvirt`, 362317c; files under `data_dir/libvirt`), and the
`cloud-hypervisor` binary turns on `cloud-hypervisor` (`runtime.CloudHypervisor`, 2598f0a; files under
`data_dir/cloud-hypervisor`). A host with both runs both. Each instance names its runtime in its spec
(`InstanceSpec.runtime`, which the control plane takes from the pool), and an instance whose runtime this host lacks
fails with `runtime <id> is not installed on this host`. Since 0fca413 the control plane does not send one there:
bcd advertises each runtime it runs in `Hello` (the node label `bc.dev/driver.<runtime>`), and vps places a hosted
instance only on hosts carrying its runtime's label.

| Concern        | `libvirt-kvm`                                                                                                                                                                                                                                                                                                                                                                                                                               | `cloud-hypervisor`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Driven through | `virsh -c qemu:///system` and `qemu-img`                                                                                                                                                                                                                                                                                                                                                                                                    | One cloud-hypervisor VMM per instance, started as a confined systemd transient unit (`systemd-run --unit=bcd-ch-<id>.service --collect --slice=bcd-instances.slice`, so the VM outlives bcd; see [confinement](#the-cloud-hypervisor-unit)) and driven over its REST API on `instances/<id>/api.sock`; `qemu-img` for disks. It boots the firmware at `/usr/share/cloud-hypervisor/CLOUDHV.fd`                                                                                                                                                                                                                                                                                              |
| Disk           | A qcow2 overlay (`instances/<id>/disk.qcow2`) on a base image downloaded once over https only, verified against the image's SHA-256 and [checked](#tenant-images) before it becomes visible, and kept as `images/<sha256>.<format>` (`qcow2`, `raw` or `iso`). An `iso` image boots from a cdrom onto a blank qcow2 disk                                                                                                                    | The same overlay on the same verified and checked base images (it reuses `libvirt-kvm`'s image code over its own directory). Disks carry an explicit `image_type` (`Qcow2` for the overlay, `Raw` for the seed), and `backing_files` is on only for bcd's own overlay, never for a restored disk. An `iso` image is refused ("cloud-hypervisor boots disk images, not an iso")                                                                                                                                                                                                                                                                                                              |
| Growing a disk | `qemu-img resize -f qcow2` while stopped, `virsh blockresize` while running; a smaller size fails the instance                                                                                                                                                                                                                                                                                                                              | `qemu-img resize -f qcow2` while stopped, `vm.resize-disk` while running; a smaller size fails the instance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| User data      | A NoCloud seed ISO (volume `cidata`, `user-data` and `meta-data` with the instance id as `instance-id` and `local-hostname`), built with `genisoimage`, else `xorriso`, attached as a read-only cdrom                                                                                                                                                                                                                                       | The same seed ISO, attached as a read-only disk                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| NICs           | virtio NICs on the named bridges, on taps named `runtime.TapName`, so bcd's nftables rules exist before the tap does                                                                                                                                                                                                                                                                                                                        | Persistent taps named `runtime.TapName` (`ip tuntap add`), put on their bridge and brought up by bcd before the VMM starts (933c51b); since b12c217 bcd opens each tap and hands its fd to the VMM (`vm.add-net`, or `net_fds` on a restore), which never names a tap. A guest reboot reopens the same tap; a failed attach means no VM; a NIC without a network is refused; the taps are deleted when the VM stops or is destroyed                                                                                                                                                                                                                                                         |
| GPUs           | VFIO passthrough (`<hostdev>` PCI, managed: libvirt rebinds the device itself), the PCI address taken from the inventory's GPU bus id; a GPU without one fails the instance                                                                                                                                                                                                                                                                 | VFIO passthrough of `/sys/bus/pci/devices/<address>/`, the address found the same way; a GPU without one fails the instance. Since be4de2a bcd binds every endpoint of the GPU's IOMMU group to `vfio-pci` itself before the start (`driver_override`, unbind, `drivers_probe`; the GPU's audio function too, since VFIO refuses a group with a device on a host driver; bridges stay on `pcieport`) and checks that they moved, so a host without `vfio-pci` loaded fails the start. It records the groups and addresses it handed to the VMM: a stop gives the groups back to root, and a destroy hands the devices back to their host driver, even when the spec dropped a GPU meanwhile |
| CPU            | `host-passthrough`, so a memory snapshot restores only on the same CPU model                                                                                                                                                                                                                                                                                                                                                                | `boot_vcpus` and `max_vcpus` both the instance's vCPUs; a memory snapshot restores only on the same CPU model                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Ownership      | bcd marks each domain it defines in its metadata (namespace `https://better-compute.dev/libvirt/1`: allocation, image digest, addresses, the snapshot it was restored from). Any other domain of the connection is listed as unmanaged, the leak the control plane reports, and is never defined over, stopped or destroyed                                                                                                                 | cloud-hypervisor keeps no registry, so bcd's record is `instances/<id>/vm.json` (allocation, image digest, addresses, MACs, bridges, GPUs and the VM configuration). `List` reports only directories with that record, with the state from `vm.info`; a directory without one is not bcd's and is never defined over or removed                                                                                                                                                                                                                                                                                                                                                             |
| States         | `start` (or `start --paused`), `suspend` and `resume`; a stop is `virsh shutdown`, then `virsh destroy` after 60 seconds; a start that fails with "Cannot allocate memory" is a capacity failure. Delete destroys and undefines the domain and removes its directory                                                                                                                                                                        | `vm.create` and `vm.boot` (then `vm.pause` for a paused start), `vm.pause` and `vm.resume`; a stop is the ACPI power button for up to 60 seconds, then `vm.shutdown`, `vmm.shutdown` and `systemctl stop`, and it fails unless `systemctl show` then reports the unit `inactive` or `failed`, so a VMM that survived is never forgotten; a boot that fails for lack of memory is a capacity failure. Delete stops the VM (and so never removes the directory of a VMM still running) and removes its taps and directory                                                                                                                                                                     |
| Snapshots      | A tar of `meta.json` (runtime, CPU model, `includeMemory`), the overlay flattened onto its base (`qemu-img convert`), the seed and, with memory, `virsh save`'s state, after which the domain carries on from that state with `virsh restore --xml` of bcd's own `domain.xml`, never the definition embedded in the state file, which the helper cannot validate (81f9c21); a disk-only snapshot of a running domain pauses it for the copy | A tar of `meta.json`, the flattened overlay, the seed and, with memory, cloud-hypervisor's own snapshot of the paused VM under `ch/` (`config.json`, `state.json`, `memory-ranges`); a running VM is paused for it and resumed after                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Restores       | Unpacks the tar into the new instance's directory, refusing one taken by another runtime or, with memory, on another CPU model, then [checks](#tenant-images) its disk and seed; a memory restore's first start is `virsh restore` with the new domain's XML                                                                                                                                                                                | The same refusals and checks. A memory snapshot's `config.json` must have exactly the instance's vCPUs (`boot_vcpus` and `max_vcpus`) and memory size, and is rebuilt before its first start, which is `vm.restore` then `vm.resume`: from the snapshot it keeps only CPU, memory and platform keys (`cpus`, `memory`, `platform` allowlists), bcd supplies the payload, disks, NICs, serial, console, rng and devices, and anything else (`fs`, `pmem`, `vsock`, memory zones, CPU affinity, a debug console, landlock rules) is dropped                                                                                                                                                   |
| Console        | The pty libvirt gave the domain's serial port, for [instance console tunnels](/docs/guides/access/tunnels#the-node-half)                                                                                                                                                                                                                                                                                                                    | The serial port's unix socket, `instances/<id>/serial.sock`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

A snapshot is tied to the runtime that took it: `meta.json` names it, and the other runtime refuses the tar. Both
runtimes refuse a memory snapshot of an instance with passthrough GPUs up front, since the GPU's state lives in the
device and no VMM saves it (`an instance with passthrough GPUs cannot be snapshotted with its memory; snapshot its
disk only`, be4de2a), and of an instance with a read-write volume (`ErrMemorySnapshotWithVolumes`, 550da91); a
disk-only snapshot of such an instance still works. vps refuses both before a command is sent (see
[snapshots](/docs/guides/vps/sandboxes#take-one)).

**QEMU reaches its files** (f7b8805). libvirt runs QEMU as its own user (`libvirt-qemu` or `qemu`), and its dynamic
ownership relabels a domain's files but never the directories on their way, while bcd's are 0700 root. Before it
defines a domain, bcd therefore adds `o+x` (search, never read) to each directory from `data_dir` down to the disks,
seed and volumes the domain names and to the overlay's base image directory. Listing stays closed, and bcd's own
files there are 0600 (identity, keys, journal) or relabelled by libvirt.

**Under bcd helper** (ed3c7d2, d418820). An unprivileged `bcd node` is not in the root-equivalent `libvirt` group.
Its `libvirt-kvm` runtime sends `define`, `start`, `shutdown`, `destroy`, `suspend`, `resume`, `undefine`,
`blockresize`, the rebind disk's `attach-disk` and `detach-disk`, `save`, `restore` and the console to the helper's
`libvirt.*` operations, where root checks every domain definition default-deny (disks only under bcd's directories
and re-checked as root, taps named for the instance on `[helper] instance_bridges`, hostdevs only of the host's GPUs)
and hands the console back as a file descriptor; the balloon's `setmem` goes there too. Since b6df6f1 (security
review 2) `libvirt.restore` refuses a request without a definition before anything runs, since `virsh restore` alone
starts the domain the node-written state file embeds: the definition is checked like a `define` (and must be the
requested domain's), staged in root's directory and always passed as `--xml`. Since 3fcd1a1 the helper also checks an
overlay's base image as root: the node's own plain image, naming no backing or data file, named by its SHA-256.
Reads (`list`, `domstate`,
`dumpxml`, `domstats`) use libvirt's read-only socket (`virsh -r`). libvirt runs with
`dynamic_ownership = 0`, so instead of the relabelling and `o+x` above, QEMU's user is in the node's group: disks are
0660 (read-only ones 0640) and the directories on the way get group search. See
[libvirt through the helper](/docs/daemon#libvirt-through-the-helper).

**cloud-hypervisor under bcd helper** (d86fcbe, 7fe74a1, 94843ec). Until 7fe74a1 an unprivileged node dropped
`cloud-hypervisor` (its VMMs need root: the `systemd-run` units, the taps, files handed to the VMM's user) and did
not advertise it. Now it runs there too, with its root-only steps behind the helper's `ch.*` operations
(`runtime.CHPrivileged`):

| Step         | As root                                                       | Under bcd helper                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Files        | bcd chowns the instance's files to the VMM's user for the run | The node keeps owning every file and grants the VMM's user POSIX ACL entries on the instance directory and what is in it, inherited by what the VMM creates there; the unit's `UMask` is `0007` instead of `0077` (its group is the VMM's own), so those entries stay effective. Volumes get a read-write or read-only entry for the run, removed at the stop                                                                                                                                                                                                                                                                                                      |
| Taps         | `ip tuntap add` by bcd                                        | `ch.taps`: root creates exactly `runtime.TapName(instance, i)`, owned by the node (which opens it and hands the VMM the fd, as above), on a bridge of `[helper] instance_bridges`, and refuses an existing interface of that name that is not a tap. `ch.untaps` deletes only those names                                                                                                                                                                                                                                                                                                                                                                          |
| GPUs         | bcd binds each IOMMU group to `vfio-pci`                      | `ch.vfio`: root takes the GPUs from the instance's record, only GPUs of this host that root does not see holding work and whose `/dev/vfio` group no other running VMM unit's user owns (94843ec), and only when every device of the group is one of this host's GPUs, a function of one, or a PCI bridge; it binds them to `vfio-pci` and gives `/dev/vfio/<group>` (0600) to the VMM's user. `ch.unvfio` gives them back to root and unbinds them, never a group a running VMM unit's user owns; `ch.stop` gives back exactly the groups the stopped unit's user owned. Root decides who holds a group from `/dev/vfio` itself, never from a node-written record |
| The VMM unit | `systemd-run` with bcd's arguments                            | `ch.launch`: root reads and checks the record itself (`runtime.CHRecord`: `vm.json` and the disk and seed the node's regular files, no symlink in the instance directory, the base image `images/<64 hex>.<qcow2 or raw>` and the overlay re-checked, volumes in the volume store, GPUs of this host, the VMM user in bcd's range and not another running unit's, vCPU and memory bounds), then builds the same confined unit (`runtime.CHLaunch`) with root's own `cloud-hypervisor` binary, under a lock. No command line of the node's reaches root. `ch.stop` stops the unit, answers its state and takes the VFIO devices back                                |

The node still drives each VMM over its API socket; what it asks there (`vm.create`'s disks and firmware,
`vm.add-disk`) the VMM reaches only as its own user, in its mount namespace, through its ACL entries. Every `ch.*`
call names the instance by the domain-name pattern and is logged with it. `bcd doctor` checks
`[helper] instance_bridges`: without it, it warns that the helper refuses every instance tap. Like the rest of
`cloud-hypervisor`, this path is tested against fake tools only (the runtime's `TestCloudHypervisorUnderTheHelper`
and `TestCHRecordRefusals`, the helper's `ch.*` tests), not a real VMM under the helper.

#### Memory ballooning [#memory-ballooning]

A guest whose `InstanceSpec.placement` asks for a balloon lends the host the memory it does not use (12e9ced, PLAN.md
§3.8). vps asks for one on every guest of a pool that overcommits memory, except huge-page-backed and GPU guests (see
[flavors](/docs/guides/vps/flavors#placement)), and bcd never balloons a guest with huge pages. Every 30 seconds (the
health interval) bcd reads the host's `MemTotal` and `MemAvailable` and the memory pressure stall (PSI `some avg10`),
and resizes the balloons of its running, managed guests that have one:

| Host state                                                          | What bcd does                                                                                                                                                                                                      |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Less than 10% of memory available, or a memory stall of 10% or more | Inflates balloons to cover the shortfall below 10% (at least 2% of host memory while stalled), taking first from the guests with the most memory above their floor, at most a quarter of a guest's memory per pass |
| More than 20% available and no stall                                | Deflates every inflated balloon a step (a quarter of the guest's memory) at a time, within what is free above 20%                                                                                                  |
| In between                                                          | Leaves the balloons as they are                                                                                                                                                                                    |

A guest never goes below its floor: `[instances] balloon_floor_percent` of its memory (10 to 100, default 50), or
what it uses plus 10% of its memory when that is higher. Its use comes from the runtime's usage reading (see
[node metrics](/docs/operations/observability#node-metrics-from-bcd)); a guest without one counts as using all its
memory, so nothing is taken from it. `libvirt-kvm` sets the size with `virsh setmem --live` (through the helper's
`libvirt.setmem` under `bcd helper`) and reads it from `domstats --balloon`; `cloud-hypervisor` with `vm.resize`'s
`desired_balloon` and `vm.info`'s balloon size. A guest defined without a balloon device is never touched (the runtime
answers `ErrNoBalloon`, e48f40e), and a memory-restored `cloud-hypervisor` VM keeps bcd's balloon, never the one saved
in the tenant's snapshot (9dddde4). Host admission still charges each VM its full configured memory, so ballooning
never lets bcd start more guests than fit. The OTLP export reports what each running ballooned guest lends the host
(its memory less what the balloon leaves it) as `bcd.instance.memory.balloon.reclaimed`, only for guests whose spec
asks for a balloon (9bdd43a; it was `bcd.instance.memory.balloon`, reported for every libvirt domain). The
policy and the controller are tested against fake runtimes (`internal/agent` `balloon_test.go`,
`balloon_agent_test.go`), and `tests/e2e/bcd-placement.test.ts` inflates and deflates one through a fakehw bcd under
simulated memory pressure (427787a); no balloon has run on a real host.

#### Persistent volumes [#persistent-volumes]

A host keeps the volumes its `instances` section lists (`InstancesSection.volumes`, 014bf0d; bcd 550da91), the
disks behind [sandbox volumes](/docs/guides/vps/sandboxes#volumes), across the instances that attach them:

* **The store.** Each volume is a sparse raw file, `<data_dir>/volumes/<id>.raw` (0600), with its serial beside it
  in `<id>.json`. Raw rather than qcow2: its format is fixed and never probed, it has no header a VMM could rewrite,
  and it grows by truncation. An id is 1 to 80 of `[A-Za-z0-9_-]`, a serial 1 to 20 of `[a-z0-9-]`, and a size a
  positive multiple of 512 bytes.
* **Created and grown, never shrunk.** bcd creates missing volumes and grows those whose size grew before any
  instance is defined; a smaller size fails. A volume that grew is resized live in every running or paused guest that
  attaches it (`virsh blockresize` on `libvirt-kvm`, `vm.resize-disk` on `cloud-hypervisor`; b5415a2).
* **Attached by serial.** `InstanceSpec.volumes` attaches them in order as `vdb` onward after the root disk: raw
  virtio disks on `libvirt-kvm` (read-only when asked), `Raw` disks with their serial on `cloud-hypervisor` (bound
  into the VMM's unit; a read-write one handed to the VMM's user while it runs), and path and serial for a bcx
  runtime. The guest finds each at `/dev/disk/by-id/virtio-<serial>`.
* **One writer.** An instance that attaches a volume beside a running or paused writer, or as a writer beside
  readers, is refused; the first in spec order keeps it. Since 448dc5f a pass applies stops and deletes before starts,
  so a volume handed from one instance to another in the same pass is let go before it is taken. Since 6b1480b bcd
  tracks which instances have which volumes open, from the running domains and as the pass stops, deletes and starts
  them, and checks every start (and a bcx runtime's live re-apply) against that, so a stop that failed never lets a
  second writer in. Spec-level refusals (a shrink, a volume the section does not list, a pending attach) no longer
  stop a running VM; only a second holder of an open volume makes one let go.
* **Destroyed when unlisted.** A volume the section no longer lists and no instance attaches is destroyed after the
  pass: its blocks are punched out, then the file is unlinked. A volume a running VM still has open is never destroyed
  (39458f9), and one whose VM is stopped or deleted in the same pass goes in that pass (408e002).
* **No memory snapshots beside a writer**, as above.
* **Sparse, but never overcommitted** (851016e). A volume's unwritten space is promised, not taken, so a tenant
  filling its volume could otherwise leave the other volumes, the instances' overlays and bcd's own journal without
  space. A new volume, or a growth, must fit in the data directory filesystem's available space (`statfs` `Bavail`)
  less a reserve (5% of the filesystem, at least 2 GiB) less every volume's unwritten space. One that does not is
  refused, keeps its size, and the instances that attach it report why (`volume <id>: … MiB needed, … MiB can still be
  promised to volumes on this host (… MiB free, … MiB reserved, … MiB promised to volumes' unwritten space)`).
* **Space reported** (37d6d6d, b8e7271). Every status report of a host with a runtime carries
  `StatusReport.volume_storage`:

  | Field               | Value                                              |
  | ------------------- | -------------------------------------------------- |
  | `total_bytes`       | The data directory's filesystem                    |
  | `available_bytes`   | Its `statfs` `Bavail`                              |
  | `reserved_bytes`    | Kept from volumes: 5% of the total, at least 2 GiB |
  | `committed_bytes`   | The sum of every volume's size                     |
  | `outstanding_bytes` | The sum of every volume's unwritten (sparse) space |

  Since 131f5ee vps keeps each host's last report (`host_volume_storage`) and books a volume, or a growth, only on a
  host whose store can hold it by the same rule, less what it booked there that bcd has not reported yet; a host
  that has not reported takes no volume (see [volumes](/docs/guides/vps/sandboxes#volumes)).
* **Closed again once detached** (bdeb452). On `cloud-hypervisor` a read-write volume is handed to the running VMM's
  user (0600) and given back to root when it stops. A read-only volume gets a POSIX ACL entry (`user:<uid>:r`) for
  each VMM user that has it open, removed when that VMM stops, so it is readable by exactly those VMMs rather than by
  a group; the last reader gone, it is root's 0600 without an ACL. On a filesystem without ACLs it falls back to
  group `kvm` and 0640, reverted once no other instance of the runtime was given it. Under `bcd helper`, a stopped or
  destroyed `libvirt-kvm` domain's volumes go back to 0600 unless another running domain still has them open, and a
  `cloud-hypervisor` volume stays the node's: the VMM's user gets an ACL entry, read-write or read-only, for the run,
  removed at the stop (d86fcbe).

A bcd that keeps volumes advertises `instance.volumes`, and only such hosts receive instances with volumes.

#### Tenant images [#tenant-images]

A tenant's image is untrusted input to a hypervisor. Security review 12 (fixed in 933c51b) found that a qcow2 image
could name a backing file, such as the host's disk or another instance's, and the hypervisor would open it. Its
verification found that probing the image with `qemu-img info` was itself a risk, since `qemu-img` may follow what an
image names, so since b12c217 no tool inspects a tenant image: bcd reads a qcow2 header and its header extensions in
Go (`runtime/qcow2.go`), which opens nothing else. A `qcow2` base image is checked when it is downloaded, before it is
renamed into `images/`, and again every time an instance uses it, and it is refused, failing the instance, when:

* it is not a qcow2 (version 2 or 3) at all, or its header is truncated or malformed;
* it names a backing file;
* it names an external data file (incompatible feature bit 2, or the data-file header extension), or sets an
  incompatible feature bit bcd does not know;
* it is encrypted.

`raw` and `iso` images are never probed: every tool opens them with their format stated, so their content is only
data. Base images are mode 0644 in a 0700 `images/` directory, so an unprivileged VMM can read only the one bound into
its unit.

An instance's own disk is checked too, because a VMM that once ran it can rewrite its header (`checkOverlay`):
`disk.qcow2` must name exactly the recorded base image and format as its backing file, or none for a restored
snapshot or an iso install. `cloud-hypervisor` checks it before every start, snapshot and resize; `libvirt-kvm`
before every start, snapshot and grow, with the base format now recorded in the domain's metadata. A restored
snapshot's disk must be such a flat qcow2, and its seed is attached as raw, never probed; a failed check removes the
new instance's directory. `qemu-img create -u` builds an overlay without opening the base, and `libvirt-kvm`'s grow
reads the virtual size from the header.

#### The cloud-hypervisor unit [#the-cloud-hypervisor-unit]

Each VMM runs in its own transient unit `bcd-ch-<id>.service` in `bcd-instances.slice`, as its instance's own
unprivileged user (b12c217): a uid from `0x7FFD0000` up, kept in `vm.json`, with `User=` and `Group=` set to it. The
unit name always ends in `.service`, so an instance id never names another kind of unit. The VMM needs no network
privilege, because it never names a tap: bcd creates and bridges the persistent taps, opens each one (`TUNSETIFF`)
and hands the file descriptors to the VMM over `SCM_RIGHTS`. `vm.create` carries no NICs, `vm.add-net` adds each NIC
with its fd, a restore passes them as `net_fds`, and the saved configuration's NICs carry no tap name. The instance's
files and its GPUs' `/dev/vfio` groups are chowned to the VMM's uid at start, and the groups are given back to root
at stop; under `bcd helper` the node grants ACL entries instead and root hands over the groups (see
[cloud-hypervisor under bcd helper](#the-runtimes)). `systemd-run` gets these properties:

| Property                | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identity and privileges | `User=` and `Group=` the instance's uid, `CapabilityBoundingSet=` and `AmbientCapabilities=` empty, `NoNewPrivileges=yes`, `LimitMEMLOCK=infinity`, `UMask=0077` (`0007` under `bcd helper`, so the inherited ACL entries stay effective)                                                                                                                                                                                                                                                                                                                                                                                   |
| Devices                 | `DevicePolicy=closed` with `DeviceAllow=/dev/kvm rw`; with GPUs also `/dev/vfio/vfio rw` and `char-vfio rw`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Files                   | `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `ProtectProc=invisible`; read-only `TemporaryFileSystem`s over `/run` and `/var` (hiding systemd's, D-Bus's and every daemon's sockets, bcd's data, libvirt and other instances), and over bcd's `instances/` and `images/` when its directory is outside `/var`; `InaccessiblePaths` for `/etc/bcd`, `/etc/ssh`, `/etc/libvirt`, `/etc/shadow` and `/etc/gshadow`; only the instance's own directory bound read-write (`BindPaths`, `ReadWritePaths`) and its base image read-only (`BindReadOnlyPaths`), so a disk naming another file finds nothing to open |
| Kernel and system       | `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictSUIDSGID`, `RestrictRealtime`, `RestrictNamespaces`, `LockPersonality`, and `SystemCallFilter=~@debug @mount @reboot @swap @module @raw-io @clock @cpu-emulation @obsolete`                                                                                                                                                                                                                                                                                                       |
| Network                 | `PrivateNetwork=yes`, `RestrictAddressFamilies=AF_UNIX`, `IPAddressDeny=any`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Limits from the flavor  | `MemoryMax` = memory + memory/32 + 256 MiB, `TasksMax` = 4 per vCPU + 128, `CPUQuota` = (vCPUs + 1) x 100%; a restored memory configuration drops hugepages, which `MemoryMax` would not charge                                                                                                                                                                                                                                                                                                                                                                                                                             |

This needs systemd 247 or later and a cloud-hypervisor whose API takes NIC file descriptors.

> **Tested against fakes only.** 
  `libvirt-kvm`'s tests (`daemon/internal/runtime/libvirt_test.go`) run it against a fake `virsh` and `qemu-img`:
  the domain XML, metadata, states, grow and shrink, snapshot and clone with memory, a CPU mismatch, destroy, a
  mismatched image digest, and the image checks against real qcow2 headers (the fake `qemu-img` writes them since
  b12c217): `TestBaseImageChecks` (a backing file, a data file, a `qcow2` that is not one, an image cached before the
  check) and `TestCheckOverlay` (an overlay naming another backing file or format). `cloud-hypervisor`'s tests
  (`cloudhypervisor_test.go`) run it against a fake VMM API and fake `systemd-run`, `qemu-img`, `ip` and seed tools:
  define, the states, a memory snapshot, a restore into a clone with its paths rewritten, a memory snapshot of a larger
  VM refused, the rebuilt restore configuration (`TestRestoredConfig`), taps bridged before the VMM exists, the unit's
  uid and confinement properties, NICs added and restored with fds, the chowns and the VFIO group handed back, a
  rewritten overlay that will not start, a VMM that survives its stop, stop and destroy, a directory that is not bcd's
  left alone, and the identity rebind. Neither has run on a real KVM host, and no end-to-end test drives either
  through the control plane.

fakehw builds run a file-backed runtime named `libvirt-kvm` instead, whose domains outlive the process and honor vCPU
and memory limits; `tests/e2e/bcd-vps.test.ts` drives that one through the control plane. On `bcd node`, a bcx plugin
whose driver lists the kind `runtime` is also a runtime (`list`, `apply`, `remove` over `Invoke`; no snapshots). A
host with neither tool and no runtime plugin runs no instances: a non-empty section fails with `this host runs no
instance runtime`, and `bcd doctor` warns ("neither virsh nor cloud-hypervisor: this host runs no VPS instances").

#### Identity rebind [#identity-rebind]

A memory restore resumes its source guest with the source's sshd principals and host key, so an exec certificate
for one restore would open the source and its siblings (security review 13 H2). When vps gives a memory restore its
own identity, its spec carries `InstanceSpec.identity_rebind_ref`, a sealed bundle
`{"v":1,"account","principals","hostKey"?,"hostCertificate"?}`, and bcd installs it into the running guest
(`agent/rebind.go`, `runtime/rebind.go`; ca08d5c):

1. Once the instance runs, the agent fetches the bundle with a `SecretRequest` bound to the instance's allocation,
   as for user data, and checks it (`runtime.ParseRebind`: version 1, the account name, principal lines, and a PEM
   host key with an OpenSSH certificate line, or neither). It works off the converge pass, since the guest may take a
   while.
2. The runtime writes a raw 1 MiB disk holding a tar at offset 0 (`nonce`, `account`, `principals` and, with a host
   key, `host_key` and `host_key-cert.pub`), reserves its last 4 KiB for the guest's answer, and hot-adds it with
   virtio serial `bc-rebind`: `vm.add-disk` on `cloud-hypervisor` (the file chowned to the VMM's user), `virsh
   attach-disk … vdz --live` on `libvirt-kvm`.
3. The guest's udev rule starts `bc-rebind@<dev>.service`, whose script installs the principals and host key under
   `/etc/ssh`, reloads sshd and writes `BCREBOUND <nonce>` at the start of the last 4 KiB.
4. bcd waits up to 2 minutes for that line, then removes the disk (`vm.remove-device`, `virsh detach-disk vdz --live`) and zeroes and deletes its file either way, so the secret never stays on the host.
5. On success it records the reference in `data_dir/instances-rebound.json` and reports it as
   `InstanceStatus.identity_rebound_ref`, from which vps re-points exec at the restore alone. Instances no spec names
   any more are pruned from the file.

A guest that does not answer is tried at most 3 times per reference, 5 minutes apart; a bundle the control plane
refuses to release, or a malformed one, is not retried. Nothing is reported then, so the control plane keeps
refusing exec while the identity is shared: it fails closed. The guest half (the udev rule, the unit and the script)
is written in `daemon/README.md`, and vps's sandbox user data installs it verbatim at first boot (3e25c78), so
restores of sandboxes booted since then answer. A bcx runtime plugin is not a rebinder: since b8a1801 its spec
carries `identityRebindRef`, `sshFromHostOnly`, `placement` and each NIC's tap name, but bcd neither rebinds through
it nor reports `identity_rebound_ref` for it (and plugins take no snapshots, so they restore none). The Fake runtime
acts as a compliant guest. Tests: `TestParseRebind`, `TestRebindImage`, the
`cloud-hypervisor` and `libvirt-kvm` rebinds against an acknowledging and a silent fake guest, and
`TestIdentityRebind`, which drives the converger through the secret fetch, the status report and a restart. See
[a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity).

## Commands [#commands]

### The command body [#the-command-body]

A `Command` is the serialized `CommandBody`, an optional signature, and the delivery `attempt` (informational,
outside the signature). The control plane stores the body and signature bytes when it enqueues the command and
the gateway resends exactly those bytes.

| Field                    | Rule                                                                                                                      |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `id`                     | A ULID; the journal's deduplication key                                                                                   |
| `node_id`                | Must be the session's node, or the command is `UNSUPPORTED`                                                               |
| `seq`                    | Per-node, from `nodes.next_cmd_seq` in the enqueue transaction; replay order                                              |
| `kind`, `payload`        | A registered kind and its request message, packed as `Any`                                                                |
| `lane`                   | `control`, `provision` or `bulk`                                                                                          |
| `not_after`              | Mandatory, in server time: after it the command must not start                                                            |
| `min_generation`         | The spec generation the command presupposes; 0 for none                                                                   |
| `retry_safe`             | May run again when a crash interrupted it before it finished                                                              |
| `idempotency_key`        | Unique per node; workflow steps use `wf:<id>:<stepId>`                                                                    |
| `self_terminating`       | The side effect ends the daemon process                                                                                   |
| `attested`               | The result must be node-signed (verified from v0.2)                                                                       |
| `workflow_id`, `step_id` | The workflow step this command executes; secrets bind to it (v0.2)                                                        |
| `traceparent`            | W3C trace context, from the api through the worker and gateway to the daemon and driver                                   |
| `batch_id`               | The destructive batch the command belongs to (v1.0 quorum), set by `enqueue({ batchId })`; empty for an unbatched command |

The gateway also guards the stored body before sending it: a body that does not decode, lacks `not_after`, or names
another command or node is cancelled with reason `invalid-body` instead of sent.

### Dispatch [#dispatch]

Enqueue inserts the command `pending` in the caller's transaction and, after commit, pokes the node's gateway.
Each gateway dispatches on a poke or every second, joining commands to the sessions it holds:

```sql
SELECT c.*, s.session_epoch FROM commands c JOIN node_sessions s USING (node_id)
 WHERE s.gateway_id = $me AND c.state = 'pending'
   AND c.not_before <= compute_now() AND c.not_after > compute_now()
 ORDER BY c.node_id, c.seq LIMIT 1000
 FOR UPDATE OF c SKIP LOCKED FOR SHARE OF s
```

A row is marked `dispatched` with the session's epoch only when that epoch equals the gateway's in-memory epoch for
the node and the lane has credit. Order within a lane is kept: once a lane is out of credit, later commands of that
lane wait. Some commands are cancelled at this point instead of sent: a pending command whose cancel was
requested, a kind outside the certificate's `allowed_kinds` (`kind-not-allowed`), and a destructive kind for a
tenant-controlled node (`tenant-controlled`).

### Acceptance checks [#acceptance-checks]

A delivered command first goes through the journal's duplicate rules (below). A new command, or a retry-safe one a
crash interrupted, then passes these checks in order:

1. **`UNSUPPORTED`**: the body names another node, or the kind is unknown or not allowed for this agent's role.
   While the session is upgrade-only, every kind but `daemon.upgrade` is refused this way.
2. **`min_generation`** above the applied generation: the daemon sends `SpecRequest` and defers the command. A newer
   spec re-runs the checks; still behind at `not_after`, the command is `STALE_GENERATION`.
3. **`EXPIRED`**: past `not_after`, in server time through the `Welcome` offset. A command without `not_after` is
   `EXPIRED`, so the control plane always sets one.
4. **`BAD_SIGNATURE`**: once the trust bundle holds command-signing roots, a missing, unknown-key or invalid
   signature on any command, whatever its kind (0d42de8).
5. **`BATCH_UNAPPROVED`*&#x2A;, &#x2A;*`UNBATCHED_LIMIT`** (v1.0): with approver keys in the trust bundle, a
   destructive command whose batch approval or scope proof does not admit it, or an unbatched one beyond the hourly
   allowance. See [the quorum](#signed-commands-and-attested-results).
6. **`BUSY`**: no free credit on the lane, or the journal is full. Not terminal: see
   [BUSY](/docs/daemon/sessions#busy).

`UNSUPPORTED`, `EXPIRED`, `STALE_GENERATION`, `BAD_SIGNATURE` and the two quorum reasons are terminal: the command becomes `rejected`, and
the rejection is journaled, so a duplicate delivery or a `Hello` replay gets the same answer. `BUSY` is never
journaled. bcd (`dispatch.Precheck`, then a lane credit) and the FakeDaemon both follow this order, so a bad
signature is refused even when the lane has no credit.

### Lanes, credits and workers [#lanes-credits-and-workers]

| Lane        | Gateway credits | Daemon workers | For                                                                                           |
| ----------- | --------------- | -------------- | --------------------------------------------------------------------------------------------- |
| `control`   | 8               | 4              | Short, latency-sensitive work: diagnostics, inventory, reboots, resets, BMC operations, kexec |
| `provision` | 4               | 2              | Long provisioning work: disk wipes, image writes, DCGM diagnostics (v0.2)                     |
| `bulk`      | 2               | 1              | Large transfers and slow jobs: log collection, image pre-warm                                 |

Credits bound the commands admitted at once, running or waiting for a worker; workers bound the ones running. Each
lane has a fixed worker pool, never a goroutine per command, and a per-resource lock (a GPU UUID, a disk, an
allocation) keeps two lanes from acting on the same device at once. A command holds its credit from acceptance until
its result is journaled.

### Kinds by lane [#kinds-by-lane]

These are the kinds bcd registers (`daemon/internal/agent/kinds.go` and `site.go`), with the lanes, roles and flags
of `commandKinds` in `@better-compute/protocol`, `metalCommandKinds` in `@better-compute/metal` and
`instanceSnapshotKind` in `@better-compute/vps`, plus `driver.invoke`, which `provisionModule()` registers. A kind
outside the agent's role is `UNSUPPORTED`.

| Lane        | Kind                                       | Roles               | Flags                                                                                                      |
| ----------- | ------------------------------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------- |
| `control`   | `inventory.refresh`                        | node, site, ramdisk | retry-safe                                                                                                 |
| `control`   | `diag.nvml`                                | node, ramdisk       | retry-safe, attested                                                                                       |
| `control`   | `gpu.reset`                                | node                | attested, destructive                                                                                      |
| `control`   | `node.reboot`                              | node, site          | retry-safe, self-terminating, destructive                                                                  |
| `control`   | `bmc.query`                                | site                | retry-safe                                                                                                 |
| `control`   | `bmc.boot`                                 | site                | retry-safe, destructive                                                                                    |
| `control`   | `bmc.power`, `bmc.firmware`, `bmc.storage` | site                | destructive                                                                                                |
| `control`   | `bmc.rotate`                               | site                | attested, destructive                                                                                      |
| `control`   | `site.httpProxy`                           | site                | destructive (since 0d42de8; never to a BMC)                                                                |
| `control`   | `boot.prepareRamdisk`, `boot.cleanUp`      | site                | retry-safe                                                                                                 |
| `control`   | `boot.kexec`                               | ramdisk             | self-terminating                                                                                           |
| `provision` | `disk.wipe`                                | ramdisk             | attested, destructive                                                                                      |
| `provision` | `deploy.image`                             | ramdisk             | destructive                                                                                                |
| `provision` | `diag.dcgm`                                | ramdisk, node       | retry-safe, attested                                                                                       |
| `provision` | `driver.invoke`                            | node, site, ramdisk | destructive; runs in a bcx plugin bcd hosts (see [drivers](/docs/extending/drivers#driver-plugins-in-bcd)) |
| `bulk`      | `logs.collect`                             | node, site, ramdisk | retry-safe                                                                                                 |
| `bulk`      | `instance.snapshot`                        | node                | none (not retry-safe)                                                                                      |

What each metal kind does on the agent is on [BMC drivers](/docs/guides/bare-metal/bmc#redfish-in-bcd-site),
[sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site) and
[deploying](/docs/guides/bare-metal/deploying#what-the-ramdisk-does).

### Results and acknowledgements [#results-and-acknowledgements]

The daemon reports `ACKED` once `accepted` is durable, `RUNNING` when the side effect starts, and one terminal
outcome: `SUCCEEDED`, `FAILED` (with an error code such as `interrupted`, `invalid-payload`, `internal` or a
kind-specific one), `REJECTED` (with a reason) or `CANCELLED`. Each update records the `boot_id` it was produced in.
Reports may skip steps; a lost `ACKED` does not stop a result from landing.

The gateway applies updates through the command state table, scoped to the reporting node, so a node can never move
another node's command, and a stale or repeated update changes nothing. The terminal result and the node's
signature are stored exactly as received. Every terminal update is then acknowledged in `Ack.command_ids`, including
ones stored earlier or unknown to this node, and only then does the daemon stop replaying it. Settled commands
publish `command.settled` and wake the node's controller.

### Timeouts [#timeouts]

The CommandSweeper, a singleton, runs every 5 seconds: a `pending` command past `not_after` becomes `expired`, and a
`dispatched`, `acked` or `running` one past `not_after` plus its kind's timeout becomes `failed` with reason
`timeout`. For self-terminating kinds the timeout covers the rejoin window. The built-in timeouts are 2 minutes for
`diag.nvml` and `inventory.refresh`, 5 for `gpu.reset`, and 15 for `logs.collect` and `node.reboot`; an unregistered
kind gets 15 minutes. Metal registers its own: 1 minute for `site.httpProxy`, 2 for `bmc.query`, `bmc.boot`,
`boot.prepareRamdisk` and `boot.cleanUp`, 5 for `bmc.power` and `bmc.rotate`, 15 for `boot.kexec`, an hour for
`bmc.firmware`, `bmc.storage`, `diag.dcgm` and `deploy.image`, and 6 hours for `disk.wipe`.

### Cancel [#cancel]

A cancel requested before dispatch wins: the command is cancelled and never sent. For an in-flight command the
gateway sends `Cancel{command_id, reason}` once per session, and again after every reconnect, until the daemon
settles it. The daemon stops between actions and never hides a completed side effect: a command still queued or
deferred is reported `CANCELLED`, while one whose side effect has started reports its real outcome. A real outcome
always wins over a late cancel.

### gpu.reset and live holds [#gpureset-and-live-holds]

`gpu.reset` (control lane, destructive, attested, not retry-safe, node role only) resets the GPUs listed in
`gpu_uuids`, after a release or for remediation such as XID 63. bcd checks the list before touching anything:

* an empty list fails `invalid-payload`;
* a GPU that a live CDI hold from the `devices` section names, whole or by one of its MIG instances or slots, or
  that a VM has passed through, fails the whole command `gpu-busy`, naming the hold or the instance (539c4ce), so a
  reset never pulls a GPU from under a running workload; `diag.dcgm` applies the same check from level 2;
* a GPU the inventory does not know comes back `reset: false` with the message `unknown GPU`.

Each remaining GPU is reset with `nvidia-smi --gpu-reset -i <uuid>`; a failure is that GPU's `message`. bcd then
collects a fresh inventory and marks each reset GPU `healthy` unless it shows uncorrectable volatile ECC errors or a
pending or failed row remap. Since cb987e5 a host build reads ECC counters and row remapping from `nvidia-smi -q -x`
when it is installed; without it the inventory carries neither, and every GPU that reset reads `healthy`. A GPU that
reset and reads healthy loses its XID conditions at once instead of keeping them for the rest of the boot (662fb55). The fake
HGX node of the `fakehw` build carries them too. The FakeDaemon's
`fakeGpuReset` (opt in with its `kinds` option) follows the same checks.

### Signed commands and attested results [#signed-commands-and-attested-results]

Two signatures keep a compromised gateway, which can delay, drop or read frames, from forging intent or results:

* **Signed intent.** `Command.signature` is a worker key's signature under purpose `command`; the key chains to the
  command-signing roots of the daemon's trust bundle. The server signs every command when `pki.commandSigning` is
  configured, and sends the matching trust bundle in `JoinResponse` and `Welcome`; without it commands go unsigned,
  as v0.1 allows. Whether a signature is required never depends on a flag the body could carry. bcd enforces
  nothing until its trust bundle holds command-signing roots; from then on every command, whatever its kind, must
  carry a valid signature, or it is `BAD_SIGNATURE` (0d42de8). PLAN.md staged this (destructive kinds from v0.2,
  all kinds from v0.4), but a kind table that exempts non-destructive kinds lets a compromised gateway inject an
  unsigned "harmless" kind that still acts: `site.httpProxy` could POST a Redfish reset or `InsertMedia` to any BMC
  of a site around the quorum. The FakeDaemon verifies every command with `required: true` too (12ccfa1). The
  destructive flag still matters for the quorum below: `gpu.reset`, `node.reboot`, `bmc.power`, `bmc.firmware`,
  `bmc.storage`, `disk.wipe`, `deploy.image`, since 3ff8131 `bmc.boot` (it decides what a machine boots) and
  `bmc.rotate` (it replaces the BMC credentials), and since 0d42de8 `site.httpProxy`. `metalCommandKinds` marks
  the same metal kinds destructive (9c14146, f66e236). `site.httpProxy` also never reaches a BMC: the site agent
  remembers every BMC host it drives through a `bmc.*` command (`data_dir/bmc-hosts.json`) and refuses those hosts,
  by name and by resolved address, any `/redfish` path, and any address in its local `[site] bmc_networks`, with
  `forbidden-target`; a `site.httpProxy` other than a `GET` needs a batch leaf bound to its payload under the
  quorum. And the URL's host must belong
  to the command's `target_node_id` (a2a853d): once the site spec lists proxy targets (`SiteSpec.proxy_targets`,
  which metal fills with the site's agentless machines and their inspected addresses, never a BMC host; 8391d74), an
  IP literal must be one of the node's listed addresses and a name must match a listed name or resolve only to
  listed addresses, else `forbidden-target`.
* **Signed desired state.** Specs have the same envelope under purpose `desired-state`. With
  `pki.commandSigning`, the NodeSpec controller signs each new generation's `DesiredStateBody` with the worker key
  (`CommandSigner.signDesiredState`) in the transaction that bumps the generation, and every bump rewrites both
  columns, so a signature never outlives its generation. The daemon checks the signature with `verifyDesiredState`:
  one is required for a spec that changes the `access` section (v0.2) and, with the v0.4 policy, for every spec; a
  present one is always checked. A refused spec is not applied. The FakeDaemon implements this
  (`intent.signedSpecs`, default `'always'`: with command-signing roots every spec must be signed, because an
  unsigned spec with a high generation would freeze all later ones; `'access'` for a control plane that signs only
  access changes; refusals in `specRejections`). bcd implements the `'always'` rule: once it holds command-signing
  roots, a spec without a valid signature is logged and not applied.
* **Attested results.** For attested kinds (`diag.nvml`, `diag.dcgm`, `gpu.reset`, `disk.wipe`, and `bmc.rotate`
  signed by the site agent) the node signs `CommandUpdate.node_sig` over the command id, the state and the SHA-256 of the result, and allocation
  reports travel as node-signed `AttestedReport`s. Workers verify them against `node_certs` before any state
  transition; the gateway never judges or strips them. With `attestedBindings` (on by default with
  `pki.commandSigning`), a reservation goes active only on the node's own signed `AllocationBound`; see
  [attested bindings](/docs/guides/capacity/reservations#attested-bindings).
* **The destructive-batch quorum (v1.0).** Signed intent does not stop a compromised api or worker, which holds a
  worker key. With `approver_keys` in the trust bundle (`pki.approvers`), a destructive command with a `batch_id`
  runs only when `Command.batch_approval` carries enough signatures of distinct approver keys over a scope whose
  `Command.batch_proof` proves the command's `(subject, kind)`; else it is `BATCH_UNAPPROVED`. Destructive commands
  without a `batch_id` are refused `UNBATCHED_LIMIT` beyond `unbatched_destructive_per_hour` (default 0), counted
  per rolling hour of the host's own clock (bcd keeps the count in `data_dir`, so a restart does not reset it). Each
  batch leaf runs at most once. bcd (e7ddb38, 8b0f50c) and the FakeDaemon enforce it after the signature check. On
  the server, the `quorum` module stores batches and collects approvals, the command queue holds batched commands
  until their batch is approved, and the gateway sends each one with its batch's `batch_approval` and its own leaf's
  `batch_proof` (40be696). The built-in workflows create their batches: metal gives each workflow one of its own
  or runs it under a `metal.bulk` batch (f66e236), and the health reconciler's reboots and GPU resets get a
  node-scoped one (e594a4f). See
  [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10).

## Events [#events]

Everything the daemon observes on its own (allocation reports, failures) is an event, journaled with a sequence
number that increases from 1 per `journal_id`, and sent in `EventBatch`es. The gateway stores the events above the
node's cumulative ack for its current journal, publishes each as `node.event` (attested reports decoded), runs the
report handlers, advances `node_sessions.acked_event_seq`, and answers `Ack{journal_id, event_seq}`. A batch for
another journal id is ignored.

A report whose body names another node is published as `security.node-mismatch` and never reaches a handler; an
undecodable one is `security.invalid-report`. The allocation handlers also ignore a report about an allocation that
is not placed on the reporting node, as they ignore unknown allocations, so one node's session cannot bind, release
or fail another node's allocation. Handlers (`allocationBound`, `allocationReleased`,
`allocationFailed`) run in the ingest transaction under a savepoint: a failing one rolls back its own writes, is
published as `node.report-failed`, and does not stop the stream, since an unacked event would only be replayed into
the same failure.

## The journal [#the-journal]

The journal is the daemon's durable memory. bcd's store is bbolt (`journal.OpenBolt`, `journal.db` under
`data_dir`); the Go package also holds the contract, the command rules and an in-memory store for tests, and the
FakeDaemon mirrors them rule for rule.

| Part         | Holds                                                                                                                       |
| ------------ | --------------------------------------------------------------------------------------------------------------------------- |
| `meta`       | Node id, `journal_id`, the event sequence, the last epoch, the applied generation, the server time mark                     |
| `commands`   | Command id to `accepted`, `running{boot_id}` or `done` with its result; done entries kept 7 days                            |
| `events`     | Unacked events                                                                                                              |
| `spec`       | The last full spec, its hash and its signature                                                                              |
| `certs`      | The certificate chain and the command-signing trust bundle (bcd keeps these in `identity.json` beside the journal)          |
| exec tickets | The `jti` of every exec ticket already used, kept until the ticket expires (at most 10,000; a full set refuses new tickets) |

The server time mark is the highest server time the daemon has seen (on `Welcome` and every heartbeat). It only
moves forward, and a `Welcome` whose `server_time` is more than 5 minutes before it is refused, so a gateway cannot
wind the daemon's clock back into an old worker certificate's window and replay expired signed intent. bcd also
mirrors the mark into `identity.json`, so a lost journal does not reset it.

**Durability.** Every write is fsynced before it returns. `accepted` reaches disk before any side effect, and
self-terminating kinds write `running{boot_id}` immediately before theirs, so a restart can tell the reboot they
caused from a crash.

**Bounds.** 64 MiB and 10,000 unacked events, which `config.toml` may lower but never raise. Events are classed
metric, transition or result. When the journal is full, the oldest metric events are dropped; results and
transitions never are. If that is not enough, the daemon answers new commands `BUSY` until acks drain the backlog.

### Duplicates and crash recovery [#duplicates-and-crash-recovery]

Every daemon process start picks a fresh incarnation id. An entry written by another incarnation belongs to a
process that no longer exists, so there is nothing to re-attach to. A delivered command is resolved against the
journal first (`journal.Resolve`):

| Journal entry                                     | Action                                                                |
| ------------------------------------------------- | --------------------------------------------------------------------- |
| none                                              | Execute: run the acceptance checks, journal `accepted`, run it        |
| `done`                                            | Resend the stored result                                              |
| `accepted` or `running` in this process           | Re-attach: the execution in flight reports the result                 |
| `running`, self-terminating, from an earlier boot | Report `SUCCEEDED` with `rebooted`: the reboot it caused completed it |
| left by an earlier process, retry-safe            | Re-execute, after the acceptance checks run again                     |
| left by an earlier process, not retry-safe        | Report `FAILED` with `interrupted`, for the controller to decide      |

At start, `journal.Recover` settles the entries whose outcome needs no server input (`rebooted` and `interrupted`)
so they go out in `Hello.unacked_results`; retry-safe entries wait for the server to redeliver them, so a
re-execution passes the `not_after` and `min_generation` checks again. `disk.wipe`, `deploy.image`, `boot.kexec`,
`bmc.power`, `bmc.rotate`, `bmc.firmware`, `bmc.storage`, `site.httpProxy`, `instance.snapshot` and `gpu.reset` are not
retry-safe. `journal.Sweep` deletes done entries older than 7 days, and accepted or running
entries that an earlier process left and nothing redelivered within those 7 days (the server has long expired them);
entries of the running process are never swept.

The server keeps the matching half at `Hello` (see [resume rules](/docs/daemon/sessions#resume-rules)): a new boot
completes self-terminating commands, and a new journal fails non-retry-safe in-flight commands with `journal-lost`
instead of replaying them into a journal that cannot deduplicate.

bcd and the FakeDaemon agree on these rules, including the `cancelled` journal outcome, the producing `boot_id` on
every result, `NodeRebootResult.previous_boot_id` on a `rebooted` result, and journaled terminal rejections.
`tests/e2e/bcd.test.ts` checks the reboot path end to end: `node.reboot` completes `SUCCEEDED` from the next boot.


# Devices and holds (/docs/daemon/devices-and-holds)

> The devices converger in bcd, which turns NodeSpec.devices into CDI holds. Covers the bindings and atoms it takes, conflicts, hold slices and host admission, the CDI spec and env file it writes, the AllocationBound, Released and Failed events and their reasons, unprivileged nodes, and Kubernetes releases.



A GPU reservation reaches a machine as one entry of its
NodeSpec's `devices` section: a `DeviceAssignment` per
allocation. bcd's **devices converger** (`daemon/internal/agent/devices.go`) turns each
into a **hold**: a CDI spec and an env file a workload manager on the node consumes, with a
cgroup slice when the reservation booked host CPU or memory. It reports every bind and release as a node-signed
event, refuses what it cannot honour with a reason, and never drops a hold on its own. [Delivery](/docs/daemon/delivery#device-assignments)
summarises it among the other sections; this page is the full account. The control plane's side (placing,
booking, binding states) is in [reservations](/docs/guides/capacity/reservations#bindings),
[atoms and bookings](/docs/guides/capacity#atoms-and-bookings), [fractional GPUs](/docs/guides/capacity/fractional-gpus)
and [Kubernetes workers](/docs/guides/capacity/kubernetes).

> **Not yet run on a real GPU host as root.** 
  The converger is tested in Go against a fake host, and end to end by `tests/e2e/bcd.test.ts` with a fakehw build
  (binding and releasing GPU reservations, also from an unprivileged node beside a real `bcd helper`). The
  real-host e2e of round 6 ran bcd as an ordinary user without `[helper]`, so on real hardware only the refusal path
  below has run: no hold's CDI spec has been written to a real `/etc/cdi`, no hold slice created on a real systemd, and
  no container started from `bcd.dev/gpu=<id>`. See [running bcd on real hardware](/docs/daemon/real-hardware).

## The assignment [#the-assignment]

| `DeviceAssignment` field | Meaning to bcd                                                                                                                               |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `allocation_id`          | The hold's name: its files, slice and events are keyed by it                                                                                 |
| `reservation_id`         | Written into the spec's annotations and the env file                                                                                         |
| `binding`                | `BINDING_HOLD` or `BINDING_KUBERNETES`; anything else is refused                                                                             |
| `devices`                | Atoms, each an `id` and a `device_class` (an empty class reads as `gpu`)                                                                     |
| `limits`                 | Host resources booked with the devices: `cpus`, `mems`, `cpu_millis`, `memory_bytes`, `hugepages_2mi`, `hugepages_1gi`. Absent: devices only |
| `env`                    | Extra `KEY=VALUE` lines for the env file                                                                                                     |

`instance_id` (for `BINDING_INSTANCE`) is not read: GPUs for VMs arrive in the `instances` section, not here.

### Bindings [#bindings]

| Binding                                           | What bcd does                                                                                                                                                                                                                                                                                                                                |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BINDING_HOLD`                                    | A CDI hold for a workload manager the operator runs on the node (containerd, Docker, Podman, Slurm, a wrapper). The spec carries NVIDIA's device edits, so `bcd.dev/gpu=<id>` is all a container needs                                                                                                                                       |
| `BINDING_KUBERNETES`                              | The same hold, marked for the kubelet: bcd's device plugins (or DRA driver) advertise exactly these devices on a node bound to a cluster. No `NVIDIA_VISIBLE_DEVICES` and no vendor edits in the spec, since a pod may get only part of the hold and the plugin sets its own. Its release waits for the pods ([below](#kubernetes-releases)) |
| `BINDING_INSTANCE`, `BINDING_NODE`, anything else | `AllocationFailed` with `unsupported-binding` (`binding BINDING_INSTANCE is not supported`)                                                                                                                                                                                                                                                  |

### Atoms [#atoms]

Every device is an atom, resolved against the node's current inventory (`resolveAtom`,
`holds.go`). The ids are the ones [inventory](/docs/guides/nodes/inventory-and-health#devices-and-atoms) gives:

| Class      | Id                      | Parent GPU  | `NVIDIA_VISIBLE_DEVICES` names | Held when                                                                   |
| ---------- | ----------------------- | ----------- | ------------------------------ | --------------------------------------------------------------------------- |
| `gpu`      | The GPU's UUID          | Itself      | The GPU's UUID                 | The inventory lists the GPU                                                 |
| `gpu.mig`  | `<gpuUuid>/gi<g>/ci<c>` | `<gpuUuid>` | The MIG device's `MIG-…` UUID  | The GPU has that GPU and compute instance, with a UUID from `nvidia-smi -L` |
| `gpu.slot` | `<gpuUuid>/s<k>`        | `<gpuUuid>` | The parent GPU's UUID          | The parent GPU is listed                                                    |

A MIG instance the control plane booked but the [`mig-layout` unit](/docs/daemon/host-config#mig-layout) has not
created yet is `device-missing` (`GPU-… has no MIG instance gi1/ci0`), as is a GPU the inventory does not list. A
malformed MIG or slot id, or another class (`nic.vf`, `disk`), is `unsupported-device`. Slots are a bookkeeping unit:
the driver sees the whole GPU, and MPS or time-slicing shares it (see
[gpu-sharing](/docs/daemon/host-config#gpu-sharing)).

## One pass [#one-pass]

The converger plans the whole section on every pass (a new spec, the 30-second retry, the 10-minute drift pass). It
reads the holds that exist from `/etc/cdi/bcd-*.json`: a hold is its CDI spec, which survives a reboot, and the real
atom ids are read back from each device's `bcd.dev/device-id` annotation. Then, in this order:

1. **Kubernetes releases pending from earlier passes** are reported once no pod holds them, or dropped unreported
   when the spec wants the allocation again ([below](#kubernetes-releases)).
2. **Holds no longer wanted** are released: their CDI spec and env file removed, then `AllocationReleased`. A plain
   hold's slice is stopped once it is empty; one still running a workload is kept with its limits.
3. **Leftover slices** whose hold is neither wanted nor released in this pass are stopped again once empty (e01d7ff),
   so a slice whose workload outlived its release goes when that workload exits.
4. **Holds still wanted** are kept. Their slice is created again when it is missing (systemd's runtime properties
   vanish on a reboot; the hold does not), and a missing env file is rewritten (it lives on tmpfs).
5. **New holds**, in spec order, are checked: binding, privilege, atoms, conflicts, host admission. Each that passes
   is bound; each that fails is collected.
6. **Failures**, if any, are reported as one `AllocationFailed` per allocation, and the section fails terminally.

Releases come before binds, so a spec that moves a GPU from one allocation to another in one generation works.
Within the pass, every atom a new hold claims counts against the holds after it.

### Conflicts [#conflicts]

A new hold is refused with `overlap` (`holdConflict`) when any of its atoms:

| Conflict                                                         | Message                                     |
| ---------------------------------------------------------------- | ------------------------------------------- |
| Is listed twice in the assignment                                | `a device is listed twice`                  |
| Has a parent GPU a VM has, or is to have, through VFIO           | `<gpu> is passed through to instance <id>`  |
| Is held by another allocation                                    | `<atom> is assigned to <allocation>`        |
| Is a whole GPU while part of it (a MIG instance, a slot) is held | `<gpu> has <atom> assigned to <allocation>` |
| Is part of a GPU another allocation holds whole                  | `<gpu> is assigned to <allocation>`         |

"Held" covers the live holds, the holds bound earlier in the same pass, and the devices of Kubernetes releases still
pending. While the releases file is unreadable, every GPU of the node is held by
`(pending Kubernetes releases that could not be read)`. The VFIO side counts the GPUs in the current instance specs
and those running domains had at the last pass, so a GPU dropped from a running VM's spec stays the VM's until its
next start. The other direction holds too: a VM start, `gpu.reset`, `diag.dcgm` from level 2 and the `mig-layout`
unit refuse a GPU a live hold or a pending release names (see [gpu.reset and live holds](/docs/daemon/delivery#gpureset-and-live-holds)).

### Hold slices [#hold-slices]

A hold whose assignment carries `limits` runs in its own cgroup-v2 slice, created before the CDI spec names it
(260e33c; `daemon/internal/slice`). A `BINDING_KUBERNETES` hold never gets one: its pods live in the kubelet's
`kubepods.slice`, where the kubelet enforces their requests and limits (e01d7ff).

| Slice      | Value                                                                                                                                                                                                                                                |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Name       | `bcd-alloc-<allocation>.slice`, refused unless it matches `^bcd-alloc-[A-Za-z0-9_]{1,80}\.slice$`                                                                                                                                                    |
| Path       | `/sys/fs/cgroup/bcd.slice/bcd-alloc.slice/bcd-alloc-<allocation>.slice`                                                                                                                                                                              |
| Created    | `systemctl start`, then `systemctl set-property --runtime` with `AllowedCPUs` (`cpus`), `AllowedMemoryNodes` (`mems`), `CPUQuota` (`cpu_millis` as whole percents of one CPU, rounded up), `MemoryMax` (`memory_bytes`) and `MemoryHigh` (90% of it) |
| Huge pages | `+hugetlb` written to `cgroup.subtree_control` of `/sys/fs/cgroup`, `bcd.slice` and `bcd-alloc.slice`, then `hugetlb.2MB.max` and `hugetlb.1GB.max` in bytes (systemd has no property for them)                                                      |
| Checked    | `cpus` and `mems` must be cpulists of at most 1024 characters; the helper applies the same check as root                                                                                                                                             |
| Stopped    | `systemctl stop` only once `cgroup.events` says `populated 0`; a release never kills a workload                                                                                                                                                      |

bcd does not move processes into the slice. It names it in the spec's `bcd.dev/cgroup-parent` annotation, in
`BCD_CGROUP_PARENT` and in `AllocationBound.cgroup_parent`, and the workload manager places the workload there. Under
`bcd helper` the slice is `slice.apply` and `slice.remove`, which accept only such a name and cpulists. A fakehw host
keeps each slice as a JSON record under `/run/bcd/slices` instead.

### Host admission [#host-admission]

Since 31555bf a new hold whose limits carry a memory limit or huge pages is admitted against the host first
(`admission.go`), the same way an instance start is, so VMs and holds never take what the other was promised. A hold
with only `cpus` or `cpu_millis` is not admitted. The host is read once per pass, when the first such hold needs it:

| Quantity                | Read from                                                                                                                                                                                                                                                    |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Normal memory available | `MemTotal` from `/proc/meminfo`, less the whole huge page pool (`Hugetlb`) and 1 GiB of headroom for the host                                                                                                                                                |
| Committed normal memory | Every running or paused domain bcd manages: its memory (unless huge-page backed) plus VMM overhead (`memory/32 + 256 MiB`); and every bound hold's `MemoryMax`, a Kubernetes hold's booked memory included                                                   |
| Free huge pages         | Per size, host-wide `free_hugepages` less `resv_hugepages`, and each NUMA node's `free_hugepages`; less what the bound holds may still take (their huge page limits less what their slice has faulted in, `hugetlb.<size>.current`), from their `mems` nodes |

A new hold must fit its `memory_bytes` in what normal memory is left, and its huge pages on its `mems` nodes (the
whole host without `mems`), never more than the host-wide count. One that does not fit fails with
`resources-exhausted`:

```text
ResourcesExhausted: 4096 MiB of memory needed, 1024 MiB left on this host
ResourcesExhausted: 8 hugepages of 1048576 KiB needed, 2 free
```

A host whose memory cannot be read binds without the check and logs `host memory unreadable; binding without
host-side admission`. A fakehw host is never admitted. The page sizes, pools and the instance side are on
[host configuration](/docs/daemon/host-config#hugepages) and [delivery](/docs/daemon/delivery#instances).

## What a bind writes [#what-a-bind-writes]

A bind runs, in order: the slice (when sliced), the CDI spec, the env file, then the `AllocationBound` event. A
failure of any write is retryable: the section stays `PROGRESSING`, the pass is tried again in 30 seconds, and no
event is sent until every file is in place. Under [`bcd helper`](/docs/daemon#privilege-separation) the node asks
root for a `file.write` of kind `cdi` or `allocation-env` with the allocation id, and the helper builds the path
itself; a refused operation is retried the same way.

### The CDI spec [#the-cdi-spec]

`/etc/cdi/bcd-<allocation>.json`, mode 0644, written atomically. A `BINDING_HOLD` on one whole GPU with limits, on a
host whose toolkit wrote `/etc/cdi/nvidia.json` (vendor mounts and hooks shortened):

```json title="/etc/cdi/bcd-alloc_01jz8k2v.json"
{
  "cdiVersion": "0.7.0",
  "kind": "bcd.dev/gpu",
  "annotations": {
    "bcd.dev/allocation-id": "alloc_01jz8k2v",
    "bcd.dev/reservation-id": "rsv_01jz8k1q",
    "bcd.dev/generation": "42",
    "bcd.dev/cgroup-parent": "bcd-alloc-alloc_01jz8k2v.slice"
  },
  "devices": [
    {
      "name": "GPU-5f1c2a90-0000-4000-8000-000000000001",
      "annotations": { "bcd.dev/device-id": "GPU-5f1c2a90-0000-4000-8000-000000000001" },
      "containerEdits": { "env": null, "deviceNodes": [{ "path": "/dev/nvidia0" }] }
    }
  ],
  "containerEdits": {
    "env": [
      "BCD_ALLOCATION_ID=alloc_01jz8k2v",
      "BCD_CGROUP_PARENT=bcd-alloc-alloc_01jz8k2v.slice",
      "NVIDIA_VISIBLE_DEVICES=GPU-5f1c2a90-0000-4000-8000-000000000001"
    ],
    "deviceNodes": [{ "path": "/dev/nvidiactl" }, { "path": "/dev/nvidia-uvm" }],
    "mounts": [{ "hostPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.580.65.06", "containerPath": "…", "options": ["ro", "nosuid", "nodev", "bind"] }],
    "hooks": [{ "hookName": "createContainer", "path": "/usr/bin/nvidia-cdi-hook", "args": ["nvidia-cdi-hook", "update-ldcache", "…"] }],
    "additionalGids": [44]
  }
}
```

| Part                       | Contents                                                                                                                                                                                                                                                      |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cdiVersion`               | `0.6.0` (device annotations need it), or the vendor spec's version when that is later                                                                                                                                                                         |
| `kind`                     | `bcd.dev/gpu`                                                                                                                                                                                                                                                 |
| `annotations`              | `bcd.dev/allocation-id`, `bcd.dev/reservation-id`, `bcd.dev/generation`; `bcd.dev/cgroup-parent` for a sliced hold; `bcd.dev/binding: kubernetes` for a Kubernetes hold (which is how the device plugins find what to advertise)                              |
| `devices[].name`           | The atom id with `/` as `_` (CDI names allow no `/`): `GPU-…_gi1_ci0`, `GPU-…_s3`                                                                                                                                                                             |
| `devices[].annotations`    | `bcd.dev/device-id`, the real atom id                                                                                                                                                                                                                         |
| `devices[].containerEdits` | For a hold: the vendor spec's edits for the device the atom stands for (a slot its GPU's, a MIG instance its `MIG-…` device's); none for a Kubernetes hold                                                                                                    |
| `containerEdits.env`       | `BCD_ALLOCATION_ID`; `BCD_CGROUP_PARENT` when sliced; `CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps` when the hold has a slot and that directory exists; for a hold (not Kubernetes) one `NVIDIA_VISIBLE_DEVICES` listing each GPU once and each MIG device's UUID |
| `containerEdits`, the rest | For a hold: the vendor spec's common device nodes, mounts, hooks and additional GIDs, once, without its `NVIDIA_VISIBLE_DEVICES=void`                                                                                                                         |

How bcd reads NVIDIA's spec, and what happens without one or with a stale one, is on
[GPUs on a node](/docs/daemon/gpus#cdi-nvidias-spec-and-bcds). Without a JSON vendor spec a hold carries only the
env, which works with a toolkit in its legacy mode but grants nothing in CDI mode.

A container then asks for the devices by name, for example `--device bcd.dev/gpu=GPU-5f1c2a90-…` with Podman or
Docker's CDI support, or the same name in containerd's CDI device list.

### The env file [#the-env-file]

`/run/bcd/allocations/<allocation>.env`, mode 0644, one `KEY=VALUE` per line, for a workload manager that reads env
files rather than CDI:

```text title="/run/bcd/allocations/alloc_01jz8k2v.env"
BCD_ALLOCATION_ID=alloc_01jz8k2v
BCD_CGROUP_PARENT=bcd-alloc-alloc_01jz8k2v.slice
BCD_RESERVATION_ID=rsv_01jz8k1q
NVIDIA_VISIBLE_DEVICES=GPU-5f1c2a90-0000-4000-8000-000000000001
```

The lines are `BCD_ALLOCATION_ID`, `BCD_CGROUP_PARENT` (sliced holds), `CUDA_MPS_PIPE_DIRECTORY` (slots with MPS
running), `BCD_RESERVATION_ID`, `NVIDIA_VISIBLE_DEVICES`, then the assignment's `env` lines. Unlike the CDI spec, the
env file carries `NVIDIA_VISIBLE_DEVICES` for a Kubernetes hold too. `/run` is tmpfs, so after a reboot the next pass
writes it again from the hold's spec.

## Events [#events]

Every bind and release is an `AttestedReport`: the body (with the node id and boot id) is signed with the node key
under purpose `attested-report`, so a gateway cannot forge a binding. Failures are plain events. All three are
journaled before they are sent and replayed until acknowledged (see [events](/docs/daemon/delivery#events)).

| Event                           | Fields                                                                                                                                                   | When                                                                                       |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `AllocationBound` (attested)    | `allocation_id`, `generation`, `device_ids`, `cdi_devices` (`bcd.dev/gpu=<name>` each), `cdi_spec_path`, `env_file`, `cgroup_parent` (sliced holds only) | Once the slice, spec and env file are written                                              |
| `AllocationReleased` (attested) | `allocation_id`, `generation`, `device_ids`                                                                                                              | A plain hold: once its files are removed. A Kubernetes hold: once no pod holds its devices |
| `AllocationFailed`              | `allocation_id`, `generation`, `reason`, `message`                                                                                                       | The assignment cannot be honoured; reported once per allocation and generation             |

With [attested bindings](/docs/guides/capacity/reservations#attested-bindings) (on by default with command signing),
a reservation turns active only on the node's own signed `AllocationBound`.

### Failure reasons [#failure-reasons]

| Reason                | Cause                                                                                                               | Example message                                                                                                                               |
| --------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `unsupported-binding` | A binding other than `HOLD` or `KUBERNETES`                                                                         | `binding BINDING_INSTANCE is not supported`                                                                                                   |
| `node-unprivileged`   | bcd runs without root and without `[helper]` ([below](#unprivileged-nodes))                                         | `bcd node runs without root and without [helper]: it cannot write /etc/cdi or create hold slices (run bcd node as root, or under bcd helper)` |
| `unsupported-device`  | A class other than `gpu`, `gpu.mig`, `gpu.slot`, or a malformed MIG or slot id                                      | `device class nic.vf cannot be held`; `"GPU-a/x" is not a MIG instance id (<gpu>/gi<g>/ci<c>)`                                                |
| `device-missing`      | A GPU the inventory does not list, or a MIG instance not created yet; every missing device is named, joined by `; ` | `GPU-… not present`; `GPU-… has no MIG instance gi1/ci0`                                                                                      |
| `overlap`             | A [conflict](#conflicts) with a hold, a pending release or a VM                                                     | `GPU-… is assigned to alloc_…`                                                                                                                |
| `resources-exhausted` | [Host admission](#host-admission) refused the limits                                                                | `ResourcesExhausted: 4096 MiB of memory needed, 1024 MiB left on this host`                                                                   |

The checks run in that order for each new hold, and the first that fails decides. Once the failures are reported,
the section is `FAILED` terminally at that generation (its message lists `<allocation>: <reason>` for each), so
`observed_generation` still advances and the other sections converge. The holds of the same pass that passed their
checks are bound anyway. A later generation that still carries a failed assignment reports it again.

On the server, the gateway's `allocationFailed` handler marks the allocation failed (`markAllocationFailed`), which
wakes the reservation at once instead of at the bind timeout: it gets the condition `BindTimeout` with reason
`NodeRejectedBinding`, its bookings are released and it is placed again, up to two times (`maxBindRetries`), after
which it fails. A node lease (`binding: 'node'`) fails at once. See
[states and conditions](/docs/guides/capacity/reservations#states-and-conditions).

## Unprivileged nodes [#unprivileged-nodes]

Writing `/etc/cdi` and creating slices is root's work. A host build of `bcd node` running as an ordinary user without
`[helper]` (`Options.Unprivileged`: not a fakehw build, mode `node`, effective uid not 0, no `[helper] socket`) can do
neither. Before da60bbd such a node retried every hold every 30 seconds on `systemctl`'s "Access denied" and stayed
`binding` forever, logging nothing above debug. Now:

* **Hello claims nothing.** A GPU host normally advertises the capabilities `cdi`, `devices.limits` and
  `binding.kubernetes` (and `nvml` when `nvidia-smi` answered). An unprivileged node advertises none of the first
  three. The gateway writes capabilities as `bc.dev/cap.<name>` system labels, and capacity requires
  `bc.dev/cap.devices.limits` for a reservation that books host resources and `bc.dev/cap.binding.kubernetes` for a
  Kubernetes one (`requiredCapabilities`), so neither is ever placed there.
* **A plain hold fails at once.** A GPU reservation that books no host resources requires no capability label and can
  still be placed on the node. bcd fails it straight away with `node-unprivileged`, writes nothing, and the control
  plane re-places it as above.
* **Doctor says so.** `bcd doctor` warns `gpu-binding` on a host with GPUs whose bcd would run this way, and section
  failures are now logged (`WARN` while retrying, `ERROR` when failed).

Run `bcd node` as root, or as the unprivileged `bcd` account beside a root `bcd helper` (see
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)); both bind holds.
`TestAnUnprivilegedNodeTakesNoHolds` covers the capabilities and the refusal, and the real-host e2e checks on a real
GPU host that the node carries no `bc.dev/cap.devices.limits` label and that a GPU reservation never becomes
`binding` or `active` there.

## Kubernetes releases [#kubernetes-releases]

A plain hold is released in the pass that sees it gone. A Kubernetes hold cannot be: pods may still have its devices.
So its release is split (`kuberelease.go`):

1. **Recorded first.** The release goes into `<data_dir>/releasing.json` (0600: device ids, generation, time) before
   anything is removed, so a crash in between never loses it.
2. **Withdrawn.** The CDI spec and env file are removed at once. The device plugins stop advertising the devices, and
   the kube reaper evicts pods still holding them.
3. **Reported when free.** Each pass asks whether any pod still holds a device of the allocation (the kubelet's
   pod-resources, and the host's pod processes for whole GPUs; unreachable counts as held). Only when none does is
   `AllocationReleased` sent and the record dropped. A pass is triggered when the reaper frees devices, with a
   30-second recheck as a backstop.
4. **Claimed meanwhile.** Until reported, the devices stay claimed on the node: no new hold, VM, `gpu.reset`,
   `diag.dcgm` or `mig-layout` gets them, and the control plane keeps them booked.
5. **Taken back.** A spec that wants the allocation again drops the pending release unreported.

A release pending longer than 5 minutes raises `KubernetesReleaseStuck` (reason `PodsHoldDevices`). An unreadable
`releasing.json` is renamed to `releasing.json.unreadable-<unix time>` (kept, never deleted), the marker
`<data_dir>/releasing.unreadable` is written, every GPU of the node counts as claimed, and the node reports
`KubernetesReleasesUnreadable` (reason `ReleasesFileUnreadable`). The marker goes once the kube side reports that no
pod holds a bcd GPU of an allocation the node does not know, or when an operator removes it. What the control plane
does meanwhile (carrying the booking, NoSchedule) is on
[Kubernetes workers](/docs/guides/capacity/kubernetes#release) and
[an unreadable releases file](/docs/guides/capacity/kubernetes#an-unreadable-releases-file).

## Seeing the holds [#seeing-the-holds]

* `bcd status` shows, per GPU, the allocation holding it or one of its MIG instances or slots (see
  [bcd status](/docs/daemon#bcd-status)).
* `ls /etc/cdi/bcd-*.json` lists the holds, and `/run/bcd/allocations/` their env files.
* `systemctl status bcd-alloc-<allocation>.slice` shows a sliced hold's cgroup and its processes.
* On the control plane, `reservations.get` returns the binding, with `cgroupParent` for a sliced hold (see
  [bindings](/docs/guides/capacity/reservations#bindings)).

## Tests [#tests]

| Test                                             | Covers                                                                                                                                                               |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `holds_test.go`                                  | Atom resolution (parents, what the driver sees), conflicts across whole GPUs, parts and VMs, the spec of MIG instances, slots and limits                             |
| `cdivendor_test.go`                              | Vendor edits per device, the common edits once, `void` dropped, `cdiVersion`, Kubernetes holds left bare, a device the vendor spec does not name, WSL's `all` device |
| `slices_test.go`, `internal/slice/slice_test.go` | A hold with limits in its slice; the properties and huge page limits; names and cpulists refused; removal never killing a workload                                   |
| `holdadmission_test.go`, `admission_test.go`     | Holds and VMs sharing the host's memory, `resources-exhausted`, committed memory rather than use, huge pages from the placement's nodes                              |
| `gpubusy_test.go`                                | A VM refused a GPU a hold has; `gpu.reset` and `diag.dcgm` refusing GPUs in use by parts or VMs                                                                      |
| `kuberelease_test.go`                            | A Kubernetes release waiting for its pods, the stuck condition, an unreadable releases file keeping every GPU claimed                                                |
| `capabilities_test.go`                           | Hello capabilities with and without GPUs, and the unprivileged refusal                                                                                               |
| `tests/e2e/bcd.test.ts`                          | Bind and release through a real gateway with a fakehw build, directly and from an unprivileged node beside a real `bcd helper`                                       |


# Failure modes (/docs/daemon/failure-modes)

> What happens when a gateway, a node, the network, the journal, PostgreSQL, the api or a worker fails, or clocks and certificates go wrong, and which invariants hold.



The daemon protocol is built on the assumption that everything fails: processes crash, links go half-open,
databases fail over, clocks drift, journals get wiped. The design goal is that each failure costs a reconnect and a
replay, never a duplicated side effect, a double booking, or a released allocation. This page lists what happens
for each failure, what state is lost, and which invariant the behavior protects.

## The invariants [#the-invariants]

PLAN.md §15 defines the fleet invariants. Six of them are checked mechanically after every simulated run, and a
seventh after a simulated night, by pure functions in `@better-compute/testing` (`checkI1`, `checkI2`, `checkI3`,
`checkI4`, `checkI5`, `checkI7`, `checkI8`):

| Id | Invariant                                                                                                                                                                                                                                                                                                      | Checked over                                                           |
| -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| I1 | No overlapping bookings or node-reported assignments: no atom booked by two allocations at one instant, no device bound by a node for two allocations at once, and no binding of a device its allocation did not book                                                                                          | `device_bookings` rows and the bindings nodes reported                 |
| I2 | At most one execution per command: a non-retry-safe or self-terminating command starts its side effect at most once; a retry-safe one re-executes only after a crash interrupted it; after a journal loss, a non-retry-safe command accepted in the lost journal never starts again and never ends `succeeded` | the side-effect facts the daemons record, journal losses, and outcomes |
| I3 | Convergence within a bound: a delivered spec is reported observed within 60 s while the node stays reachable                                                                                                                                                                                                   | specs delivered and status reports, from the simulator's links         |
| I4 | No false liveness positives: no node is marked `unreachable` or `lost` within 20 s of a gateway hearing from it, whatever the gateway's clock says                                                                                                                                                             | connectivity transitions against the frames the transport delivered    |
| I5 | No allocation released only because a node disconnected                                                                                                                                                                                                                                                        | release reasons against connectivity transitions                       |
| I7 | Nothing pending that an offline solver could place (checked after the nightly simulation's quiet period)                                                                                                                                                                                                       | capacity's `placeablePending`                                          |
| I8 | Billing conservation: the sum of usage equals the sum of booking duration times atoms, with no duplicate keys                                                                                                                                                                                                  | allocations and flushed usage windows                                  |

I6 (latency) is part of the design but has no checker in this module; the real-time benchmarks measure it.

## At a glance [#at-a-glance]

| Failure                                                             | What happens                                                                                                                                                         | Lost                                        | Invariant kept      |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------- |
| [Gateway crash](#gateway-crash)                                     | Daemons redial elsewhere with jitter; `Hello` bumps the epoch and replays; 120 s grace before `unreachable`                                                          | Open streams and tunnels                    | I2, I5              |
| [Gateway cut off from PostgreSQL](#gateway-cut-off-from-postgresql) | The gateway fences itself within 10 s and drops every stream; its stale writes fail on the epoch                                                                     | Open streams                                | One router per node |
| [Daemon crash](#daemon-crash-or-node-reboot)                        | systemd restarts it; the journal settles interrupted commands                                                                                                        | Work in progress of non-retry-safe commands | I2                  |
| [Node reboot](#daemon-crash-or-node-reboot)                         | New `boot_id`: self-terminating commands complete on rejoin                                                                                                          | Nothing                                     | I2                  |
| [Network partition](#network-partition-and-half-open-links)         | Session closes after three missed heartbeats; node `disconnected`, later `unreachable`; workloads keep running                                                       | Nothing                                     | I5                  |
| [Journal loss](#journal-loss)                                       | New `journal_id`: in-flight non-retry-safe commands fail `journal-lost`                                                                                              | Unacked events of the lost journal          | I2                  |
| [PostgreSQL unavailable](#postgresql-unavailable)                   | Gateways fence and drop streams; daemons back off and keep enforcing their last spec; 5 min grace afterwards                                                         | Nothing durable                             | I5                  |
| [api or worker crash](#api-or-worker-crash)                         | Stateless api restarts; worker leases expire and move                                                                                                                | Nothing durable                             | none needed         |
| [Clock skew](#clock-skew)                                           | Ages instead of timestamps; server time on the daemon; `ClockSkew` on control-plane processes                                                                        | Nothing                                     | I4                  |
| [Certificate expiry](#certificate-expiry)                           | Renewal at 60% of the lifetime; the session closes at expiry                                                                                                         | The session                                 |                     |
| [TPM cleared or unavailable](#tpm-cleared-or-unavailable)           | A certificate naming the TPM's AK does not renew without that AK; bcd retries and fails loudly, an operator re-joins it with `bcd join --reset --yes --attestor tpm` | The session, until the re-join              |                     |
| [Certificate revocation](#certificate-revocation)                   | `RevocationNotice`, then the stream closes; the daemon stops redialing                                                                                               | The session                                 |                     |
| [Reconnect storm](#reconnect-storm)                                 | Token bucket, pending-Hello bound, jitter, hash-only `Hello`                                                                                                         | Nothing                                     | Bounded load        |
| [Compromised gateway](#compromised-gateway)                         | Can delay, drop or read frames; cannot mint certificates, read tokens or secrets, or forge results, bindings or signed intent                                        |                                             | P14                 |
| [Cloned identity](#cloned-identity)                                 | The sessions alternate; more than 3 alternations in 10 minutes revoke the certificate                                                                                | Both clones' sessions                       | I5                  |

## Gateway crash [#gateway-crash]

A gateway process dies: no disconnect writes, no lease release. Everything it held in memory is gone, and it held
nothing else.

1. Each daemon sees its stream end and redials with full-jitter back-off (0 to 1 s, doubling to 60 s). The L4 load
   balancer sends it to a live gateway.
2. The new gateway's `Hello` transaction bumps the epoch and routes the node to itself. In-flight commands
   (`dispatched`, `acked`, `running`) move to the new epoch and are replayed in `seq` order, except those the daemon
   reports as running; its unacked results are applied first. Pending commands need no rewrite: dispatch is a join
   on `node_sessions.gateway_id`, which now names the new gateway.
3. The daemon's journal deduplicates every replay: a done command resends its result, a running one re-attaches.
   Nothing runs twice (I2).
4. Nodes that do not come back are handled by the Liveness controller: once the dead gateway's `gateway:<id>`
   lease expires (20 s), its nodes become `disconnected`, and only after the 40-second lease TTL plus a 120-second
   gateway-loss grace do they become `unreachable`. That transition adds a NoSchedule reason and a condition and
   releases nothing (I5).

Tunnels on the dead gateway close; the caller opens a new one. In the fleet simulator with three real gateways and
the real control plane, killing one re-homed its nodes with a p99 of 1 second at 1,000 nodes and 13.25 seconds at
10,000, with no invariant violations (`bench/multi-gateway`). PLAN.md's v0.4 target is under 60 seconds with no
unreachable marks and no duplicate executions.

## Gateway cut off from PostgreSQL [#gateway-cut-off-from-postgresql]

A gateway that loses the database, or stalls, cannot renew its `gateway:<id>` lease. After half the lease TTL (10
seconds) without a successful renewal, or as soon as it finds another owner on the lease, it fences itself: it
forgets its lease and closes every stream `UNAVAILABLE`. Its daemons redial elsewhere.

A gateway that does not know it is stale (a paused process that wakes up) is stopped by two fences:

* every routing transaction begins by re-reading its lease `FOR SHARE`; a lease that moved aborts the transaction
  (`FencedError`), which fences the gateway;
* routing and liveness writes carry `AND session_epoch = $myEpoch`. After the node's `Hello` elsewhere, the zombie's
  dispatch marks and disconnect writes match nothing, and its liveness flush reports those nodes as gone, so it
  closes their streams `ABORTED`. Dispatch holds the session row `FOR SHARE`, so it cannot race an epoch bump.

Results and events the zombie still ingests are facts and are accepted from any epoch, deduplicated as usual. At
most one gateway routes to a node at a time. `tests/sim/multi-gateway.test.ts` runs a stalled gateway that comes back
as a zombie holding sessions its nodes have left.

## Daemon crash or node reboot [#daemon-crash-or-node-reboot]

**Daemon crash.** systemd restarts bcd after 5 seconds. The new process has a new incarnation id, and
`journal.Recover` settles what the old one left behind:

| Left in the journal                     | Outcome                                                                               |
| --------------------------------------- | ------------------------------------------------------------------------------------- |
| `done`                                  | Its result is replayed in `Hello.unacked_results` until acked                         |
| `accepted` or `running`, retry-safe     | Waits for the server to redeliver it, then runs again after the acceptance checks     |
| `accepted` or `running`, not retry-safe | `FAILED` with `interrupted`; the owning controller decides                            |
| `running`, self-terminating, same boot  | Treated like any other entry by its retry-safety (the process died before the reboot) |

The rule that makes this safe: `accepted` is fsynced before any side effect starts. A command is either known to the
journal before it could have done anything, or it never ran.

**Node reboot.** The kernel `boot_id` changes. A self-terminating command (`node.reboot`, `boot.kexec`,
`daemon.upgrade`) journaled `running{boot_id}` immediately before its side effect, so on restart the daemon reports
it `SUCCEEDED` with `rebooted`, and the server's `Hello` transaction completes any in-flight self-terminating command
the same way when it sees the new `boot_id` or role. Neither side ever replays it. Files under `/run` are gone after a
reboot; the devices converger rewrites missing env files for assignments it still holds.

**Self-update.** A `daemon.upgrade` restarts bcd without a reboot, so the boot id does not change; bcd decides it from
`upgrade.json` instead (de9f380). The new binary reports `SUCCEEDED` once its first `Welcome` arrives. A binary that
crashes or never gets a `Welcome` within 120 seconds is swapped back (under systemd a crash loop also triggers
`bcd-rollback.service`), and the old binary reports `FAILED` with `update-failed` and never tries that version again.
On a node under `bcd helper`, the helper's own watchdog also swaps back an update still unconfirmed 30 seconds after
its deadline and restarts `bcd.service`, so a new binary that never gets as far as asking cannot keep the node on it
(2d67b25). The restarted process decides by where the `current` link points, never by the version it claims to run,
so a lying binary cannot mark the new version failed and disarm the watchdog (c139b9c).

**A bad release that confirmed.** The swap back covers only an update that never confirmed. Once the new binary
confirms, the release floor rises to its sequence and bcd refuses every older binary, the one still in its other slot
included, so neither the watchdog nor a rollout can take the node back (c139b9c). `rollouts.abort` with `rollback`
returns only nodes still upgrading (8347a3d). A release that confirmed and then misbehaves is fixed by rolling
forward: sign and register a newer release with a higher sequence and roll it out; see
[rollouts](/docs/guides/rollouts#pause-resume-abort). See [self-update](/docs/daemon/install#self-update).

**The helper is down or refuses.** A `bcd node` running unprivileged beside [`bcd helper`](/docs/daemon#privilege-separation)
(28dc2ae) keeps its session and journal when the helper is gone, but every operation it sends there (a CDI spec or
env file, the sshd files, the nftables table, a GPU reset, a reboot, a self-update step) fails with an error that
starts `helper:`, like
the direct operation failing. Such an error is not terminal, so a `devices` or `access` section stays `PROGRESSING`
and is retried on the next pass (the spec is not observed meanwhile), and a command such as `gpu.reset` or
`node.reboot` fails as it would without root. A node and a helper of different protocol versions
refuse each other's messages, so restart both after replacing the binary. On a node installed with
`--unprivileged`, `bcd-helper.service` restarts the helper after 2 seconds (`Restart=always`), and `bcd.service`
`Requires=` it, so stopping the helper stops the node too; see
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged).

## Network partition and half-open links [#network-partition-and-half-open-links]

When a node's link is cut, both ends notice through silence. The gateway closes the session after three missed
heartbeats (30 seconds) and writes an epoch-fenced disconnect; HTTP/2 PINGs every 20 seconds detect a dead TCP
connection. A half-open connection, where one end died silently, ends the same way, and when the node comes back its
new session supersedes the old one by epoch.

While partitioned, the daemon keeps enforcing its last desired state and keeps every device assignment: it never
drops one on its own. Workloads keep running. It journals events (up to 64 MiB and 10,000 unacked; metric events are
dropped oldest-first when full, results and transitions never) and redials with back-off.

On the control plane the node goes `disconnected`, then `unreachable` after the 40-second lease TTL, with a
NoSchedule reason and the `NodeUnreachable` condition. Only nodes of `provider` and `vps-host` pools can become
`lost`, and only when the pool sets `policy.lostAfter`; metal never does automatically. No step releases an allocation, a booking or a command (I5).
Commands that pass `not_after` while the node is away are swept to `expired`, or rejected `EXPIRED` by the daemon if
they were already delivered.

## Journal loss [#journal-loss]

A ramdisk reboot, a reinstall or corruption loses the journal. The daemon starts a new one with a new `journal_id`;
its identity stays with its key, so it reconnects as the same node. bcd moves a journal file that does not open aside
(`journal.db.corrupt-<unix time>`) rather than deleting it, and starts its server time mark from the copy in
`identity.json`, so a lost journal does not reopen the clock-rewind window.

* The server sees the new `journal_id` in `Hello` and fails every in-flight command that is not retry-safe with
  reason `journal-lost`, instead of replaying it into a journal that cannot deduplicate. Controllers re-decide
  (I2).
* Retry-safe in-flight commands are replayed and may run again, which their kind allows.
* The event ack restarts at 0 for the new journal. Events still unacked in the lost journal are gone.
* The new journal holds no spec, so the daemon reports applied generation 0 and receives a full `DesiredState` at
  `Hello`.

The fleet simulator's `journalLoss` chaos and `sim.loseJournal(nodeId)` exercise this path.

## PostgreSQL unavailable [#postgresql-unavailable]

PostgreSQL is the only source of truth, so an outage stops the fleet from changing without losing anything.

* **Gateways** cannot renew their leases and fence themselves within 10 seconds, dropping every stream. Without a
  lease they refuse new sessions `UNAVAILABLE`. A frame that fails to write also drops its stream. The lease loop
  keeps retrying, and once it renews, the gateway admits sessions again.
* **Daemons** back off up to 60 seconds between attempts, keep enforcing their last spec, and keep results and events
  in the journal until a gateway acknowledges them. Nothing is buffered in the gateway beyond its memory.
* **Commands** stay `pending`; nothing is dispatched. After the outage, those past `not_after` are swept to
  `expired`, and a stale command that does reach a node is rejected on `not_after` or `min_generation`.
* **Liveness.** When the Liveness controller finds a gap of more than 20 seconds since its last sweep, it holds every
  timed transition back for 5 minutes, so the ages the gateways could not flush do not read as dead nodes.

PLAN.md sizes this for a database failover of about 30 seconds: the fleet reconnects, nothing is marked
unreachable, and nothing is released.

## api or worker crash [#api-or-worker-crash]

**api.** api processes are stateless; a crash fails the requests in flight, and clients retry with their
idempotency keys. `Join`, `Renew` and secret requests reach JoinService and the secret broker through the gateway's
`GatewayServices` port. While the process serving them is down, `Join` and `Renew` fail and the daemon retries
(a node certificate lasts 24 hours and renews at about 60% of its life, which leaves hours of margin), and secret
requests are answered with the error `unavailable`.

**Worker.** Workers run the reconcilers, the CommandSweeper and the Liveness controller under table leases (15
seconds for shards, renewed every 5). When a worker dies, its leases expire and another worker takes them over; the
work queue loses no wake-up because a claim that is not completed simply becomes claimable again. While no worker
runs, gateways keep dispatching commands that are already pending and keep ingesting results, but nothing new is
decided and nothing is swept. The Liveness controller's outage grace (above) applies when it comes back. See
[leases and fencing](/docs/guides/concepts/consistency#leases-and-fencing).

## Clock skew [#clock-skew]

No component trusts another's wall clock.

* **Gateways report ages.** Liveness is flushed as "seen `age_ms` ago", and PostgreSQL computes
  `compute_now() - age`. A gateway whose clock is off cannot make a node look alive or dead (I4).
* **Daemons use server time.** `Welcome.server_time` gives the daemon the offset between its clock and
  `compute_now()`, and it evaluates `not_after` through that offset. Conditions carry `age_ms` on the daemon's
  monotonic clock, and the control plane derives `since` itself; event and inventory timestamps are advisory.
  Because `server_time` is unsigned, the daemon keeps a server time mark that only moves forward and refuses a
  `Welcome` more than 5 minutes before it, so a gateway cannot rewind the daemon into an expired signer's window
  (see [Welcome](/docs/daemon/sessions#welcome)).
* **Control-plane processes** anchor their business clock to `compute_now()`, re-sampled at every lease renewal. A
  process more than 1 second off raises `ClockSkew`: a log warning and a `selfCheck` finding, and the
  `clock_offset_seconds` metric shows the offset.
* **Proofs** tolerate 60 seconds: `Renew` proofs of possession, WebSocket proofs.

## Certificate expiry [#certificate-expiry]

Node certificates last at most 24 hours (ramdisk certificates at most 2), and bcd renews at about 60% of the
lifetime with `Renew`, proving possession of its current key. A session whose certificate expires is closed
`UNAUTHENTICATED` ("certificate expired; renew and reconnect") by the gateway's liveness loop.

A certificate that expired while the machine was offline cannot be renewed. From v0.2, when the server has the
`x509pop` attestor configured, the machine re-joins with its old key within 7 days unless the certificate was
revoked. See [renewal](/docs/guides/nodes/joining#renewal). Gateway serving certificates last 30 days in the PLAN.md
design, which lists serving-certificate expiry among the fleet checks of `doctor`.

## TPM cleared or unavailable [#tpm-cleared-or-unavailable]

A node that joined with the `tpm` attestor and a certified node key has a certificate naming its TPM's attestation
key (`urn:bc:tpm-ak:`), and since d94f1f3 it renews only with a next key that AK certifies. bcd never falls back to
a file key for such a certificate:

* **The TPM cannot be reached** (a missing `/dev/tpmrm0`, or, for an unprivileged `bcd node` under `[helper]`, a
  `bcd helper` that is down or cannot open the TPM: since 8870b8d that node reaches the TPM only through the
  helper). The renewal fails before any request: "the certificate's node key is in the TPM, which is unavailable"
  when bcd opens the TPM itself, or `renewal:` with the helper's error. bcd logs `renewal failed` with the attempt
  and `expiresIn`, and retries with backoff from 30 seconds to 10 minutes. Once the TPM answers again, the next
  retry renews. A TPM-held current key needs the TPM to load too, so a restarted `bcd node` does not start without
  it.
* **The TPM derives another AK** (another TPM, or one whose endorsement seed changed). The renewal fails the same
  way, with "this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed
  changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join" (a1c1a46),
  and retrying cannot help.
* **The TPM was cleared** (`TPM2_Clear`). A clear replaces the owner hierarchy's seed but not the endorsement one,
  so the AK survives and the node key does not: the key is a primary of the owner hierarchy, the stored record
  derives another key, and bcd refuses its identity with "the node key does not match the node certificate: the TPM
  derives another key from node.key (was the TPM cleared?); join again: bcd join --reset --yes, then bcd join".
  Since a1c1a46 the cleared-TPM hint is here, where a clear shows, rather than on the AK check. `bcd doctor` fails
  its `identity` check with the same text.

Either way the certificate runs out within its 24 hours and the session closes as in
[certificate expiry](#certificate-expiry). The recovery is to join again with the TPM, as root:
`bcd join --reset --yes --attestor tpm` moves the old identity and journal aside (renamed
`<name>.reset-<UTC time>`, not deleted) and joins, and the attestor reclaims the node bound to the machine's EK, with a new identity generation, a
new certified key and a fresh AK statement. A machine whose EK changed (another TPM) joins as a new node instead,
once the old node is cordoned and deleted (`better-compute nodes delete <node> --yes`, 4049621; `nodes.delete`,
54cad9e), which frees its EK binding and the machine's hardware key; the old node's `journal.db` is already aside
(b4fcc2e), so `bcd node` does not refuse it as another node's. See
[recovering a node whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed). Nodes
whose certificate names no AK (every other join) renew as before; a key of theirs in a cleared TPM is lost the same
way and needs a new join (`bcd join --reset --yes` with a token).

Refused renewals are visible to the node owner since 0e42018: once the node proved its current key, JoinService
publishes `security.renew.refused` naming the node, with the reason (for example
`renew:key-attestation-missing`); see [refused renewals](/docs/operations/security#refused-renewals). A renewal that
fails on the node before any request (the TPM unavailable, another AK) reaches no server and publishes nothing; it
shows only as `renewal failed` in bcd's log.

## Certificate revocation [#certificate-revocation]

A revoked serial reaches every gateway within about a second through the revocation feed, which polls
`node_certs.revoked_at`. The gateway sends the session `RevocationNotice{reason, serials}`, closes the node's
tunnels, and ends the stream `UNAUTHENTICATED`. The daemon stops using the certificate and does not redial with it.
New opens with the serial are refused at the handshake, and the `Hello` transaction checks `node_certs` directly,
also refusing a certificate from an older identity generation of the node. PLAN.md's target is a revoked node out
within 5 seconds.

## Reconnect storm [#reconnect-storm]

When a gateway dies or a network heals, thousands of daemons redial at once. Four things keep that bounded:
full-jitter back-off on the daemon, the handshake token bucket (200 per second per gateway), the pending-Hello bound
(64 transactions in flight, beyond which a `Hello` is refused `RESOURCE_EXHAUSTED` and retried), and a small `Hello`
that carries inventory hashes instead of inventory. Without the pending-Hello bound, a storm once queued enough
`Hello` transactions to starve the gateway's own lease renewal until it fenced itself. `Drain` spreads planned
reconnects over up to 60 seconds.

## Compromised gateway [#compromised-gateway]

A gateway holds no CA and no signing key. A compromised one can delay, drop or read frames. It cannot:

* mint certificates: `Renew` needs a proof of possession by the node's current key, verified by JoinService;
* harvest join tokens: they are sealed to the JoinService encryption key, and `JoinResponse` is signed;
* read secrets: they travel as JWE to a one-time key of the node;
* forge attested results: they carry `node_sig`, verified against `node_certs` by workers;
* forge a GPU binding: with `attestedBindings`, a reservation goes active only on the node's own signed
  `AllocationBound`; an unsigned, foreign-signed or replayed report is rejected (`reservation.binding-rejected`),
  and a forged re-report of an active binding marks it lost instead of replacing it;
* forge intent: commands and NodeSpecs are signed by worker keys the daemon verifies; the gateway relays the stored
  signed spec bytes verbatim;
* open a tunnel of its own: the daemon verifies every exec ticket again against the JWKS pinned in its NodeSpec and
  its own `jti` set (bcd keeps it in its journal), and requires the ticket's `sid` to be the tunnel it is asked to
  open.

Signed commands and specs and attested bindings all come with `pki.commandSigning` on the server. Daemons require
signatures for destructive kinds from v0.2 and for all kinds from v0.4, and for specs that change `access` from v0.2
and for every spec from v0.4. `tests/e2e/attested-bindings.test.ts` runs a compromised gateway's forged reports
through a real gateway. See [security](/docs/operations/security).

## Cloned identity [#cloned-identity]

Two machines presenting the same certificate, a copied key, keep taking the node's session from each other: each
`Hello` from the other boot wins and publishes `security.node.duplicate-session`. More than 3 such alternations within
10 minutes quarantine the certificate: it is revoked with reason `duplicate-identity`, both clones lose their
sessions through the revocation feed, and `security.node.quarantined` is published. Workloads on the real machine
keep running (nothing is released, I5); an operator re-joins it with a new token. See
[duplicate identity](/docs/daemon/sessions#duplicate-identity).

## How the failures are exercised [#how-the-failures-are-exercised]

  - [createFleetSim](/docs/operations/testing): N FakeDaemons on seeded fake hardware and in-memory links with drops, duplicates and reordering, against a gateway stub or a real gateway. Chaos: crashes, reboots, journal loss, partitions, isolation, half-open links, resets, stalls and gateway kills. Same seed, same event log.

  - [createControlPlaneSim](/docs/operations/testing): The same simulator with the real control plane on the sim clock and real gateways; facts for the checkers come from compute's own tables.

  - [better-compute sim](/docs/reference/cli): Runs the fleet simulator with a steady workload and `--chaos none`, `light` or `heavy`, then checks the simulator's invariants (I1, I2, I3, I4, I5 and I8) and prints the event-log digest.

  - [The nightly and bench suites](/docs/operations/testing): `tests/sim/multi-gateway.test.ts` (a kill mid-command, a zombie gateway, a revocation), the nightly simulation with worker outages, and `bench/multi-gateway` at up to 10,000 nodes.

```bash
better-compute sim --nodes 100 --chaos heavy --seed 7
```

The real Go daemon is tested at a smaller scale. Its unit tests cover the journal rules, crash recovery, the server
time mark and the session handling against scripted streams, and `tests/e2e/bcd.test.ts` runs the fakehw build
against a real gateway through a join, commands, a GPU binding and a reboot, plus five simulated machines joining
through [`bcd sim`](/docs/daemon#bcd-sim) (see [testing](/docs/operations/testing#the-go-daemon)). It does not run
under chaos yet: `bcd sim` exists (fakehw builds, many real agents in one process), but PLAN.md's protocol and skew
suite over lossy links and the nightly `bcd sim --nodes 10000` against real gateways and PostgreSQL through
Toxiproxy (v0.4) are not built.


# Gateway (/docs/daemon/gateway)

> The gateway holds daemon sessions in memory and writes what matters to PostgreSQL. Its loops, dispatch by join, group commit, lease, control API, tunnels and metrics.



The gateway is the process every daemon connects to. It terminates the daemons' mTLS,
keeps their sessions, heartbeats and lane counters in memory, dispatches commands, ingests results, events,
status and inventory, relays `Join`, `Renew` and secret requests, and relays interactive tunnels. It holds no CA and
no signing key (principle P14), and PostgreSQL stays the only source of truth (P1): restarting a gateway loses open
streams and nothing else.

`@better-compute/gateway` implements it in TypeScript on `node:http2`. It depends on `core`, `protocol`,
`inventory`, `pki` and the server's types only; the server never imports it.

## Running a gateway [#running-a-gateway]

`better-compute serve` runs the gateway role from the deployment's configuration; see
[deployment](/docs/operations/deployment). Embedded in your own process:

```ts title="gateway.ts"
import { createGateway, serveGateway, watchTlsFiles } from '@better-compute/gateway';

const gateway = createGateway({
  region: 'eu-1',
  database, // the compute store: postgres() or pglite()
  services: compute.gatewayServices, // JoinService, renewal, secret broker
  work: compute.context.work,
  events: compute.context.events,
  revocations, // pki RevocationSet: refused at open, pushed to live sessions
  reports: gatewayReportHandlers, // allocation transitions from @better-compute/server
});
await gateway.start(); // directory row and the gateway:<id> lease
const served = await serveGateway(gateway, {
  port: 443,
  tls: { key, cert, ca: nodeCaPem },
  nodeRoots: [nodeCa.certificate], // certificateAuthenticate; there is no weaker default
  revocations,
});
// A renewed serving certificate (pki issue-gateway) reaches new handshakes; open sessions keep theirs.
watchTlsFiles({ key: '/etc/better-compute/tls/gateway.key', cert: '/etc/better-compute/tls/gateway.crt' }, (tls) =>
  served.setTls(tls),
);
```

`serveGateway`, `serveGatewayWs` and `serveInternal` each return `setTls({ key, cert, ca? })`, a hot reload of the
listener's certificate over its original TLS options; a key that does not match its certificate throws and changes
nothing. `watchTlsFiles` (from `@better-compute/gateway`) polls the files every 60 seconds by default and reports a
broken set to `onError`. `serve` does this for you with `host.gateway.tlsFiles`; see
[PKI](/docs/operations/pki#pki-issue-gateway).

| Option                                | Default                        | Meaning                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                                  | a fresh `gw_...` id            | One per process                                                                                                                                                                                                                                                                                                                                          |
| `region`                              |                                | Required                                                                                                                                                                                                                                                                                                                                                 |
| `internalUrl`                         |                                | Where the api reaches this gateway's control API (`gateways.internal_url`)                                                                                                                                                                                                                                                                               |
| `maxStreams`                          | 5,000                          | Sessions this gateway admits                                                                                                                                                                                                                                                                                                                             |
| `heartbeatIntervalMs`                 | 10,000                         | Sent in `Welcome`                                                                                                                                                                                                                                                                                                                                        |
| `sessionLeaseTtlMs`                   | 40,000                         | The daemon liveness lease, sent in `Welcome`                                                                                                                                                                                                                                                                                                             |
| `credits`                             | control 8, provision 4, bulk 2 | Commands in flight per node and lane                                                                                                                                                                                                                                                                                                                     |
| `handshakesPerSecond`                 | 200                            | Session opens and joins admitted per second                                                                                                                                                                                                                                                                                                              |
| `maxPendingHellos`                    | 64                             | Hello transactions in flight at once                                                                                                                                                                                                                                                                                                                     |
| `protocol`                            | `PROTOCOL_VERSION` (1..1)      | The `{ min, max }` protocol range this gateway speaks, for skew tests and staged bumps; see [version skew](/docs/daemon/protocol#version-skew)                                                                                                                                                                                                           |
| `duplicateHellos`                     | `{ max: 3, windowMs: 600000 }` | More than `max` duplicate Hellos (a second live session from another boot) within `windowMs` quarantine the certificate; see [duplicate identity](/docs/daemon/sessions#duplicate-identity)                                                                                                                                                              |
| `gatewayLeaseTtlMs`, `leaseRenewMs`   | 20,000, 5,000                  | The `gateway:<id>` lease                                                                                                                                                                                                                                                                                                                                 |
| `livenessFlushMs`                     | 15,000                         | How often ages are flushed                                                                                                                                                                                                                                                                                                                               |
| `dispatchIntervalMs`, `dispatchBatch` | 1,000, 1,000                   | The dispatch loop                                                                                                                                                                                                                                                                                                                                        |
| `pokeWindowMs`                        | 10                             | Minimum spacing of poke-driven dispatch runs                                                                                                                                                                                                                                                                                                             |
| `specSweepMs`                         | 10,000                         | How often open sessions are checked for a newer generation                                                                                                                                                                                                                                                                                               |
| `busyBackoff`                         | 1,000 to 30,000 ms             | Back-off for commands answered `BUSY`                                                                                                                                                                                                                                                                                                                    |
| `trustBundle`                         |                                | A function returning the pre-signed trust bundle for `Welcome`, read per session                                                                                                                                                                                                                                                                         |
| `revocations`                         |                                | Revoked serials; without it, `start()` warns that revoked certificates keep their sessions until they expire                                                                                                                                                                                                                                             |
| `tunnels`                             |                                | Exec-ticket verification and re-authorization (v0.3)                                                                                                                                                                                                                                                                                                     |
| `reports`                             |                                | Report and status handlers, below                                                                                                                                                                                                                                                                                                                        |
| `commandOwner`                        |                                | Maps a command's idempotency key (`<reconciler>:<key>:…`) to the work item that owns it, which is woken when the command settles, so a reconciler waiting on a result sees it at once. `better-compute serve` and the dev stack pass `settledCommandOwner(compute)` (exported from the CLI), which accepts only reconcilers registered in the deployment |
| `manual`                              | false                          | No timers: tests and the simulator step `gateway.jobs` themselves                                                                                                                                                                                                                                                                                        |

## Memory and PostgreSQL [#memory-and-postgresql]

| In memory, lost on restart                                           | In PostgreSQL                                                                        |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Open sessions: node, epoch, role, boot and journal id, peer identity | `node_sessions`: routing, epoch, boot and journal id, event ack, last seen, versions |
| Last-seen time per session                                           | Flushed as ages every 15 s                                                           |
| Lane counters and in-flight command ids                              | `commands`: state, epoch, attempt, result                                            |
| The generation last sent per session                                 | `nodes.generation`, `nodes.spec`, `nodes.observed_generation`                        |
| The latest status report per session                                 | Written about once a second                                                          |
| `BUSY` counts, cancels already sent                                  | `commands.not_before`, `cancel_requested_at`                                         |
| Open tunnels                                                         | nothing: a tunnel dies with its gateway                                              |
| The lease token                                                      | `leases` row `gateway:<id>`; `gateways` directory row                                |

Everything in the left column is rebuilt by the next `Hello`, which is why a gateway crash costs a reconnect and
nothing more; see [failure modes](/docs/daemon/failure-modes#gateway-crash).

The `gateways` table is the directory: id, region, `internal_url`, software version, `max_streams`, current
`sessions`, `draining`, `started_at` and `heartbeat_at`. `start()` upserts the row, and every lease renewal updates
the session count and drain flag, which `doctor` reads for headroom.

## The loops [#the-loops]

| Job        | Period            | Does                                                                                                                                                        |
| ---------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lease`    | 5 s               | Renews `gateway:<id>`; fences the gateway when it cannot                                                                                                    |
| `dispatch` | 1 s, and on pokes | Sends pending commands; at most once per interval, sends `Cancel` for in-flight commands whose cancel was requested                                         |
| `liveness` | 5 s               | Closes silent sessions (three missed heartbeats) and sessions with expired certificates; sweeps tunnels; every 15 s, flushes ages and drops zombie sessions |
| `specs`    | 10 s              | Pushes a full spec to sessions whose node's generation moved past what they hold and were sent                                                              |
| `status`   | 1 s               | Writes each session's latest status report and runs the status handlers                                                                                     |

`gateway.status()` reports each loop's consecutive failures, total failures, last clean run and last error, so a
stuck loop can be alerted on. `GET /v1/health` on the control API returns the same object.

## Dispatch by join [#dispatch-by-join]

Routing is a join between `commands` and `node_sessions` at dispatch time, never a column copied onto commands:

```sql
SELECT c.*, s.session_epoch FROM commands c JOIN node_sessions s USING (node_id)
 WHERE s.gateway_id = $me AND c.state = 'pending'
   AND c.not_before <= compute_now() AND c.not_after > compute_now()
 ORDER BY c.node_id, c.seq LIMIT 1000
 FOR UPDATE OF c SKIP LOCKED FOR SHARE OF s
```

`SKIP LOCKED` lets several gateways and runs work side by side, and the `FOR SHARE` lock on the session row makes a
dispatch and a concurrent `Hello` epoch bump serialize. A row is marked `dispatched` only when its session epoch
matches the gateway's live session and the lane has credit; a session that closed after commit leaves the row
`dispatched`, and the next `Hello` replays it. Because routing is a join, a reconnect needs no rewrite and a command
enqueued during a reconnect cannot be stranded. Enqueues poke the gateway after commit; pokes are coalesced, so a
burst costs a few dispatch passes, and a `Hello` pokes only when the node has pending commands, so a reconnect storm
does not turn every `Hello` into a pass over every session.

Delivery rules (credits, cancel, refusals) are in [delivery](/docs/daemon/delivery#dispatch).

## Group commit [#group-commit]

A fleet sends many small frames. Writing each in its own transaction would serialize the gateway on the database,
so command updates, event batches and inventory work are group-committed: frames from every session that arrive in
the same I/O turn share one transaction, at most 500 entries each. A session still waits for its own entry before
its next frame is handled, so per-session order holds, and a lone frame never waits for company.

Update transactions lock every command involved in id order, so two concurrent flushes never deadlock; event
transactions lock session rows in node-id order. When a batch fails, each entry is retried alone, so one bad frame
cannot fail the others. Inventory sections are not awaited by the frame queue at all: a section is an idempotent
upsert, so the sections of every session in one flush (a node's sections sent back to back after `Welcome`, and
other nodes' at the same time) share one transaction. They are ingested set-wise through the inventory port's
`ingestSections` (inventory's `ingestInventorySections`): one `INSERT … SELECT FROM unnest` for the whole batch and
one `enqueueMany` of the distinct nodes, where a corrupt section or an unknown node fails only its own slot. Heartbeat
hash reports, and sections when the port has no `ingestSections`, still run each under its own savepoint. A section
lost to an error is simply requested again at the next `Hello`.

## The lease and fencing [#the-lease-and-fencing]

Each gateway holds a `leases` row named `gateway:<id>`, renewed every 5 seconds with a 20-second TTL. The epoch of
the lease rises whenever its owner changes. Every routing transaction (the `Hello` transaction, dispatch) starts
by re-reading the lease `FOR SHARE`:

```sql
SELECT 1 FROM leases
 WHERE name = $name AND owner = $me AND epoch = $epoch AND expires_at > compute_now()
FOR SHARE
```

A zombie either still sees its lease as valid, or aborts; a takeover blocks until the zombie's transaction ends. If
another owner holds the lease, or renewals have failed for half the TTL (10 seconds), the gateway fences itself:
it forgets its lease and closes every stream `UNAVAILABLE` ("gateway fenced; reconnect"). Daemons redial, possibly to
another gateway. Separately, the per-node session epoch fences routing and liveness writes; see
[epochs and fencing](/docs/daemon/sessions#epochs-and-fencing).

`stop()` closes every session `UNAVAILABLE` with a disconnect write, marks the gateway draining and releases the lease
so a successor need not wait for the TTL. `drain()` sends every daemon `Drain` with a random delay up to 60 seconds
and refuses new sessions, for rolling deploys: PLAN.md upgrades gateways one at a time with `Drain`.

## Report and status hooks [#report-and-status-hooks]

The gateway imports no domain module. State transitions driven by what nodes report are injected by the host:

| Hook                         | Called with                  | Typical wiring                                                                                                      |
| ---------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `reports.allocationBound`    | an `AllocationBound`         | capacity's `markAllocationBound`: the binding and the signed report are stored for the reservation worker to verify |
| `reports.allocationReleased` | an `AllocationReleased`      | capacity marks the binding lost or released                                                                         |
| `reports.allocationFailed`   | an `AllocationFailed`        | capacity retries or fails the binding                                                                               |
| `reports.status`             | the coalesced `StatusReport` | `xidStatusHandler` (XID conditions to inventory's `recordXid`), VPS instance status                                 |

Each report hook runs in the transaction that stores the event and advances the ack, so a report is applied exactly
once, and receives `{ nodeId, tenantId, journalId, seq, bootId, signed, attested, work }`. `signed` says whether a
`node_sig` came with it; `attested` is the signed envelope as received (the `AttestedReport` body bytes, `kid` and
`sig`) for the worker to verify. Workers verify signatures; the gateway never treats one as proof. Status handlers
may be a list, each run under its own savepoint. A failing hook rolls back to its savepoint and is published as
`node.report-failed`; ingestion goes on. `@better-compute/server` exports `gatewayReportHandlers` with the capacity
hooks, and `better-compute serve` always adds `xidStatusHandler`.

`gatewayReportHandlers` accepts a report only about an allocation placed on the reporting node: a report naming
another node's allocation is ignored like an unknown allocation, so node A's session can neither bind, release nor
fail node B's. `AllocationBound` passes the session's node id and the signed envelope to `markAllocationBound`,
which stores the report as received (`allocations.bound_report`, `bound_sig_kid`, `bound_sig`); with
`attestedBindings` the reservation worker verifies it before the reservation goes active. See
[attested bindings](/docs/guides/capacity/reservations#attested-bindings).

## Certificate authentication [#certificate-authentication]

`Join` accepts clients without a certificate. `Renew`, `Session` and `Tunnel` need one that the authenticator
accepts, and the node id always comes from the certificate. `serveGateway` requires either `nodeRoots` (for the
production `certificateAuthenticate`) or an explicit `authenticate` hook; anything else is `INVALID_CONFIG`.

`certificateAuthenticate` verifies the chain to the node CA with the `clientAuth` usage, refuses revoked serials,
optionally requires a SPIFFE trust domain, and reads the node id and site from the single SPIFFE URI SAN. It returns
the serial, the role and the expiry, which drive `node_certs.allowed_kinds`, revocation, the `Hello` role check and
the close at expiry, plus `tenantControlled` for a `node-tenant` certificate: such a session gets no secrets and no
destructive platform commands. `unsafeSanOnlyAuthenticate` reads only the SAN and skips all of that; it is for tests,
as is `insecure: true` (cleartext h2c).

## Revocation [#revocation]

Revocations reach live sessions within about a second. In `better-compute serve`, pki's `createRevocationFeed`
polls `node_certs.revoked_at` into the revocation set the gateway enforces, and the gateway subscribes to it: a
session whose certificate serial is revoked receives `RevocationNotice{reason, serials}`, its tunnels close, and the
stream ends `UNAUTHENTICATED`. New opens with that serial are refused at the handshake, and the `Hello` transaction
re-checks `node_certs` itself, since the database is the revocation truth and the in-memory feed may lag a poll.
The control API's `revoke` does the same for one node on demand. The gateway also revokes by itself in one case: a
certificate that keeps alternating between two machines is quarantined with reason `duplicate-identity` (see
[duplicate identity](/docs/daemon/sessions#duplicate-identity)).

## Control API [#control-api]

Each gateway serves a small internal API on `gateways.internal_url`, HTTP/2 over the internal CA's mTLS, called by
api and worker processes. Every call but tunnels is an accelerator: dispatch polls every second, specs are re-pushed
by the sweep, revocations arrive through the feed and cancellations from the database, so correctness never depends
on it.

| Call                           | Body                                                                                                  | Answer                                                                                     |
| ------------------------------ | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `GET /v1/health`               |                                                                                                       | `GatewayStatus`: id, region, started, draining, sessions, `maxStreams`, lease epoch, loops |
| `POST /v1/poke`                | `{ nodeIds? }`                                                                                        | 202; dispatch now, and a spec check for the named nodes                                    |
| `POST /v1/revoke`              | `{ nodeId, reason, serials?: hex[] }`                                                                 | `{ revoked }`                                                                              |
| `POST /v1/drain`               | `{ reason? }`                                                                                         | 202                                                                                        |
| `POST /v1/tunnels/close`       | `{ tunnelId?, nodeId?, subject?, agentId?, delegationId?, sandboxId?, tenantId?, anyAgent?, reason }` | `{ closed }`; the filter must name an id key, or `tenantId` with `anyAgent: true`          |
| `POST /v1/tunnels/reauthorize` | the same filter                                                                                       | `{ checked }`; re-authorizes the matching tunnels now                                      |
| `POST /v1/tunnels`             | a duplex stream                                                                                       | The api's side of one tunnel                                                               |

Host wiring uses three helpers. `gatewayControlClient(...)` calls one gateway. `gatewayNotifier(...)` implements the
core `Notifier`: `publish('gw:<id>')` pokes that gateway, and the runtime publishes it after an enqueue or a spec
change commits. `gatewayTunnelRelay(...)` is the server's `TunnelRelay`: it opens a tunnel through the gateway named
by `node_sessions.gateway_id`, caching gateway URLs for 30 seconds; an unregistered gateway is `UNAVAILABLE`.

## Tunnels (v0.3) [#tunnels-v03]

The api authorizes the caller and issues a 60-second single-use exec ticket for one node, then asks the owning
gateway to open a tunnel. The gateway verifies the ticket (signature against the api's JWKS, audience, its own
`jti` set, which fails closed when full), sends `OpenTunnel` on the node's session, and waits up to 10 seconds for the
node's `Tunnel` stream. The daemon verifies the ticket again before opening it (bcd and the FakeDaemon both do, each
with its own `jti` set), and a daemon that refuses simply opens nothing. The gateway then only relays frames,
with backpressure in both directions. It re-authorizes the caller every 60 seconds: a refusal closes the tunnel, and
2 minutes of IAM failures close it too. A tunnel lives at most 12 hours, and revocation or
`POST /v1/tunnels/close` ends it at once. See [tunnels](/docs/guides/access/tunnels).

## Admission and scale [#admission-and-scale]

Three bounds protect a gateway from a reconnect storm: the handshake token bucket (200 opens and joins per second),
`maxStreams`, and `maxPendingHellos`. The last one exists because the bucket can admit `Hello`s faster than the
database commits them; an unbounded queue once starved the lease renewal until the gateway fenced itself and dropped
every stream. With the bound, a refused `Hello` retries with jitter and the database queue stays short.

Each of these refusals (`RESOURCE_EXHAUSTED`) tells the daemon when to come back, so a herd spreads by the gateway's
measure rather than by each daemon's doubling back-off. `GrpcError` carries `retryAfterMs`, sent as the standard
`grpc-retry-pushback-ms` trailer:

| Refusal                              | `retryAfterMs` (jittered)                                                |
| ------------------------------------ | ------------------------------------------------------------------------ |
| Handshake bucket (`Session`, `Join`) | 250 ms up to refusals this second / `handshakesPerSecond` (at least 1 s) |
| `maxPendingHellos`                   | 250 ms to 2 s                                                            |
| `maxStreams`                         | 5 to 30 s                                                                |

The same hint reaches every transport: the HTTP/2 trailers-only or trailing status; the WebSocket upgrade refusal,
as `retry-after` (seconds) and `grpc-retry-pushback-ms` headers, which `connectGatewayWs` turns into a
`RATE_LIMITED` `ComputeError` with `details.retryAfterMs` (from `grpc-retry-pushback-ms`, else `retry-after`); and
`attachFrameConn`, whose `close(reason, { retryAfterMs })` hands it to simulated FakeDaemons. bcd and the FakeDaemon
wait exactly that long; see [opening a session](/docs/daemon/sessions#opening).

PLAN.md's v0.1 gate is one Node gateway holding at least 5,000 mTLS sessions at 10-second heartbeats for 30 minutes,
with dispatch p99 under 500 ms and RSS under 2 GiB; if it fails, the gateway is rewritten in Go on the same proto
(decision D20). On a development workstation (PGlite, 60 seconds), the gateway in its own process held 5,000 mTLS
sessions, one TLS 1.3 connection per daemon, with dispatch p99 26 ms, a gateway process RSS of 1,240 MiB (PGlite
included) and 0 drops, and survived a reconnect storm with 5,000 of 5,000 back within 39 seconds. A 30-minute run of
the same split setup (14ff010) held 5,000 sessions with dispatch p99 24 ms over 90,000 commands, 0 drops and every
command of the storm `succeeded`, but the gateway process's RSS grew from 1,240 to 2,016 MiB while the JS heap stayed
at 235 MiB; a 5-minute run with memory attributed traced the growth to PGlite's WebAssembly memory inside the gateway
process, not the gateway's own. Every
criterion is met on PGlite, and the gateway's own RSS stays near 1.1 GiB; the run with Postgres out of process, on
the reference host, is still to confirm it. See
[scaling](/docs/operations/scaling#gateway-sessions) and `bench/gateway-streams`.

### Several gateways [#several-gateways]

A region runs three or more gateways behind the L4 load balancer. Nothing is shared between them but PostgreSQL:
each dispatches only the nodes routed to it (`node_sessions.gateway_id`), and a node that reconnects elsewhere
simply moves its row with the next `Hello`. In the fleet simulator with the real control plane
(`bench/multi-gateway`), killing one of three gateways re-homed its nodes with p99 of 1 second at 1,000 nodes and
13.25 seconds at 10,000, with no invariant violations. Peering between gateways is planned for v0.4.

## Metrics [#metrics]

With a metrics registry, `better-compute serve` wraps the gateway services with `meteredGatewayServices` and
registers `gatewayCollector`, which export (all prefixed `better_compute_`, no tenant labels):

| Metric                                                                                           | Labels              | Meaning                                    |
| ------------------------------------------------------------------------------------------------ | ------------------- | ------------------------------------------ |
| `gateway_open_sessions`                                                                          | `gateway`, `role`   | Open sessions by agent role                |
| `gateway_stream_headroom`                                                                        | `gateway`           | Streams the gateway can still admit        |
| `gateway_draining`                                                                               | `gateway`           | 1 while draining                           |
| `gateway_lease_held`                                                                             | `gateway`           | 1 while it holds its lease, 0 while fenced |
| `gateway_commands_in_flight`                                                                     | `gateway`           | Commands in flight on its sessions         |
| `gateway_sessions_upgrade_required`                                                              | `gateway`           | Sessions told to upgrade                   |
| `gateway_loop_consecutive_failures`, `gateway_loop_failures`, `gateway_loop_last_ok_age_seconds` | `gateway`, `loop`   | Loop health                                |
| `gateway_service_calls_total`                                                                    | `method`, `outcome` | `Join`, `Renew` and secret calls           |
| `gateway_service_seconds`                                                                        | `method`            | Their duration                             |

The collector samples `gateway.status()` and `gateway.sessions()` at every scrape. See
[observability](/docs/operations/observability).

## Testing without a network [#testing-without-a-network]

`gateway.open(peer, sink)` drives a session without any transport, and `attachFrameConn(gateway, peer, conn)` serves
one over the testing kit's in-memory frame stream, so the fleet simulator runs real FakeDaemons against a real
gateway on simulated time. `gateway.crash()` simulates process death: loops stop and every session is forgotten
without a database write, so the lease simply expires. See [testing](/docs/operations/testing).

With a network, `createDevStack({ externalTls })` serves the same gateway a second time over mTLS, with a dev serving
CA and client certificates checked against the dev node CA, for daemons outside the test process.
`tests/e2e/bcd.test.ts` runs the real Go bcd against it (see [the Go daemon](/docs/operations/testing#the-go-daemon)).


# GPUs on a node (/docs/daemon/gpus)

> How bcd reads a node's NVIDIA GPUs through nvidia-smi, merges them into the inventory, keeps passed-through GPUs, builds CDI specs from NVIDIA's, handles persistence mode, XIDs, diag.nvml and gpu.reset, what works on WSL 2, and what bcd doctor checks.



bcd manages NVIDIA GPUs only, and it talks to them through one tool: `nvidia-smi`, which ships with every NVIDIA
driver. There is no NVML library binding in the binary. This page follows a GPU through bcd: how it is read, how it
enters the inventory, how a GPU passed through to a VM stays there, how bcd builds the
CDI specs that hand GPUs to containers, what it does with persistence mode, XIDs and resets, and
what is different on WSL 2. Holding GPUs for a reservation is on [devices and holds](/docs/daemon/devices-and-holds);
the host units that install the driver, partition MIG and set up MPS are on
[host configuration](/docs/daemon/host-config); what the control plane does with GPU names, atoms and health is on
[inventory and health](/docs/guides/nodes/inventory-and-health).

> **One real GPU so far.** 
  The parsers are tested against captured `nvidia-smi` output, and since round 6 (2026-10-04) against one real card: a
  GeForce RTX 5090 under WSL 2, driver 616.56, CUDA 13.4 (0915ab7). No data-center GPU, no MIG, no NVLink, no
  passthrough and no CDI container has been run on real hardware. See
  [running bcd on real hardware](/docs/daemon/real-hardware#what-has-run-on-real-hardware).

## Where GPU facts come from [#where-gpu-facts-come-from]

A host build of bcd collects the inventory in two layers (`collectInventory` in `daemon/cmd/bcd/hardware_host.go`):

1. **procfs and sysfs.** `inventory.Collector` lists `/proc/driver/nvidia/gpus/*/information`: each GPU's UUID,
   model, VBIOS and bus id, its NUMA node from `/sys/bus/pci/devices/<bus>/numa_node`, a driver version from
   `/proc/driver/nvidia/version`, and an architecture guessed from the model name. The GPUs are indexed in bus
   order, as `nvidia-smi` numbers them by default.
2. **nvidia-smi.** `nvml.SMI.Snapshot` reads everything else, within one minute for the whole read, and
   `nvml.Enrich` merges it into the first layer by UUID (below).

A host without `nvidia-smi` on the `PATH` has no NVIDIA driver to manage: the snapshot fails with
`nvml: nvidia-smi not found (no NVIDIA driver on this host)` and the inventory keeps whatever procfs showed. Only the
second layer fills the CUDA driver version, so bcd uses that field to tell the two apart: without it, `Hello` does not
advertise the `nvml` capability and [`diag.nvml`](#diagnvml) fails its `nvml.init` check.

### The nvidia-smi queries [#the-nvidia-smi-queries]

A snapshot runs these, each bounded to 30 seconds (`SMI.Timeout`; a wedged GPU can hang `nvidia-smi`):

| Query                                                | Gives                                                                                                                                                                                  | When                                   |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| `nvidia-smi -q -x`                                   | Identity, framebuffer memory, ECC mode and counters, row remapping, MIG mode and instances, power readings, clocks, fabric cluster and clique, persistence and compute mode, processes | Always. Its failure fails the snapshot |
| `nvidia-smi -L`                                      | Each MIG device's UUID and profile, matched to the `-q -x` instance by its device number                                                                                               | Always; optional                       |
| `--query-gpu=uuid,compute_cap --format=csv,noheader` | CUDA compute capability                                                                                                                                                                | Always; optional                       |
| `nvidia-smi topo -m`                                 | The topology matrix (`NV<n>`, `PIX`, `PXB`, `PHB`, `NODE`, `SYS`/`SOC`)                                                                                                                | With more than one GPU; optional       |
| `nvidia-smi nvlink -s`, then `nvlink -R`             | Link states, then each link's far end: a GPU of this node by bus id, else `nvswitch:<bus>`                                                                                             | Always; optional                       |
| `nvidia-smi mig -lgip`                               | The MIG profiles each GPU offers                                                                                                                                                       | When a GPU has MIG enabled; optional   |

Only the `-q -x` query is required. An optional query that fails leaves its fields empty, which is how a driver that
answers "not supported here" with a non-zero exit is handled: `topo -m` exits 3 on WSL and `mig -lgip` exits 6 on a
GPU without MIG, and the captured RTX 5090 fixtures replay exactly that.

### What the parser tolerates [#what-the-parser-tolerates]

`nvidia-smi`'s XML changes between driver branches and SKUs. The parser in `daemon/internal/nvml/smi.go` reads these
cases, most of them found on the real RTX 5090 (0915ab7):

| Case                                                                                                                                  | Reading                                                                                                                                                         |
| ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `N/A`, `[N/A]`, `Not Supported`, `[Not Supported]`, `Unknown Error`, `None`, and R616's `Requested functionality has been deprecated` | No value: the field is left empty                                                                                                                               |
| Serial `0` (or all zeros), as GeForce and most workstation boards answer                                                              | No serial, rather than one serial every such GPU shares                                                                                                         |
| `kmd_version` and `cuda_umd_version` (R616 deprecates `driver_version` and `cuda_version`)                                            | The new names win when present                                                                                                                                  |
| `gpu_power_readings` (R535 and later) or `power_readings`                                                                             | Draw from `instant_power_draw`, else `power_draw`, else `average_power_draw`; limit from `current_power_limit`, else `enforced_power_limit`, else `power_limit` |
| ECC counters as `sram_*`/`dram_*` (R525 and later) or `single_bit`/`double_bit` totals                                                | Both layouts sum to correctable and uncorrectable counts                                                                                                        |
| A fabric cluster UUID of zeros                                                                                                        | No fabric                                                                                                                                                       |
| `Exclusive_Process` compute mode                                                                                                      | `exclusive-process`                                                                                                                                             |
| The DTD the XML declares                                                                                                              | Not loaded (the decoder is not strict)                                                                                                                          |

### Live samples [#live-samples]

The inventory holds facts that only change with the hardware: identity, memory size, ECC mode and counters, row
remapping, MIG layout, NVLink, topology. Usage is a **sample**, and never enters the inventory hash, so a busy GPU
does not resend its section every heartbeat.

A full snapshot runs `nvidia-smi` up to six times and `-q -x` walks every field of every GPU, which takes seconds of
driver time on an 8-GPU board. Since 9a6fee9 the readers that sample often use `SMI.Samples` instead: one CSV query
for exactly a sample's fields, plus one for the compute processes.

```bash
nvidia-smi --query-gpu=uuid,memory.total,memory.used,memory.reserved,memory.free,utilization.gpu,utilization.memory,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,pstate,mig.mode.current --format=csv,noheader,nounits
nvidia-smi --query-compute-apps=gpu_uuid --format=csv,noheader
```

`Samples` falls back to the full snapshot when a GPU is in MIG mode (a MIG device's memory is in `-q -x` only), when
the driver refuses a field it predates (`memory.reserved` before R510), or when a line has the wrong number of
columns. A host without `nvidia-smi` gets `ErrNoDriver`, not a fallback.

| Reader                                                                       | Uses                                                                                               |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| The OTLP export of the `telemetry` section (every 5 seconds at the shortest) | `Samples` (`gpuSampler`); see [node metrics](/docs/operations/observability#node-metrics-from-bcd) |
| `bcd status`, one line per GPU                                               | `Samples`, waited for at most 10 seconds; see [bcd status](/docs/daemon#bcd-status)                |
| `bcd doctor`                                                                 | The full `Snapshot` (`gpuSnapshot`), for the health fields                                         |
| The inventory                                                                | The full `Snapshot`                                                                                |

A sample's process count is **compute processes only**: in the `-q -x` path, entries whose type contains `C` (`C`,
`C+G`, `M+C` for MPS clients); in the CSV path, the lines of `--query-compute-apps`. A desktop compositor or WSL's
Xwayland is a graphics process and no workload. The live test checks that both paths agree on the real GPU.

## The inventory merge [#the-inventory-merge]

`nvml.Enrich(base, snapshot)` merges by UUID:

* The section's driver, CUDA driver and NVML versions are replaced by the snapshot's when it has them.
* A GPU both layers know takes every field from `nvidia-smi`, but keeps the collector's index, and the collector's
  NUMA node and architecture when `nvidia-smi` gave none (it reports no NUMA node at all).
* A GPU only `nvidia-smi` knows is appended with the next index. That is the WSL 2 case: the paravirtual GPU has no
  `/proc/driver/nvidia` entry and no PCI sysfs, so the whole section comes from `nvidia-smi`, its bus id names a
  virtual bus (`00000000:F1:00.0` in the fixture) and it has no NUMA node.

The result is canonicalized and hashed like every other section. bcd collects the inventory again every 10 minutes
(`DefaultInventoryInterval`), since ECC counters, row remapping and MIG state change without an event bcd sees, about
2 seconds after an XID burst, after a host unit changed something, and on `inventory.refresh`.

### GPUs passed through to VMs [#gpus-passed-through-to-vms]

A GPU a VM holds is bound to `vfio-pci`. The NVIDIA driver lets go of it, so neither `/proc/driver/nvidia` nor
`nvidia-smi` lists it any more, and an inventory read from them would drop the very GPU the VM uses: the control plane
would retire its device mid-allocation, and a VM restarting would no longer find its GPU's PCI address. Since c5c52c4
bcd keeps them:

* **Remember.** After every enriched collection, `inventory.GPUMemory` records each GPU by its PCI address in
  `<data_dir>/gpus-seen.json` (0600, written atomically): UUID, name, architecture, VRAM, serial, VBIOS, board part
  number and compute capability. GPUs the collection did not list are kept, since they may be passed through right
  now.
* **Find.** `Collector.PassedThrough` walks `/sys/bus/pci/devices` for NVIDIA functions (vendor `0x10de`) of class
  `0x0300xx` (VGA) or `0x0302xx` (3D controller) bound to `vfio-pci` that the driver did not list, and reports the
  GPU last seen at that address, with its NUMA node from sysfs.
* **Order.** They come after the driver's GPUs, so the driver-bound GPUs keep the indices `nvidia-smi` gives them.
* **Limits.** A function never seen on the NVIDIA driver is left out: without a UUID there is nothing to name it by.
  A remembered GPU carries only what stays true while a VM has it, not MIG, NVLink or topology state, which the
  guest's driver owns meanwhile.

Only `bcd node` keeps the memory (`newHost` and `newRuntimes` point it at the data directory). `bcd hardware` and
`bcd doctor` collect without it, so they do not list a GPU that is passed through at that moment. The runtimes'
`GPUAddress`, which resolves a GPU's UUID to its PCI address for VFIO, reads through the same memory.

## CDI: NVIDIA's spec and bcd's [#cdi-nvidias-spec-and-bcds]

Containers get GPUs through CDI specs in `/etc/cdi`. Two kinds live there on a bcd GPU host:

| Spec                                                 | Written by                                                                                                                                      | Names devices                                                                                                        |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `/etc/cdi/nvidia.json`, kind `nvidia.com/gpu`        | The [`container-toolkit` unit](/docs/daemon/host-config#container-toolkit): `nvidia-ctk cdi generate --format=json --device-name-strategy=uuid` | By GPU or MIG device UUID, with their device nodes, the driver's libraries as mounts and the `nvidia-cdi-hook` hooks |
| `/etc/cdi/bcd-<allocation>.json`, kind `bcd.dev/gpu` | The `devices` converger, for each hold                                                                                                          | By atom id, `bcd.dev/gpu=<id>`                                                                                       |

With the toolkit in CDI mode, `NVIDIA_VISIBLE_DEVICES` grants nothing, and NVIDIA's own device names are not tied to
any hold. So since 6ecb860 bcd copies NVIDIA's edits into each hold's spec (`agent/cdivendor.go`):

* bcd reads every `*.json` spec of kind `nvidia.com/gpu` in `/etc/cdi`, then `/var/run/cdi`, skipping its own
  `bcd-*` files. A YAML spec (the GPU operator's, or the `nvidia.yaml` older bcd versions wrote) is not read, and a
  hold then carries only `NVIDIA_VISIBLE_DEVICES`, as before 6ecb860.
* A hold's spec gets the vendor spec's common edits (device nodes such as `/dev/nvidiactl` and `/dev/nvidia-uvm`,
  library mounts, hooks, additional GIDs) once, without the vendor's `NVIDIA_VISIBLE_DEVICES=void`, and each held
  device gets exactly the edits of the device it stands for: a whole GPU or a slot its GPU's, a MIG instance its
  `MIG-…` device's. Another GPU's device node never leaks in.
* The hold spec's `cdiVersion` rises to the vendor's when that is later (additional GIDs need 0.7.0).
* A held device the vendor spec does not name is logged ("NVIDIA's CDI spec does not name held devices"): the spec
  is stale until the toolkit unit regenerates it, which it does whenever the driver version or `nvidia-smi -L`
  changes.

`bcd.dev/gpu=<id>` alone is therefore all a container needs. A Kubernetes binding is left alone: its spec carries no
vendor edits, since the kubelet device plugin names NVIDIA's devices itself. The spec's full contents are on
[devices and holds](/docs/daemon/devices-and-holds#the-cdi-spec).

## Persistence mode around passthrough [#persistence-mode-around-passthrough]

The [`nvidia-driver` unit](/docs/daemon/host-config#nvidia-driver) keeps `nvidia-persistenced` running, which holds
every GPU open in persistence mode, and `bcd doctor` warns when persistence is off. But the NVIDIA driver's unbind
waits until a GPU is closed, so on a converged GPU host every passthrough start hung until the unbind timed out.
Since f25abbe and 5bb9f22 bcd turns persistence off for exactly the GPUs a VM takes, and on again once they are back
on `nvidia`, with `nvidia-smi -i <pci address> -pm 0` or `-pm 1`:

| Path                                                                        | Off                                                                                                       | On again                                                                     |
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `cloud-hypervisor` (`runtime.SysfsVFIO`)                                    | In `Bind`, before unbinding a GPU from `nvidia`                                                           | In `Release`, for each device the kernel gave back to `nvidia`               |
| `libvirt-kvm`, bcd running `virsh` as root                                  | Before `virsh start` or `virsh restore`, for every PCI hostdev in the domain's XML (`runtime.DomainGPUs`) | After the domain stops, once libvirt's managed hostdevs are back on `nvidia` |
| `libvirt-kvm` under [`bcd helper`](/docs/daemon#libvirt-through-the-helper) | `libvirt.start` and a checked `libvirt.restore`, as root                                                  | `libvirt.destroy`, as root                                                   |

Every call is best effort: a failure is ignored, and the unbind still has its own 30-second timeout
(`SysfsVFIO.UnbindTimeout`), whose error names the likely holder: `its driver has not let go after 30s: something
still has the device open (nvidia-persistenced, a process on the GPU)`. Tests inject the persistence call; nothing
runs the real `nvidia-smi` under a fake `virsh` or a fake sysfs.

No passthrough has run on real hardware, so this sequence is tested against fakes only.

## XIDs and GPU health [#xids-and-gpu-health]

### XIDs from the kernel log [#xids-from-the-kernel-log]

bcd watches XIDs for the life of the process (`watchXids`, `daemon/internal/health`). It opens `/dev/kmsg`, seeks to
its end, and parses every record whose text matches the driver's line:

```text
NVRM: Xid (PCI:0000:18:00): 79, pid=1234, name=python, GPU has fallen off the bus.
```

The bus id is widened to NVML's form (`00000000:18:00.0`) and matched against the inventory's GPUs. A match raises the
device condition `Xid<n>` (status true, `device_id` the GPU's UUID, the driver's text as message), one per XID number
and GPU; an XID on a bus the inventory does not know is only logged. Each XID also:

* asks for an inventory collection about 2 seconds later (a burst is read once), since the GPU's ECC and row-remap
  counters move with it;
* triggers the Kubernetes side, whose device plugins mark the GPU unhealthy and whose DRA slice drops it;
* re-sends the status report at once, with the condition.

The conditions are kept for the rest of the boot in `<data_dir>/faults.json` (0600) with the boot id, so a bcd restart
in the same boot reports them again although it reads the kernel log from its end; the first start after a reboot
finds another boot id and drops them. A [`gpu.reset`](#gpureset) that leaves the GPU healthy drops that GPU's
(662fb55). What the control plane does with each XID (quarantine, NoSchedule, remediation) is on
[inventory and health](/docs/guides/nodes/inventory-and-health#health-quarantines-and-remediation).

PLAN.md's NVML event stream is not implemented: XIDs come only from the kernel log, and SXid lines from NVSwitches are
not parsed.

### Healthy [#healthy]

`gpuHealthy` is the one rule bcd uses for "may this GPU take work":

* no failed or pending row remap;
* no uncorrectable volatile ECC errors;
* no `Xid*` condition for the GPU this boot.

`bcd status` prints `UNHEALTHY` for a GPU that fails it, and the Kubernetes device plugins and DRA driver offer it as
unhealthy.

### diag.nvml [#diagnvml]

`diag.nvml` (control lane, roles `node` and `ramdisk`, attested result) collects the inventory again and checks the
GPUs it lists, or only those in `gpu_uuids`:

| Check               | Fails when                                                                                                                                                                                                                               |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nvml.init`         | The inventory has no CUDA driver version: `nvidia-smi` did not answer ("the NVIDIA driver's management interface (nvidia-smi) did not answer"), or no GPU section at all ("NVML is not available on this node"). Nothing else is checked |
| `gpu.count`         | With `expected_gpu_count`: fewer GPUs visible than expected                                                                                                                                                                              |
| `gpu.present`       | A UUID in `gpu_uuids` the inventory does not list ("not visible to NVML")                                                                                                                                                                |
| `ecc.uncorrectable` | Uncorrectable volatile ECC errors since the driver loaded                                                                                                                                                                                |
| `row-remap.pending` | A row remap waits for a GPU reset                                                                                                                                                                                                        |
| `row-remap.failure` | Row remapping failed: the GPU needs replacing                                                                                                                                                                                            |
| `pcie.link`         | The negotiated PCIe link width (`current_link_width` in sysfs) is below `max_link_width`. Width only: an idle GPU lowers its link speed on purpose, never its width. A GPU without those sysfs files (WSL) never fails it                |
| `xid.recent`        | The GPU has `Xid*` conditions this boot; the message lists them                                                                                                                                                                          |

The result's `passed` is true only when every check of every GPU passed.

### gpu.reset [#gpureset]

`gpu.reset` runs `nvidia-smi --gpu-reset -i <uuid>` per GPU, through `bcd helper`'s `gpu.reset` operation when the
node is unprivileged. It refuses the whole command with `gpu-busy` when a listed GPU, or one of its MIG instances or
slots, has a live hold or a pending Kubernetes release, or is passed through to a VM; it answers `unknown GPU` for a
UUID the inventory does not list; and after the resets it collects the inventory again and marks each reset GPU
`healthy` unless it shows uncorrectable volatile ECC errors or a pending or failed row remap. The full rules are on
[delivery](/docs/daemon/delivery#gpureset-and-live-holds).

## WSL 2 paravirtual GPUs [#wsl-2-paravirtual-gpus]

Under WSL 2 a GPU is a paravirtual device: the Windows host's driver serves it through `/dev/dxg`, CUDA works through
the libraries in `/usr/lib/wsl/lib`, and the Linux side has no NVIDIA kernel module, no `/dev/nvidia*` and no PCI
function for the GPU. bcd recognises it as `/dev/dxg` without `/dev/nvidiactl` (`paravirtualGPU` in host config,
the `gpu-host` check in doctor). It is the machine round 6 ran on.

| Works                                        | How                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Inventory                                    | Entirely from `nvidia-smi` (procfs has no entry): name, UUID, architecture, VRAM, VBIOS, power limit, persistence and compute mode, compute capability. No serial (the board answers `0`), no NUMA node, a virtual bus id, no topology (`topo -m` exits 3), no NVLink, no MIG                                                                                                                                                                      |
| Live samples, OTLP GPU metrics, `bcd status` | Through `Samples`, as on any host; the real-host e2e checks `bcd.gpu.memory.total` against `nvidia-smi`                                                                                                                                                                                                                                                                                                                                            |
| `nvidia-driver` unit                         | Converges when `branch` is the Windows driver's branch (read from `nvidia-smi`). Any other branch is terminal: `nvidia-driver: this is WSL 2's paravirtual GPU, served by the Windows host's driver 616.56; bcd never installs a Linux driver here (set branch 616, or install that driver on Windows)`. bcd never installs or loads a Linux driver there, which would shadow the Windows host's libraries and break every GPU of the VM (2e71a60) |
| `container-toolkit` unit                     | The CDI stamp takes the driver version from `nvidia-smi`, and `nvidia-ctk cdi generate` writes its WSL mode spec: one device, `all`, with `/dev/dxg`                                                                                                                                                                                                                                                                                               |
| Hold specs                                   | The vendor spec's single `all` device stands for each GPU, so a hold's `bcd.dev/gpu=<uuid>` carries `/dev/dxg` and the WSL library mounts                                                                                                                                                                                                                                                                                                          |

| Does not work             | Why                                                                                     |
| ------------------------- | --------------------------------------------------------------------------------------- |
| VFIO passthrough          | No PCI function to bind to `vfio-pci`                                                   |
| `gpu.reset`, MIG, DCGM    | The paravirtual device supports none of them; `bcd doctor`'s `gpu-host` warning says so |
| XIDs                      | They are printed by the NVIDIA kernel module, which does not run in the WSL kernel      |
| `diag.nvml`'s `pcie.link` | No sysfs link width to compare, so the check never fails                                |

The `container-toolkit` behaviour on WSL is tested against a fake host only: the WSL machine round 6 used has no
`nvidia-ctk`, so no WSL CDI spec was generated for real. `TestRealGPUUnitsPlan` planned the unit there and saw it plan
the toolkit install, the runtime configuration and `/etc/cdi/nvidia.json`.

## bcd doctor's GPU checks [#bcd-doctors-gpu-checks]

`bcd doctor` reads every GPU once with the full snapshot and prints these lines (f9d7af9, f84f1f9, da60bbd). The
real-host checks around them (systemd, cgroups, KVM, VFIO, IOMMU groups, the clock) are on
[running bcd on real hardware](/docs/daemon/real-hardware#bcd-doctor-as-preflight).

| Check            | Status               | When                                                                                                                                                                                                                                                |
| ---------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nvml`           | `ok`                 | `nvidia-smi` answered: `driver <version>, <n> GPUs`                                                                                                                                                                                                 |
| `nvml`           | `warn`               | No `nvidia-smi`, or a binary built with `-tags nonvml` (`nvml: not available in this build`)                                                                                                                                                        |
| `nvml`           | `fail`               | `nvidia-smi` failed otherwise                                                                                                                                                                                                                       |
| `hardware`       | `ok`                 | The inventory collected and hashed: `host: <n> GPUs, <m> inventory sections hashed`                                                                                                                                                                 |
| `gpu`            | `ok`, `warn`, `fail` | One line per GPU: UUID, name, busy %, temperature, power draw of its limit, memory used of its total. `warn` for a pending row remap or a MIG mode change waiting for a reset; `fail` for failed row remapping or uncorrectable volatile ECC errors |
| `gpus`           | `fail`               | The snapshot failed; the message carries a hint (below)                                                                                                                                                                                             |
| `gpu-host`       | `warn`               | WSL 2's paravirtual GPU: "CUDA containers work (CDI: nvidia-ctk cdi generate --mode=wsl), but there is no VFIO passthrough, gpu.reset, MIG or DCGM here". On WSL the checks below are skipped                                                       |
| `nouveau`        | `fail`               | `nouveau` is in `/proc/modules`: blacklist it, rebuild the initramfs, reboot                                                                                                                                                                        |
| `gpu-bus`        | `ok`                 | `<n> NVIDIA GPU(s) on the PCI bus: <k> on nvidia`, plus how many are on `vfio-pci` (passthrough)                                                                                                                                                    |
| `gpu-bus`        | `fail`               | More GPUs bound to `nvidia` than `nvidia-smi` sees: a GPU fell off the bus or did not initialize (`dmesg`: Xid 79, `RmInitAdapter failed`)                                                                                                          |
| `gpu-bus`        | `warn`               | GPUs on the PCI bus but no NVIDIA driver answers (install it with the `nvidia-driver` unit), or GPUs with no driver bound at all                                                                                                                    |
| `persistence`    | `warn`               | Persistence mode is off on a GPU `nvidia-smi` sees: every first CUDA call then re-initializes it                                                                                                                                                    |
| `fabric-manager` | `ok`, `fail`         | NVSwitch functions (class `0x0680`) on the bus, and whether `nvidia-fabricmanager` is active; without it CUDA fails on every GPU                                                                                                                    |
| `gpu-binding`    | `warn`               | GPUs, but bcd would run as a non-root user without `[helper]`; see [unprivileged nodes](/docs/daemon/devices-and-holds#unprivileged-nodes)                                                                                                          |

`gpu-bus` compares NVIDIA display functions (class `0x03xxxx`) in `/sys/bus/pci/devices` with the GPUs `nvidia-smi`
lists. A `fakehw` build skips every host check, since it has no real bus to compare.

When the snapshot fails, `gpuErrorHint` adds what to do for the failures seen on real hosts:

| `nvidia-smi` says                             | Hint                                                                                                                                            |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `Driver/library version mismatch`             | The loaded kernel module and the user-space libraries differ, usually after a driver upgrade: reboot, or unload and reload the `nvidia` modules |
| `couldn't communicate with the NVIDIA driver` | The `nvidia` module is not loaded: `modprobe nvidia`; under Secure Boot it must be signed by an enrolled key (see `dmesg`)                      |
| `No devices were found`                       | The driver sees no GPU: compare `lspci -d 10de:` with `dmesg` (`RmInitAdapter failed`, Xid 79)                                                  |
| A timeout or `signal: killed`                 | `nvidia-smi` did not answer: a GPU may be wedged (Xid 79 or 119); `gpu.reset` or a reboot recovers it                                           |

## The NVML backend [#the-nvml-backend]

`daemon/internal/nvml` is named for NVML, but no build links it today. Build tags pick what `nvml.Probe` (doctor's
`nvml` check) does; the inventory and samples go through `nvml.SMI` in every host build:

| Build                                                                                    | `nvml.Probe`                                                    | Inventory and samples                                 |
| ---------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------- |
| Default, as the release builds it (no tags, `daemon/.goreleaser.yaml`)                   | `SMI.Probe`: `nvidia-smi --query-gpu=driver_version`            | `nvidia-smi`                                          |
| `-tags nonvml` (the container image, `tests/e2e/bcd-host-real.test.ts`, the logic tests) | Fails with `ErrNotBuilt`                                        | `nvidia-smi` in a host build; fake GPUs with `fakehw` |
| `-tags fakehw`                                                                           | As above                                                        | The fake HGX node of `internal/fakehw`                |
| A go-nvml (cgo) backend                                                                  | Not built: a TODO in `nvml.go` until the dependency is approved |                                                       |

Both host builds need no cgo and never load `libnvidia-ml`. The costs of going through `nvidia-smi` are that every
read spawns a process (bounded to 30 seconds), and that there are no NVML events: XIDs come from the kernel log
instead.

## Tests and fixtures [#tests-and-fixtures]

| Test                                                      | Against                                                                                                                                                                                                                                        |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `smi_test.go`                                             | `testdata/q.xml`, a two-GPU answer in `nvidia-smi`'s XML (an H100, and an A100 in MIG mode), with hand-written answers to the other queries: every query read, optional queries failing, `Probe`'s count, and `Enrich` keeping the sysfs facts |
| `TestSnapshotRealRTX5090WSL`, `TestSamplesRealRTX5090WSL` | `testdata/rtx5090-wsl`: every query bcd makes, as the real RTX 5090 answered under WSL 2, identifiers anonymized, with the non-zero exits of `topo -m` and `mig -lgip` replayed                                                                |
| `TestSamplesFallBack`                                     | A MIG GPU, a driver refusing `memory.reserved`, short columns, and no driver                                                                                                                                                                   |
| `TestSnapshotLiveHost`                                    | The host's real `nvidia-smi`, only with `BCD_REAL_GPU=1`: every GPU must yield a UUID, name, architecture, VRAM, compute capability and bus id, and the light samples must agree with the full ones                                            |
| `TestRealGPUUnitsPlan` (`internal/hostconfig`)            | Plans `nvidia-driver`, `gpu-settings`, `mig-layout`, `fabric-manager` and `container-toolkit` against the real driver with `BCD_REAL_GPU=1`, applying nothing                                                                                  |
| `TestHoldSpecCarriesNvidiaEdits`, `TestHoldSpecOnWSL`     | Hold specs built from a vendor spec in the shape `nvidia-ctk` writes, and from WSL's                                                                                                                                                           |
| `doctor_gpu_test.go`                                      | The doctor checks over fake `/dev`, `/proc` and `/sys` trees                                                                                                                                                                                   |

```bash
# On a host with an NVIDIA GPU (scripts/daemon-test.mjs forwards BCD_* variables into WSL on Windows):
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestSnapshotLiveHost ./internal/nvml/
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestRealGPUUnitsPlan ./internal/hostconfig/
```


# bcd helper (/docs/daemon/helper)

> How bcd helper runs as root beside an unprivileged bcd node, which callers it serves, the versioned request format, every operation family and what root checks before it acts.



`bcd helper` is the root half of bcd's privilege separation (PLAN.md §6, v0.4; 28dc2ae). With `[helper] socket` and
`[helper] user` set in `config.toml`, `bcd node` runs as an unprivileged account and sends the few things that need
root to `bcd helper`, which runs as root beside it and serves typed operations over a local unix socket. The node
parses everything that comes from the network: the session, commands, specs, tunnels. The helper parses only its own
small requests, and trusts nothing in them.

This page is the reference for the helper itself: who it serves, the wire format, each operation family and the
checks root makes before it acts. The [daemon overview](/docs/daemon#privilege-separation) gives the summary, and
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged) covers the install script that
sets it up.

> **Tested end to end, not on a production host.** 
  The helper is tested in three layers. Go unit tests in `internal/helper` run each operation against fakes (a fake
  `virsh`, fake `systemd-run`, `tpmsim`, a fake host for config units). `tests/e2e/bcd.test.ts` runs an unprivileged
  fakehw `bcd node` beside a real `bcd helper` binary, which writes and removes a GPU hold's CDI spec and swaps in a signed
  self-update. The root end-to-end test (`node scripts/helper-e2e.mjs`, da8751e) runs the install script's `--unprivileged`
  mode and a real `bcd helper` as root, inside private mount and network namespaces in WSL, and drives it as the `bcd`
  user through `bcd helper-call`. Its steps cover signed host units, libvirt definitions against a fake `virsh`,
  `cloud-hypervisor` taps on a real bridge (b9667d5) and self-update swap and rollback as root (035fc41). No hardware
  TPM, real libvirtd or real VMM has run under the helper.

## Why a helper [#why-a-helper]

Without `[helper]`, `bcd node` runs as root under systemd hardening (see
[the systemd unit](/docs/daemon/install#the-systemd-unit)). Everything the agent parses is then parsed by root: the
session protocol, desired state, signed commands, tunnel frames, plugin traffic. A parser bug there gives an attacker
root on the machine.

With the helper, root's attack surface is one socket that takes one JSON request per connection. Each operation is
small and typed: a file is named by a kind and an id rather than a path, a GPU by a UUID the host has, a config unit
by its name and settings, a domain by an instance id. Root rebuilds each privileged action from those values and
its own view of the host, so a compromised node can ask only for what the node could already legitimately ask for.

What the node keeps doing with its own rights: the journal, the identity and key record in `data_dir`, the status
socket, reading `/dev/kmsg` for XIDs, libvirt's read-only socket (`virsh -r` for `list`, `domstate`, `dumpxml` and
`domstats`), the `cloud-hypervisor` runtime's files (shared with each VMM's uid through ACL entries) and each VMM's
API socket.

An unprivileged `bcd node` without `[helper]` on a real host cannot write `/etc/cdi` or create hold slices. Its `Hello`
then claims no GPU binding capability, and a hold that arrives anyway fails at once with reason `node-unprivileged`
instead of retrying every 30 seconds (da60bbd).

## The socket and the peer check [#the-socket-and-the-peer-check]

<Mermaid
  chart="flowchart LR
  N[&#x22;bcd node (helper.user)&#x22;] -->|&#x22;connect: one request&#x22;| S[&#x22;[helper] socket, 0660 root:bcd&#x22;]
  S --> P{&#x22;SO_PEERCRED: uid and gid; /proc/pid/exe&#x22;}
  P -->|&#x22;not allowed&#x22;| X[&#x22;refused, logged&#x22;]
  P --> D[&#x22;strict JSON v1 decode&#x22;]
  D --> O[&#x22;operation: checks, then act as root&#x22;]
  O --> L[&#x22;log: pid, op, kind, name, gpu, unit, instance, outcome&#x22;]"
/>

For every connection, the helper reads the caller's pid, uid and gid from the kernel with `SO_PEERCRED`
(`peer_linux.go`). The helper is Linux only for this reason. A caller is served only when both of these hold:

| Check      | Rule                                                                                                                                                                 |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Account    | The uid and gid are both `[helper] user`'s (its uid and primary group), or the uid is 0                                                                              |
| Executable | `/proc/<pid>/exe` is the same file (`os.SameFile`) as an installed bcd: `[update] entrypoint`, `bcd-a` or `bcd-b` in the slots directory, or the helper's own binary |

A refused caller gets only `not allowed`. The helper logs `helper: connection refused` with the pid, uid and reason
(`uid <n> gid <n> is not <user>`, or `the peer does not run an installed bcd binary`). A copy of bcd run from
elsewhere by the `bcd` user is refused, and so is any other account.

The socket is created so that root never changes a path the node could swap for a symlink (2d67b25):

1. The socket's directory is created 0755 if missing. As root, the helper refuses to start unless that directory
   is root's and closed to group and other writes (`must be a directory root owns that no one else can write`).
2. It binds the socket inside a private `.bcd-helper-*` directory made with `mkdtemp`, chmods it 0660, chowns it to
   root and the node account's primary group, then renames it into place. A stale socket is replaced atomically by
   that rename.

The install script uses `/run/bcd-helper/helper.sock`, in the helper unit's own `RuntimeDirectory`. It never uses
`/run/bcd`, which the `bcd` user owns. The package default `helper.DefaultSocket` is `/run/bcd/helper.sock`, but the
configuration must name the socket explicitly: without `[helper] socket` the helper exits 1 (`[helper] socket and
user are not configured`).

## Requests [#requests]

One connection carries one request and one answer, as newline-delimited JSON:

```json title="Request (host.unit)"
{"v":1,"op":"host.unit","unit":"sysctl","settings":{"net.ipv4.tcp_keepalive_time":600},"intent":{"body":"...","signature":"..."}}
```

```json title="Response"
{"v":1,"ok":false,"error":"host.unit: the settings of sysctl are not the signed spec's"}
```

| Rule              | Value                                                                                                                                                                     |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Version           | `v` is `helper.Version`, 1. A request of another version is answered `protocol version <n>; this helper speaks 1`, and the client refuses an answer of another version    |
| Decoding          | Strict: unknown fields are refused on both sides (`malformed request`), so a node and a helper of different builds refuse each other rather than guess                    |
| Size              | A request is at most 4 MiB (`MaxRequest`), an answer at most 1 MiB, a written file at most 1 MiB (`MaxFile`)                                                              |
| Time              | Each side gives a request 2 minutes; `host.unit`, `libvirt.*` and `ch.*` get 30 minutes plus one on the connection                                                        |
| Concurrency       | At most 32 requests at once (`maxConns`); a connection over the cap is answered `busy: the helper is serving its maximum of requests; retry`                              |
| Terminal refusals | `refused: true` marks a self-update refusal no retry can fix (`selfupdate.Refusal`); the node then reports the update as failed instead of retrying                       |
| Results           | `result` carries an operation's JSON answer (`update.*`, `dmi.read`, `hw.read`, `tpm.*`). `libvirt.console` answers with a file descriptor (`SCM_RIGHTS`) beside the JSON |

Every operation is logged as `helper: operation` with the peer's pid, `op`, the file `kind` and `name`, the `gpu`,
the update `version`, the config `unit`, the `instance` (for `ch.*`) and the `outcome`. The one exception is a
successful `tpm.sign`: every TLS handshake and node statement signs, so only its refusals are logged.

A self-update replaces only the node's binary. The helper keeps running the binary it started with until it restarts,
which works while both speak protocol version 1. A [rerun of the install
script](/docs/daemon/install#before-v04-rerun-the-script) restarts both units.

## Operations [#operations]

The table lists every operation `Server.do` dispatches. Anything else is `unknown operation "<op>"`. An operation
whose backend this helper was not configured with answers `... not configured on this helper`.

| Family                                    | Operations                                                                                                                                                                          | Sent by the node for                                                                               |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| [Files](#files)                           | `file.write`, `file.remove`                                                                                                                                                         | CDI specs and allocation env files (`devices`), sshd files (`access`), the kubeadm bootstrap token |
| [Firewall](#firewall)                     | `firewall.apply`, `firewall.check`                                                                                                                                                  | The instances' nftables tables and their 5-second check                                            |
| [Host actions](#gpureset-reboot-and-sshd) | `gpu.reset`, `reboot`, `ssh.reload`                                                                                                                                                 | `gpu.reset`, `node.reboot`, a changed sshd drop-in                                                 |
| [Config units](#hostunit)                 | `host.unit`                                                                                                                                                                         | Each `NodeSpec.host` unit that is not converged                                                    |
| [Hold slices](#slices)                    | `slice.apply`, `slice.remove`                                                                                                                                                       | GPU holds with limits                                                                              |
| [Self-update](#self-update)               | `update.status`, `update.prepare`, `update.swap`, `update.confirm`, `update.rollback`, `update.settle`                                                                              | The `daemon` section and `daemon.upgrade`                                                          |
| [libvirt](#libvirt)                       | `libvirt.define`, `.start`, `.shutdown`, `.destroy`, `.suspend`, `.resume`, `.undefine`, `.blockresize`, `.setmem`, `.attach-disk`, `.detach-disk`, `.save`, `.restore`, `.console` | The `libvirt-kvm` runtime                                                                          |
| [cloud-hypervisor](#cloud-hypervisor)     | `ch.taps`, `ch.untaps`, `ch.vfio`, `ch.unvfio`, `ch.launch`, `ch.stop`                                                                                                              | The `cloud-hypervisor` runtime                                                                     |
| [TPM](#tpm)                               | `tpm.nodekey`, `tpm.sign`, `tpm.ak`, `tpm.certify`, `tpm.release`                                                                                                                   | The node key in root's TPM                                                                         |
| [Inventory reads](#inventory-reads)       | `dmi.read`, `hw.read`                                                                                                                                                               | Root-only DMI values, DIMMs, NVMe health and the BMC                                               |

PLAN.md also names `runtime.*` and `mig.apply` operations. They do not exist: runtimes go through `libvirt.*` and
`ch.*`, and a MIG layout is the `mig-layout` config unit, applied through `host.unit`.

### Files [#files]

The node never names a path. `helper.Client.WriteFile` maps the absolute path the node would have written to a kind
and an id (`Classify`) and refuses any path outside the table before it sends anything. The helper builds the path
again from its own table:

| Kind             | Path                                            | Id                        | Mode |
| ---------------- | ----------------------------------------------- | ------------------------- | ---- |
| `cdi`            | `/etc/cdi/bcd-<id>.json`                        | `^[A-Za-z0-9_-]{1,128}$`  | 0644 |
| `allocation-env` | `/run/bcd/allocations/<id>.env`                 | `^[A-Za-z0-9_-]{1,128}$`  | 0644 |
| `ssh-dropin`     | `/etc/ssh/sshd_config.d/10-better-compute.conf` | none                      | 0644 |
| `ssh-ca`         | `/etc/ssh/bc_ca.pub`                            | none                      | 0644 |
| `ssh-krl`        | `/etc/ssh/bc_krl`                               | none                      | 0644 |
| `ssh-principals` | `/etc/ssh/bc_principals/<id>`                   | `^[A-Za-z0-9._@-]{1,64}$` | 0644 |
| `k8s-token`      | `/var/lib/bcd-hostconfig/k8s/bootstrap-token`   | none                      | 0600 |

What root checks before it writes:

* **The kind and id.** An unknown kind is `"<kind>" is not a kind of file bcd owns`. An id that does not match is
  refused, and a kind without ids refuses any name (`<kind> takes no name`). Data over 1 MiB is refused.
* **The directory walk.** The helper opens `/` as an `os.Root` and walks each component of the kind's directory with
  `Lstat`. A symlink or anything that is not a directory is refused (`... is not a plain directory`), and a missing
  one is created 0755. Running as root on the real filesystem, every directory on the way must also be root's and
  closed to group and other writes (02485b3), or the write fails with `... is not a directory only root can write`.
* **The write.** It creates a temporary `.<name>.bcd-helper` exclusively (`O_EXCL`) in that directory, writes it,
  sets the mode and renames it over the target. A symlink planted at the target is replaced, never followed.
  `file.remove` removes the file and treats a missing one as done.

### Firewall [#firewall]

`firewall.apply` loads the ruleset the `instances` converger renders (`nft -f -`). Root accepts only that ruleset's
shape (`CheckRuleset`):

* at the top level, only `table`, `delete table` and `flush table` lines, and `table … {` blocks, of exactly three
  tables: `bridge bcd_instances`, `inet bcd_taps` (933c51b) and `arp bcd_taps` (b12c217). The same name in another
  family is refused;
* inside a block, no nested `table`;
* no line starting with `include` or `define`, and no line containing `ruleset`;
* balanced braces.

`firewall.check` takes a rule count from 0 to 1,000,000. Root lists the three tables with `nft -j` and compares; an
absent `inet` or `arp bcd_taps` counts as no rules. What the rules do is on
[instances](/docs/daemon/delivery#instances) and [bcd's nftables
tables](/docs/guides/vps/networking#bcds-nftables-tables).

### gpu.reset, reboot and sshd [#gpureset-reboot-and-sshd]

| Operation    | Root checks                                                                                                | Root does                                      |
| ------------ | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| `gpu.reset`  | The UUID matches `^GPU-[0-9A-Fa-f-]{8,64}$` and is a GPU in the host's own inventory, read at each request | `nvidia-smi --gpu-reset -i <uuid>`             |
| `reboot`     | The reason is cut to 200 bytes                                                                             | The host's reboot through systemd              |
| `ssh.reload` | Nothing: it takes no argument                                                                              | `systemctl reload-or-restart ssh`, then `sshd` |

The node still refuses a `gpu.reset` command for a GPU with a live hold before it asks (see
[gpu.reset and live holds](/docs/daemon/delivery#gpureset-and-live-holds)). The helper's own check is only that the
GPU is the host's.

### host.unit [#hostunit]

`host.unit` applies one `NodeSpec.host` config unit as root. The node plans every unit itself, unprivileged, and
sends only the units that are not converged: the unit name, its settings (none when the unit left the spec and
should be cleaned) and the signed spec that asks for it. The full rules are on [host configuration under bcd
helper](/docs/daemon/host-config#under-bcd-helper). In short, root checks:

| Check                                                                                                                                                                                                                               | Refusal                                                                                                    |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Root's own `[host] units` is not `false` (3fa9021)                                                                                                                                                                                  | `host.unit: this host's configuration is managed outside bcd ([host] units = false in root's config.toml)` |
| The unit name matches `^[a-z0-9-]{1,64}$`                                                                                                                                                                                           | `"<name>" is not a config unit name`                                                                       |
| At most 1024 busy GPUs in the request                                                                                                                                                                                               | `too many busy GPUs`                                                                                       |
| The signed spec: pinned or recorded command-signing roots, a worker signature on a `DesiredStateBody`, this node, a generation not older than the last, settings byte for byte the signed ones (03c023d, 2d18327, a966cd4, 71ccf71) | `host.unit: ...`, see [signed specs](/docs/daemon/host-config#signed-specs)                                |
| The unit's own validation, run again as root (`PlanUnit`), and the sysctl denylist                                                                                                                                                  | The unit's own error                                                                                       |

Root ignores the node's list of busy GPUs (a894cd2) and works it out itself (`rootBusyGPUs`): the GPUs with compute
processes (`nvidia-smi --query-compute-apps=gpu_uuid`; a `MIG-` UUID marks every GPU busy) and the GPUs named by the
live holds' CDI specs in `/etc/cdi/bcd-*.json`. A `k8s-join` under the helper also needs root's own
`[kubernetes] allowed_clusters` (`RequireAllowed: true`).

Root keeps two records for this operation in `/var/lib/bcd-hostconfig`:

| File                      | Written by                                                                       | Holds                                                                                                                                                    |
| ------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `join-response.pb` (0600) | `bcd join` run as root on a node with `[helper]`, before the identity is written | The verified `JoinResponse` as signed. The helper reads the node id and command-signing roots from it, re-verified against root's `join_signing_key_pin` |
| `helper-intent.json`      | The helper                                                                       | `nodeId`, the newest accepted `generation` and `bodySha256`, the newest `trustVersion`, and `signed` once a trust bundle with roots was accepted         |

### Slices [#slices]

`slice.apply` and `slice.remove` create and remove a GPU hold's cgroup slice (260e33c). Root checks the slice name
against `^bcd-alloc-[A-Za-z0-9_]{1,80}\.slice$` and every cpulist against the cpulist grammar (`slice.Check`), so only
such names and lists reach `systemctl` and cgroupfs. `slice.apply` needs limits. `slice.remove` stops a slice only
once its `cgroup.events` shows `populated 0`, and otherwise answers `the slice still runs processes`. Both are bounded
to 30 seconds (`SliceTimeout`). See [device assignments](/docs/daemon/delivery#device-assignments).

### Self-update [#self-update]

The `update.*` operations are self-update's privileged half, `selfupdate.Manager`, with the slots and records in
root's `[update] slots_dir` (default `/usr/local/lib/bcd` under the helper). The node only downloads into
`<data_dir>/staging` and restarts. Root decides from its own configuration and files, never from the node's claims:

| Operation         | Root checks and does                                                                                                                                                                                                                                                                                 |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `update.status`   | Answers the pending update, the release record and the slots                                                                                                                                                                                                                                         |
| `update.prepare`  | Verifies the target: the manifest under root's `[update] release_pins` and threshold, the product, version, release floor and expiry, and the artifact for this platform. Answers the artifact to download, or nothing when the other slot already holds it                                          |
| `update.swap`     | Verifies again, opens `bcd-<version>` beneath the staging directory without following a symlink or blocking on a FIFO, takes only a regular file with one link owned by the node's uid (`StagedUID`), copies it into the other slot hashing as it writes, records `upgrade.json` and swaps `current` |
| `update.confirm`  | Moves the release floor up, only while `current` is the pending slot                                                                                                                                                                                                                                 |
| `update.rollback` | Swaps back a pending update to the version named                                                                                                                                                                                                                                                     |
| `update.settle`   | Decides what a starting node's start means from where `current` points, never from the version the node claims (c139b9c)                                                                                                                                                                             |

The helper arms its own watchdog after each swap, and again at start when an update is pending. An update still
unconfirmed 30 seconds after its deadline is swapped back by the helper, which then runs `systemctl restart
bcd.service`. [Self-update](/docs/daemon/self-update) has the whole flow.

### libvirt [#libvirt]

Under the helper, the `bcd` user is not in the root-equivalent `libvirt` group (ed3c7d2, d418820). The
`libvirt-kvm` runtime sends every privileged `virsh` verb to the helper, which runs `virsh -c qemu:///system` as root:

| Operation                                                       | Root checks                                                                                                                                                                                                                 |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `libvirt.define`                                                | The XML passes the default-deny domain validator (`ValidateDomainXML`, c305a05, fea4d2e), and no domain of that name exists that bcd did not define. The definition is staged in root's own directory before `virsh define` |
| `start`, `shutdown`, `destroy`, `suspend`, `resume`, `undefine` | The domain name matches `^[a-z]{2,8}_[0-9a-z]{10,52}$` and its `dumpxml` carries bcd's metadata namespace naming it. `start` may add `--paused`; `undefine` adds `--nvram --managed-save`                                   |
| `libvirt.blockresize`                                           | A `^vd[a-z]{1,2}$` target and a non-zero size                                                                                                                                                                               |
| `libvirt.setmem`                                                | At least 1 MiB, applied `--live` (the balloon)                                                                                                                                                                              |
| `libvirt.attach-disk`, `libvirt.detach-disk`                    | Only target `vdz` (the identity rebind disk), live; the attached file passes the same file checks as a domain's disks                                                                                                       |
| `libvirt.save`                                                  | The destination is in the instances' state directories; root saves into its own staging directory first, then copies through an `os.Root` and hands the file to the node's uid                                              |
| `libvirt.restore`                                               | A definition is required (b6df6f1), checked like a `define` and for the same domain, and always passed as `--xml`; the state file is checked as the node's own regular file                                                 |
| `libvirt.console`                                               | A bcd domain with a pty console; root opens it (`O_NOCTTY`) and passes the descriptor back                                                                                                                                  |

Since 5bb9f22 the helper also turns NVIDIA persistence mode off for a domain's passthrough GPUs before
`libvirt.start` and back on after `libvirt.destroy`, because `nvidia-persistenced` holds the GPUs open and libvirt's
managed hostdevs cannot unbind them while it does. The validator's admitted elements are listed on [libvirt through
the helper](/docs/daemon#libvirt-through-the-helper).

### cloud-hypervisor [#cloud-hypervisor]

Since 7fe74a1 the `cloud-hypervisor` runtime runs under the helper too. Each `ch.*` request names an instance id
(the domain-name pattern above), and root reads everything else from its own checks:

| Operation   | Root checks and does                                                                                                                                                                                                                                                                                                                          |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ch.taps`   | At most 16 NICs; each bridge must be in root's `[helper] instance_bridges`; creates exactly `runtime.TapName(instance, i)`, owned by the node                                                                                                                                                                                                 |
| `ch.untaps` | Deletes only those names, and refuses an interface of that name that is not a tap                                                                                                                                                                                                                                                             |
| `ch.vfio`   | Takes the GPUs from the instance's `vm.json`; only this host's GPUs, none root sees holding work, none whose `/dev/vfio` group a running VMM unit's user owns (94843ec), and only groups whose other devices are functions of this host's GPUs or PCI bridges. Binds them to `vfio-pci` and gives `/dev/vfio/<group>` (0600) to the VMM's uid |
| `ch.unvfio` | Gives groups back to root and unbinds them, never a group a running VMM unit's user owns                                                                                                                                                                                                                                                      |
| `ch.launch` | Reads and checks the instance record as root (`runtime.CHRecord`), then runs `systemd-run` with the confined unit it builds itself (`runtime.CHLaunch`) and root's own `cloud-hypervisor` binary, one launch at a time                                                                                                                        |
| `ch.stop`   | Stops `bcd-ch-<id>.service`, answers its state, and gives back exactly the VFIO groups its user owned                                                                                                                                                                                                                                         |

Without a `cloud-hypervisor` binary on root's `PATH`, every `ch.*` is refused. The details are on
[cloud-hypervisor through the helper](/docs/daemon#cloud-hypervisor-through-the-helper).

### TPM [#tpm]

Since 8870b8d (security review 3) an unprivileged `bcd node` never opens `/dev/tpmrm0`. The kernel's resource manager
filters no command, so a user in the `tss` group could change the hierarchies' authorization, clear the TPM or trip
its lockout. The helper opens the TPM as root and serves five operations on node keys named by a 32-byte unique:

| Operation     | Does                                                                                             | Refuses                                 |
| ------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------- |
| `tpm.nodekey` | Derives the node key from `tpm.NodeKeyTemplate` and answers its `TPMT_PUBLIC`                    | A unique that is not 32 bytes           |
| `tpm.sign`    | Signs a digest (ASN.1 ECDSA)                                                                     | A digest that is not 32, 48 or 64 bytes |
| `tpm.ak`      | Answers the AK's public area                                                                     |                                         |
| `tpm.certify` | `TPM2_Certify` of the key by the AK over 32 bytes of qualifying data (a renewal's `sha256(csr)`) | Qualifying data that is not 32 bytes    |
| `tpm.release` | Unloads the key                                                                                  | A unique that is not 32 bytes           |

Root builds every TPM command, always with the template's empty password. The node chooses no template, handle,
hierarchy or authorization. At most 8 node keys stay loaded; the least recently used is unloaded and derived again
on its next use. A helper on a host without a TPM refuses all five (`node keys in the TPM are not served by this
helper (the host has no TPM)`). The node checks every answer: the key's attributes must be `tpm.NodeKeyAttributes`,
every signature must verify against the key's public half, and a certification must come from the AK the helper
announced. See [node identity](/docs/daemon/identity#node-keys-in-the-tpm).

### Inventory reads [#inventory-reads]

Two read-only operations give an unprivileged node the inventory details sysfs keeps root-only. Both take no
arguments: a request with any field besides `v` and `op` is refused (`dmi.read takes no arguments`).

| Operation                    | Reads                                                                                                                                                                                                                    | Bounds                                                                                                                                             |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dmi.read` (29ed48d)         | `product_uuid`, `product_serial`, `board_serial`, `chassis_serial`, `board_asset_tag` and `chassis_asset_tag` under `/sys/class/dmi/id`, the values the hardware key is built from                                       | Each file opened without following a symlink, regular files only, at most 128 bytes, printable ASCII only, trimmed; no SMBIOS answers an empty map |
| `hw.read` (32beca7, 45bc17f) | DIMMs parsed from SMBIOS type 17 entries (`/sys/firmware/dmi/entries/17-*/raw`; only parsed values go back, never the raw table), each NVMe controller's SMART / Health log page, and the BMC as it answers in-band IPMI | At most 64 entries and 64 controllers, 4 KiB per SMBIOS entry, 30 seconds overall; a read that overruns is left out, as for an unprivileged read   |

The node asks for `dmi.read` with a 10-second timeout and `hw.read` with 45 seconds. Either failing reads as "no
values": the inventory then carries what the unprivileged read saw. The IPMI BMC read is inventory only, never an
attestation input, and its command set is allowlisted in the device layer.

## Starting the helper [#starting-the-helper]

`bcd helper --config /etc/bcd/config.toml` exits 1 unless all of these hold:

| Check                                                                                                                                                                                      | Error                                                                |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| `[helper] socket` is set                                                                                                                                                                   | `[helper] socket and user are not configured`                        |
| It runs as root (a fakehw build excepted)                                                                                                                                                  | `runs as root`                                                       |
| `[helper] user` names an account                                                                                                                                                           | `helper.user: ...`                                                   |
| `[kubernetes] allowed_clusters` parses                                                                                                                                                     | The parse error                                                      |
| `command_root_sha256` parses (at most 16 pins)                                                                                                                                             | The parse error                                                      |
| As root: `config.toml` is a regular file (not a symlink) root owns, writable by no one else, in directories only root can write (3fa9021)                                                  | `<file> must be a regular file root owns that no one else can write` |
| As root: every existing directory from `/` to the slots directory, the socket's directory and the entrypoint's directory is root's and closed to group and other writes (c139b9c, 9e71b1f) | `<dir> must be a directory root owns that no one else can write ...` |
| The self-update manager builds from `[update]`                                                                                                                                             | The error                                                            |
| The socket binds                                                                                                                                                                           | The error                                                            |

The configuration check (`config.Validate`) also refuses a relative socket path, a `user` of `root`, and, under the
helper, an `[update] slots_dir` or `[update] entrypoint` inside `data_dir`: the node owns that directory and could
plant the binary root trusts (9e71b1f).

That root-owned `config.toml` decides what root does for the node, so these keys are read from root's copy, never
from the node:

| Key                             | What root uses it for                                                  |
| ------------------------------- | ---------------------------------------------------------------------- |
| `[helper] socket`, `user`       | Where it listens and whom it serves                                    |
| `[helper] instance_bridges`     | The bridges `libvirt.define` and `ch.taps` admit; empty admits no NIC  |
| `[host] units`                  | `false` refuses every `host.unit`                                      |
| `command_root_sha256`           | Pins for host units' signed specs                                      |
| `join_signing_key_pin`          | Verifies root's join record and presented trust bundles                |
| `[kubernetes] allowed_clusters` | The only clusters `k8s-join` may join                                  |
| `[update]`                      | Release pins, threshold, sequence, slots and entrypoint for `update.*` |

`bcd doctor`, run as root on a helper node, checks the same conditions: the `helper` check fails for each directory
on the way to the slots, socket or entrypoint that is not root's alone, says what host units bind to, and fails when
no command-signing roots are pinned or recorded. The `libvirt`, `cloud-hypervisor` and `tpm` checks cover the group
memberships (see [bcd doctor](/docs/daemon#modes)).

## Install layout [#install-layout]

`daemon install-script --unprivileged` (169c459) sets up this layout; [running bcd node
unprivileged](/docs/daemon/install#running-bcd-node-unprivileged) shows the units and the steps.

| Path                                                  | Owner and mode  | What                                                                                                                 |
| ----------------------------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------- |
| `/etc/bcd`                                            | `root:bcd` 0750 | The configuration directory                                                                                          |
| `/etc/bcd/config.toml`                                | `root:bcd` 0640 | Read by both processes; only root may write it                                                                       |
| `/etc/systemd/system/bcd-helper.service`              | root 0644       | `bcd helper --config /etc/bcd/config.toml`, `Before=bcd.service`, `UMask=0077`, `RuntimeDirectory=bcd-helper` (0755) |
| `/etc/systemd/system/bcd.service.d/unprivileged.conf` | root 0644       | `User=bcd`, `Group=bcd`, `SupplementaryGroups=systemd-journal adm`, `Requires=` and `After=bcd-helper.service`       |
| `/run/bcd-helper/helper.sock`                         | `root:bcd` 0660 | The socket                                                                                                           |
| `/usr/local/lib/bcd`                                  | root 0755       | The self-update slots and records (`[update] slots_dir`)                                                             |
| `/var/lib/bcd`                                        | `bcd:bcd`       | `data_dir`: journal, identity, key record, staging, instance files                                                   |
| `/var/lib/bcd-hostconfig`                             | root            | Root's records: `join-response.pb`, `helper-intent.json`, the config units' records, the libvirt staging directory   |

The `bcd` account is in neither the `libvirt` group (since d418820) nor the `tss` group (since 8870b8d). On a host
with `/etc/libvirt/qemu.conf` the script sets `dynamic_ownership = 0` and puts QEMU's user in the `bcd` group, so QEMU
reaches the node's disk files through that group. `--unprivileged` with `--plugins` is a usage error, since bcx
plugins need root to start. Only `bcd node` uses the helper: `bcd site` ignores `[helper]`, and the ramdisk has no
configuration file.

## Testing the helper [#testing-the-helper]

| Test                                                                                     | Covers                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/helper` unit tests                                                             | The file table and walk, the ruleset check, `host.unit` re-planning and refusals (`TestHelperHostUnit`, `TestHostUnitNeedsTheSignedSpec`, `intent_bundle_test.go`), the domain validator (`domain_test.go`), libvirt against a fake `virsh`, `ch.*` (`ch_test.go`), `TestHelperTPMKeys` against `tpmsim`, `dmi.read` and `hw.read`                                                                                                    |
| `tests/e2e/bcd.test.ts`                                                                  | An unprivileged fakehw `bcd node` beside a real `bcd helper`: a GPU hold's CDI spec written and removed (the helper log shows `file.write` and `file.remove`), and a signed update delivered through `update.*`                                                                                                                                                                                                                       |
| `internal/helper/roote2e` (`node scripts/helper-e2e.mjs`, build tags `nonvml,helpere2e`) | As root in private namespaces: the install script moves a root-mode node under the helper; a copy of bcd and another user are refused; signed sysctls applied and unsigned, tampered, foreign, older or unpinned specs refused; libvirt definitions with foreign disks, root-owned disks, `qemu:commandline`, foreign PCI devices or bridges refused; real taps for `cloud-hypervisor`; self-update swap, rollback and failure record |

`bcd helper-call` exists only in `helpere2e` builds, never in a release: it reads one request from standard input
and sends it to `[helper] socket` as the installed bcd binary would.


# Host configuration (/docs/daemon/host-config)

> The config units bcd converges from NodeSpec.host, in order, with their settings, files, refusals and reboot cases, and how bcd helper applies them as root.



A node's NodeSpec has a `host` section: a list of
config units, each a unit name and
its settings as canonical JSON. `bcd node` applies them with its host converger (6c2343f): kernel modules and
parameters, huge pages, the CPU governor and isolated CPUs, the IOMMU, the NVIDIA driver, Fabric Manager, GPU power and
ECC settings, the MIG layout, GPU sharing, the container toolkit and a Kubernetes join. This page describes what each
unit does to the machine, and what it refuses. The control plane's side of each unit, its settings manifest field by
field, is the generated [host config units reference](/docs/reference/host-units).

> **A host configured by something else: [host] units = false.** 
  When another tool owns the machine's configuration, for example the NVIDIA GPU operator when bcd runs as a
  Kubernetes DaemonSet, set `units = false` under `[host]` in `config.toml` (a193249). bcd then has no host converger:
  it applies no unit, and a `host` section that lists units is reported `UNSUPPORTED` with reason
  `HostManagedExternally`. That state does not hold back `observed_generation` and raises no `SectionUnsupported`
  condition; the gateway records it as the informational node condition `HostManagedExternally` (c7613e1). The
  inventory still reports what the host has, so capacity never books huge pages or MIG instances the other tool did
  not set up. Under [`bcd helper`](#under-bcd-helper) root reads the same key from its own root-owned `config.toml` and
  refuses every `host.unit` (3fa9021).

> **Mostly tested against fakes.** 
  The units are tested against `FakeSystem`, an in-memory host whose files and command answers the tests script. Only
  `TestRealKernel` touches a real kernel, as root on request: it sets a sysctl, the transparent huge page policy and
  2 MiB huge pages and puts each back, and checks the rule that root writes only under root's directories (60143bf,
  02485b3); it passes on WSL Ubuntu. No test has installed a driver, partitioned a GPU, reserved 1 GiB pages, set boot
  arguments or joined a cluster on a real machine. See [what the tests cover](#what-the-tests-cover).

## Where the units come from [#where-the-units-come-from]

bcd does not choose units. The control plane builds the `host` section, and provision is the module that fills it
(`hostSpecFor` in `packages/provision/src/config-units.ts`):

* **A profile's `hostConfig`.** When a workflow succeeds, the `hostConfig` of its profile becomes the node's units. A
  unit may carry `when:`, a label expression that turns it on per node. See
  [profiles and images](/docs/guides/bare-metal/profiles-and-images#host-config).
* **The pool.** `policy.hugepages` becomes `hugepages`, `policy.kernelModules` becomes `kernel-modules`, `policy.cpu`
  becomes `cpu` (09669d3), `policy.iommu` becomes `iommu` and `policy.gpu` becomes `gpu-settings` (e4b0c01),
  `policy.kubernetes` becomes `k8s-join` when it names the cluster (`apiServer`, `caCertSha256` and `joinTokenRef`,
  all three or none; 7db110a), and a pool shared by `mps` or `timeslice` gets `gpu-sharing` (1f0f683), on every
  machine of the pool, adopted ones included. A pool's unit replaces the profile's unit of the same name.
* **The MIG layout gate.** `mig-layout { from: 'pool' }` takes the pool's `policy.migLayout`, or the layout the
  capacity module froze the node at while a layout change waits for its GPUs to be free (see
  [the MIG layout gate](/docs/guides/capacity/fractional-gpus#the-mig-layout-gate)).

The control plane validates each unit against its manifest and refuses an unknown unit with `INVALID_INPUT`. Since
37dcd02 the pools API runs the same checks on a pool's host policy when `pools.create` or `pools.update` writes it
(provision registers them as a `pool-policy` extension), so a policy no node could take is refused there, not when a
node's spec is built. The server writes the units to the wire sorted by name, each settings object as canonical JSON
(`wireNodeSpec`). bcd decodes the settings again, strictly: an unknown field or trailing data is a terminal failure,
so settings it does not understand are never half applied.

## Order [#order]

bcd applies the units in a fixed order, whatever order the spec lists them in:

<Mermaid
  chart="flowchart LR
  KM[&#x22;kernel-modules&#x22;] --> SY[&#x22;sysctl&#x22;] --> HP[&#x22;hugepages&#x22;] --> CPU[&#x22;cpu&#x22;] --> IO[&#x22;iommu&#x22;] --> ND[&#x22;nvidia-driver&#x22;]
  ND --> FM[&#x22;fabric-manager&#x22;] --> GS[&#x22;gpu-settings&#x22;] --> MIG[&#x22;mig-layout&#x22;] --> SH[&#x22;gpu-sharing&#x22;] --> CT[&#x22;container-toolkit&#x22;] --> KJ[&#x22;k8s-join&#x22;]"
/>

Modules load before the parameters they provide (`net.bridge.*` needs `br_netfilter`), memory and CPUs are set up
before workloads need them, the driver comes before Fabric Manager, the GPU settings and the MIG layout, GPU sharing
sits on the partitioned GPUs, the container toolkit comes after them so its CDI spec describes the GPUs as
partitioned, and a cluster join is last.

Each unit that is not converged becomes one action, and that action plans the unit again when it runs, against the
host as the units before it left it. So when `kernel-modules` loads `br_netfilter`, the `sysctl` unit in the same
pass finds the `net.bridge.*` keys it could not find when the pass started.

## Level-triggered [#level-triggered]

A unit's plan compares its settings with what the host shows now: the files bcd owns, `/proc/sys`, `/sys`, the
loaded modules, the installed packages, `nvidia-smi`. It writes only what differs. A converged unit plans no steps,
and a changed setting or a local edit is planned away the next time the unit runs: a sysctl someone changed by hand
is set back, a deleted drop-in is written again, and every file bcd writes starts with a header saying local edits
are replaced.

bcd runs a converge pass when it starts, when a new spec arrives, every 30 seconds (`ConvergeRetry`) while any
section is `PROGRESSING`, after a few events of other sections (a failed firewall reload, a finished snapshot
download) and, since 3c0fe02, every 10 minutes whatever happened (`driftInterval`). So a sysctl, module, service, CDI
spec or mount changed behind bcd's back is repaired within 10 minutes, with no new spec.

After a unit applies, bcd refreshes its inventory, so the driver version, the MIG devices and the huge page pools
it reports follow what the unit changed. Since 427787a the heartbeat right after such a change carries the inventory
hashes too, so the gateway applies the new sections at once and capacity sees, say, reserved huge pages within a
heartbeat instead of up to ten heartbeats later.

## Status and failures [#status-and-failures]

The host converger reports one section, `host`, in `StatusReport` (see
[convergence and status](/docs/daemon/delivery#convergence-and-status)):

| Outcome                                                              | Section state                                            | Examples                                                                                                                                                                                                                                    |
| -------------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Every unit converged                                                 | `CONVERGED`                                              |                                                                                                                                                                                                                                             |
| At least one unit failed, and at least one failure may succeed later | `PROGRESSING`, retried every 30 seconds                  | A package install failed, a sysctl key the kernel does not have yet, a unit waiting for a reboot, a busy GPU that mig-layout skipped, no driver loaded yet for `fabric-manager`, `gpu-settings` or `container-toolkit`                      |
| Every failure is one no retry can fix                                | `FAILED`, terminal: `observed_generation` still advances | Unknown fields, a bad driver branch, a refused sysctl, a MIG profile the GPU does not have, a governor the CPU does not offer, a power limit outside the GPU's range, a distribution bcd has no packages for, a unit this bcd does not know |

Since c21ef84 no unit holds back another. Every unit gets its turn in the pass, in order: a unit that fails, whether
while it is planned or while it applies, is recorded and the pass goes on to the next unit. A last step then reports
every failure together, each prefixed with its unit's name. The section is retryable while any failure is, and
`FAILED` only when all of them are terminal. So a `hugepages` unit waiting for a reboot, or a `mig-layout` unit
waiting for busy GPUs, no longer keeps `nvidia-driver`, `gpu-sharing` or `container-toolkit` from converging. A unit
named twice fails the whole section, since bcd cannot tell which settings apply. The `host` section never holds back
the other sections either: `devices`, `instances` and the rest converge on their own.

Three units report **conditions** besides the section state: `FabricManagerDown` and `MPSDown`, both with reason
`NotRunning`, and `k8s-join`'s `KubernetesTaintsDrift` (reason `NodeRestriction`, 6ef198b), as
`StatusReport.conditions` with status true while the fault lasts, and the converger adds `RebootRequired` (below). The
gateway's `xidStatusHandler` turns them into node conditions (c7613e1, 0aea5c0, 88219ea), written and published as
`node.condition.changed` only when they change:

| Condition               | Effect on the node                                                                                         |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| `FabricManagerDown`     | NoSchedule while true: an NVSwitch system without Fabric Manager cannot initialize CUDA for multi-GPU jobs |
| `MPSDown`               | NoSchedule while true: every MPS client fails                                                              |
| `KubernetesTaintsDrift` | Informational: the message is the `kubectl taint` command a cluster admin runs (see [k8s-join](#k8s-join)) |
| `RebootRequired`        | Informational; the node keeps running and taking work                                                      |
| `HostManagedExternally` | Informational, from the `host` section's state (`[host] units = false`)                                    |

On a `tenant-controlled` node these are advisory: the condition is recorded, never a NoSchedule reason. See
[conditions](/docs/guides/nodes/inventory-and-health#conditions).

## Reboots [#reboots]

Some settings take effect only after a reboot, and **bcd never reboots on its own**. A unit in that case returns a
`RebootRequired` error whose message says `reboot the node`; the unit stays retryable, so the section stays
`PROGRESSING`, and the node reports the condition `RebootRequired` with status true, its reason the names of the
waiting units (comma-separated) and its message their errors (at most 1024 bytes). A unit applied through
[bcd helper](#under-bcd-helper) comes back as text, and the phrase is how the node still recognises it.

| Unit            | Waits for a reboot when                                                                         |
| --------------- | ----------------------------------------------------------------------------------------------- |
| `hugepages`     | 1 GiB pages could not be allocated at runtime and were reserved on the kernel command line      |
| `cpu`           | `isolate` names CPUs that `/sys/devices/system/cpu/isolated` does not show yet                  |
| `iommu`         | `/sys/class/iommu` lists no IOMMU yet                                                           |
| `gpu-settings`  | A GPU's current ECC mode is not yet the one set as pending                                      |
| `nvidia-driver` | Another driver is loaded and in use, or the loaded driver is still not the branch after loading |
| `mig-layout`    | A MIG mode change waits for a GPU reset, and the reset failed                                   |

Nothing in the control plane reboots on this condition. An operator reboots the node with
[`node.reboot`](/docs/guides/nodes/commands), within the pool's disruption budget (under the v1.0 quorum, as part of
an approved batch). After the reboot the next pass finds the setting in effect, and the condition goes.

## What removing a unit does [#what-removing-a-unit-does]

Removing a unit from the spec stops bcd managing that concern. It does not undo it: a driver stays installed, a
toolkit stays configured, huge page pools and their boot arguments stay, a MIG layout stays, Fabric Manager keeps
running, GPU power limits and ECC modes stay, and a CPU governor stays. Five units clean up after themselves, because
their state is only bcd's own files, and `k8s-join` hands the machine back:

| Unit             | When it leaves the spec                                                                                                                                                                                                                                                                                                                                                                                                             |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sysctl`         | `/etc/sysctl.d/90-bcd.conf` is removed; the live values stay until the next boot                                                                                                                                                                                                                                                                                                                                                    |
| `kernel-modules` | `/etc/modules-load.d/bcd.conf` and `/etc/modprobe.d/bcd.conf` are removed; loaded modules stay loaded                                                                                                                                                                                                                                                                                                                               |
| `cpu`            | Its kernel arguments are removed from the boot loader, effective at the next boot; the governor stays                                                                                                                                                                                                                                                                                                                               |
| `iommu`          | Its kernel arguments are removed from the boot loader, effective at the next boot                                                                                                                                                                                                                                                                                                                                                   |
| `gpu-sharing`    | `bcd-mps.service` is disabled and stopped, its unit file removed, and systemd reloaded; stopping MPS sets the GPUs' compute mode back to `DEFAULT`                                                                                                                                                                                                                                                                                  |
| `k8s-join`       | On a node bcd joined (9f860ad, 028b81c): the Node is cordoned, its running and pending workload pods are deleted with their own grace periods (never DaemonSet, static or finished pods), and the release is recorded, which stops the device plugins or DRA driver and raises `KubernetesReleased`. The kubelet is not reset: the machine stays a cordoned member until an operator removes it, and the unit's return uncordons it |

## Root's records [#roots-records]

The units keep what they record about the host in `/var/lib/bcd-hostconfig` (`hostconfig.StateDir`), a directory
only root creates and writes, never in bcd's `data_dir` (f8e13ef, 1bb78d6). Under bcd helper the unprivileged node
owns `data_dir`, and a record there would let it plant a symlink and steer root's writes.

| Path under `/var/lib/bcd-hostconfig` | Written by                                                                                                                               |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `bootargs/<unit>`                    | `cpu` and `iommu`: the kernel arguments the boot loader was last set up with                                                             |
| `hugepages-1g-boot`                  | `hugepages`: the 1 GiB page count reserved on the kernel command line                                                                    |
| `cdi-nvidia.stamp`                   | `container-toolkit`: what the NVIDIA CDI spec describes                                                                                  |
| `k8s-join.json`                      | `k8s-join`: the cluster, Node name, labels, taints and GPU mode it manages (0644, since the unprivileged device-plugin manager reads it) |
| `k8s-join.yaml`                      | `k8s-join`: the JoinConfiguration, only while `kubeadm join` runs (0600)                                                                 |
| `k8s-release.key`                    | `k8s-join`: the `pkgs.k8s.io` key download                                                                                               |
| `k8s/bootstrap-token`                | An unprivileged node's staged bootstrap token for the helper (helper file kind `k8s-token`, 0600)                                        |
| `helper-intent.json`                 | bcd helper: the node it serves and the newest signed host section it accepted (see [under bcd helper](#under-bcd-helper))                |

Whenever bcd runs as root (`bcd node` as root, or `bcd helper` applying a unit), every file a unit writes or removes
is checked first (02485b3): each existing directory from `/` down to it must be a real directory, not a symlink,
owned by root and writable by neither group nor others. `/tmp`, and anything the unprivileged node owns or could have
created, is refused with `<dir> is not a directory only root can write: bcd will not write as root under it`. The
helper's own file kinds are held to the same rule.

## Under bcd helper [#under-bcd-helper]

When `bcd node` runs unprivileged beside [`bcd helper`](/docs/daemon#privilege-separation), it still plans every
unit itself, reading the host as its own account can. A unit that is not converged is sent to the helper as one
`host.unit` request: the unit name, its settings (none when it left the spec and should be cleaned) and the GPUs
that live holds use. The helper builds its own units, plans the unit again as root from the name and settings alone,
with the unit's own validation, and applies it.

| `host.unit` | Value                                                                                                                                                                               |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Unit name   | `^[a-z0-9-]{1,64}$`, else `not a config unit name`                                                                                                                                  |
| Busy GPUs   | At most 1024; for `mig-layout` the helper uses its own view instead (below)                                                                                                         |
| Timeout     | 30 minutes (a driver install builds kernel modules), plus a minute on the connection                                                                                                |
| Checks      | The units decode and validate the settings themselves, exactly as in `bcd node`; an unknown unit is `not supported`, and a helper built without host units answers `not configured` |
| Signed spec | The signed spec that asks for the unit, whenever the helper knows the command-signing roots (below)                                                                                 |

`[host] units = false` is honoured on both sides (3fa9021). `bcd node` then has no host converger and sends no
`host.unit`, and `bcd helper` reads the key from its own `config.toml` and refuses every `host.unit`, applies and
cleans alike, with `host.unit: this host's configuration is managed outside bcd ([host] units = false in root's
config.toml)`, writing nothing. Root's copy is the one that counts, so a compromised node that dropped the setting
from its own view still gets no host change applied as root.

For the same reason, `bcd helper` running as root refuses to start unless its configuration file is a regular file
(not a symlink) that root owns and no one else can write, in directories only root can write: `<file> must be a
regular file root owns that no one else can write`. That file decides what root does for the node (`[host] units`,
`command_root_sha256`, the join pin). The install script already writes it that way, `root:bcd` 0640 in `/etc/bcd`
0750 (see [install](/docs/daemon/install)); a hand-made `config.toml` owned by the `bcd` account, or writable by its
group, stops the helper.

> **libvirt hosts.** 
  Since ed3c7d2 and d418820 a VPS or sandbox host that runs `libvirt-kvm` is separated too: `bcd node` is no longer in
  the root-equivalent `libvirt` group, and the helper defines and runs its libvirt domains after checking every
  definition (see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper)). A host installed with
  `--unprivileged` before d418820 keeps the node in that group until the install script runs again; `bcd doctor` fails
  it until then.

**Busy GPUs are root's view** (a894cd2). For `mig-layout` the helper ignores the busy GPUs the node sends and works
them out itself: the GPUs with compute processes (`nvidia-smi --query-compute-apps`; a MIG device's UUID makes every
GPU busy) and those the live holds' CDI specs name (`/etc/cdi/bcd-*.json`). A node that lied could otherwise have GPUs
in use repartitioned under their tenants.

### Signed specs [#signed-specs]

Since 03c023d the helper applies a unit only when a spec the control plane signed asks for exactly those settings.
Every `host.unit` request carries the signed `DesiredState` body that last set the host section (a full spec, or a
patch that sets `host`) and its worker signature; `bcd node` keeps that body in `data_dir/host-intent.json` (0600),
where it is worth nothing without the signature. The helper takes the command-signing roots the signature must chain
to from one of two places:

| Source                                                                                                                                                | When                                                                                                                                                                                      |
| ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `command_root_sha256` in root's `config.toml`: the SHA-256 of each command-signing root certificate's DER, lowercase hex, comma-separated, at most 16 | When set, these pins decide alone: the request names its roots, and each must be pinned (03c023d)                                                                                         |
| The trust bundle in root's join record, `/var/lib/bcd-hostconfig/join-response.pb`, which the JoinService key signed                                  | Without `command_root_sha256`, since 71ccf71, when the record carries command-signing roots: signed intent is then mandatory from the first request, whatever the node sends              |
| A newer `SignedTrustBundle` the node presents, checked against root's `join_signing_key_pin`                                                          | Without `command_root_sha256` (2d18327): it rotates the roots forward only, from the newest version accepted (the record's included), so a command-signing root rotation needs no new pin |

```toml title="/etc/bcd/config.toml"
join_signing_key_pin = "sha256/..."       # always there: the install's out-of-band pin
command_root_sha256 = "<64 hex digits>"   # optional: pin the roots themselves
```

Both pins are given at install, never taken from the control plane or the node, and a malformed value stops bcd from
starting. The helper refuses the request unless all of these hold:

* its roots check out: each is pinned in `command_root_sha256`, or they come from root's join record or a trust bundle
  signed by the pinned JoinService key whose version is not older than one the helper already accepted (an older one
  is a rollback). Once a presented bundle rotated the roots, leaving it out does not bring the record's older roots
  back: the node must present the current bundle (71ccf71);
* the signature verifies, from a certificate of the worker role, for a desired state;
* the body names this node. Since a966cd4 the node is the one root's own join record names: `bcd join`, run as root
  on a node with `[helper]` (the install script's join), writes the verified `JoinResponse`, as signed, to
  `/var/lib/bcd-hostconfig/join-response.pb` (0600, a root-only path) before the identity is written, and a failed
  record fails the join and leaves no identity. The helper takes the node id from that record, re-verified against
  root's `join_signing_key_pin`, so another node's signed spec is refused from the first request on. A record the
  pinned key did not sign refuses every `host.unit`; when root joins the machine again as another node, the binding
  moves and the generations start over. A node joined before a966cd4 has no record and keeps the older rule: the first
  signed spec the helper accepts fixes the node in `/var/lib/bcd-hostconfig/helper-intent.json`;
* its generation is not older than one the helper already accepted;
* the unit's settings are exactly the body's, and a clean is allowed only for a unit the body no longer names.

`helper-intent.json` also records the newest trust bundle version and that the node's specs are signed: from the
first trust bundle with command-signing roots on, a `host.unit` without a signed spec, or with a bundle without
roots, is refused, whatever the node sends. A body the helper already verified stays good after its short-lived
worker certificate expires, so a spec unchanged for days keeps working on its first signature; a new body needs a
valid one. A node and a helper of different versions fail closed, since the helper refuses the unknown request field.

**The remaining caveat.** Signed intent is root's decision since 71ccf71 (security review 2): before it, without
`command_root_sha256`, it turned on only once the node itself sent a trust bundle with roots, so a node that never sent
one got unchecked units. Now the pins or root's join record decide. What remains is an install where neither names
command-signing roots: no `command_root_sha256`, and a join record without roots (a deployment that signs nothing, or
a node joined before a966cd4 with no record). There the helper checks nothing until the node presents a bundle with
roots, and applies what the unprivileged node asks for, as on an install without signed intent, bounded only by
strict unit validation, the sysctl denylist, root writes only under root-only directories, and `k8s-join` only to
root-pinned `allowed_clusters`. `bcd doctor` (as root, with `[helper]`) fails on such an install ("bcd helper applies
host units unsigned unless the node itself presents a trust bundle"). Pinning `command_root_sha256` closes it from the
first request, and so does re-joining as root against a control plane that signs intent.
[`daemon install-script`](/docs/daemon/install#generate-the-script) writes the pins (59985ae): the digests given with
`--command-root-sha256`, else the local PKI bundle's command-signing root and, during a rotation, the added one, and
none when the bundle has no command signing; an upgrade by a script without them keeps the node's pins, and
`better-compute pki trust-version` prints them. The unit checks stay in force either way: that is why the `sysctl`
unit refuses keys that would run a program as root, and why a Kubernetes join under the helper needs root's own
allowlist of clusters (see [k8s-join](#k8s-join)). Programs a unit runs are bounded to 20 minutes each.

A second window, which node the specs must name, is closed since a966cd4 by root's join record. A node joined before
a966cd4 has no `join-response.pb` and still binds to the first signed spec it presents, so a node compromised before
that first `host.unit` could bind root to another node's specs. `bcd doctor`, run as root on a node with `[helper]`,
says which binding a node has (f77c6f9):

| Result | When                                                                                                                                                      |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ok`   | "host units bind to `<node>` (root's signed join record)"                                                                                                 |
| `warn` | The node joined before the record, so the helper binds on trust on first use; the finding says how to pin it                                              |
| `fail` | The record names another node than this machine's identity, or the pinned key did not sign it; the helper refuses host units either way                   |
| `fail` | Neither `command_root_sha256` nor root's join record names command-signing roots (71ccf71), as in the caveat above; a second finding beside the binding's |

To pin a node that joined before the record, re-join it as root: since a1c1a46 `bcd join --reset --yes` moves the
data directory's identity aside and joins, here with a new token (a `tpm` or `provisioned` re-join keeps the node id;
see [joining again with --reset](/docs/daemon/install#joining-again-with---reset)). Re-running the install script is not
enough: its join returns at once on a joined node and writes no record.

`TestHelperHostUnit` (e0fe5ec) covers the operation against a real helper socket and a fake host: a sysctl
re-planned and applied as root with the busy GPUs passed through, `kernel.core_pattern` refused with nothing written,
invalid names (`../etc`, `Sysctl`, empty, 65 characters) and an unknown unit refused, a clean that removes the
drop-in, and a helper without host units refusing the operation. `TestHostUnitNeedsTheSignedSpec` covers the signed
spec checks with pinned roots, and `intent_bundle_test.go` the trust-bundle path: a command-signing root rotation
(add, switch, drop) on a node pinned only by the join key, a replayed old bundle, a bundle from another key and a
rootless bundle. The agent's `TestSignedHostSectionKeptForTheHelper` covers the node keeping the body.

## kernel-modules [#kernel-modules]

Loads kernel modules now and at every boot, with their options. A sandbox or VPS host loads `kvm_intel` or
`kvm_amd`, `vhost_net` and `tun` this way, and a host that passes GPUs through to VMs loads `vfio-pci`.

| Setting   | Type                     | Rules                                                                                                                                     |
| --------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `load`    | `string[]`               | At most 64 names, each `^[a-z0-9_-]{1,64}$`                                                                                               |
| `options` | `Record<module, string>` | At most 64 entries; each value `key=value ...` of letters, digits and `_.,:=/ -`, at most 256 characters, each key `^[A-Za-z0-9_]{1,64}$` |

* **Files.** `/etc/modules-load.d/bcd.conf` lists the modules to load, sorted. `/etc/modprobe.d/bcd.conf` holds one
  `options <module> <options>` line per module with options, and is removed when there are none.
* **Loading.** A module not loaded (no `/sys/module/<name>`, dashes read as underscores) is loaded with
  `modprobe -- <module>`.
* **Changed options.** When a loaded module's parameter under `/sys/module/<name>/parameters/` differs from the
  option (`Y`/`N` and `1`/`0` compare equal), bcd reloads it (`modprobe -r`, then `modprobe`) only when its
  `refcnt` is 0. A module in use, such as `kvm_intel` under running VMs, is never pulled from under them: its new
  options take effect the next time it loads, which in practice means the next reboot. bcd does not report that it is
  waiting.
* **Refused options** (a894cd2). `vfio`'s `enable_unsafe_noiommu_mode` and `vfio_iommu_type1`'s
  `allow_unsafe_interrupts` fail the unit terminally: they let a passed-through device reach host memory. Provision
  refuses them too (`forbiddenModuleOptions`, b6d5548; `-` and `_` count alike).

## sysctl [#sysctl]

Kernel parameters, each a sysctl name with a scalar value: `{ "vm.swappiness": 10 }`.

* **Names** match `^[a-z0-9_]+(?:\.[a-z0-9_-]+)+$` (the same check as provision's). **Values** are numbers,
  booleans (`true` is `1`, `false` is `0`) or single-line strings of at most 4096 characters; whitespace is
  collapsed, since the kernel shows multi-value tunables tab-separated.
* **Files.** `/etc/sysctl.d/90-bcd.conf`, one `name = value` line per key, sorted, for early boot. bcd also sets
  each live value under `/proc/sys` that differs. An empty object removes the file.
* **Missing keys.** A key the kernel does not have yet is retried, not refused: bcd applies the others, then reports
  `this kernel has no <keys> (is the module that provides it loaded?)` and the section stays `PROGRESSING`. Load the
  module with `kernel-modules` and the same pass sets it.
* **Refused keys.** Some keys are refused whoever sends them, because they run a program as root or lower the
  host's defenses, and bcd helper applies this unit as root on behalf of an unprivileged node. A refused key fails
  the unit terminally and nothing is written. Since 1f0f683 provision holds the same list (`forbiddenSysctls`), so a
  profile or pool naming one of these keys is `INVALID_INPUT` before it reaches a node.

| Refused                                                                                                                                                                                                                                                                                                                           | Why                                               |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| `kernel.core_pattern`, `kernel.modprobe`, `kernel.usermodehelper.*`, `kernel.poweroff_cmd`, `kernel.hotplug`                                                                                                                                                                                                                      | Name or control a program the kernel runs as root |
| `fs.binfmt_misc.*`                                                                                                                                                                                                                                                                                                                | Registers interpreters the kernel runs            |
| `fs.suid_dumpable`, `fs.protected_*`, `kernel.randomize_va_space`, `kernel.kptr_restrict`, `kernel.dmesg_restrict`, `kernel.yama.*`, `kernel.kexec_load_disabled`, `kernel.unprivileged_bpf_disabled`, `kernel.perf_event_paranoid`, `kernel.sysrq`, `vm.mmap_min_addr`, `vm.unprivileged_userfaultfd`, `net.core.bpf_jit_harden` | Lower the host's defenses                         |
| `kernel.modules_disabled`                                                                                                                                                                                                                                                                                                         | Cannot be undone without a reboot                 |
| `user.*`, `kernel.unprivileged_userns_clone`, `kernel.apparmor_restrict_unprivileged_userns`                                                                                                                                                                                                                                      | Change who may create namespaces                  |

Since a894cd2 some keys take only their hardened values: raising the defense is fine, lowering it fails the unit
terminally (`<key> = <value> is refused: it would lower the host's defenses`). Provision holds the same rules
(`restrictedSysctls`, b6d5548), so a profile or pool with another value is `INVALID_INPUT` before it reaches a node.

| Key                                   | Allowed                                   |
| ------------------------------------- | ----------------------------------------- |
| `dev.tty.legacy_tiocsti`              | `0` only (no TIOCSTI keystroke injection) |
| `dev.tty.ldisc_autoload`              | `0` only (no autoloaded line disciplines) |
| `kernel.io_uring_disabled`            | `1` or `2` (io\_uring restricted or off)  |
| `net.ipv4.ip_unprivileged_port_start` | `1024` or above (ports below stay root's) |

## hugepages [#hugepages]

The host's persistent huge page pools, the memory VMs back their guests with, and the transparent huge page policy.

| Setting       | Type                               | Rules                                                                   |
| ------------- | ---------------------------------- | ----------------------------------------------------------------------- |
| `pages2m`     | integer                            | 2 MiB pages; 0 to 2^24 on the node (provision allows at most 4,194,304) |
| `pages1g`     | integer                            | 1 GiB pages; 0 to 2^24 on the node (provision allows at most 16,384)    |
| `transparent` | `'always' \| 'madvise' \| 'never'` | Written to `/sys/kernel/mm/transparent_hugepage/enabled`                |
| `perNuma`     | boolean                            | The counts are per NUMA node instead of host totals                     |

A pool's `policy.hugepages: { size: '2Mi' | '1Gi', count, perNuma? }` becomes this unit with `pages2m` or `pages1g`.

* **2 MiB pages** are allocated at runtime: bcd writes the count to `nr_hugepages` (the host's under
  `/sys/kernel/mm/hugepages/hugepages-2048kB`, or each NUMA node's under
  `/sys/devices/system/node/node<N>/hugepages` with `perNuma`) and reads back what the kernel allocated. A shortfall
  (memory fragmented or short) is retried.
* **1 GiB pages** are tried at runtime too, which works on a freshly booted host. When the kernel cannot find that
  much contiguous memory, bcd reserves the pages on the kernel command line instead: with `update-grub`, a GRUB
  drop-in `/etc/default/grub.d/90-bcd-hugepages.cfg` adding `hugepagesz=1G hugepages=<n>` to `GRUB_CMDLINE_LINUX`,
  then `update-grub`; with `grubby`, `grubby --update-kernel=ALL --args=...` (removing the previous arguments first);
  with neither, the step fails. It records the reserved count in `/var/lib/bcd-hostconfig/hugepages-1g-boot` and
  reports `... the boot loader reserves them from the next boot: reboot the node`, which is a
  [`RebootRequired`](#reboots). With `perNuma` the boot reservation is the host total (the count times the NUMA
  nodes), since the kernel spreads boot pages over the nodes. Once a boot reservation exists it is kept in step with
  the count, and `pages1g: 0` removes it.
* **A kernel without 1 GiB pages** (no `hugepages-1048576kB` directory) fails the unit terminally unless `pages1g`
  is 0; so does a kernel without transparent huge pages when `transparent` is set, and `perNuma` on a kernel that
  shows no NUMA nodes.
* **Mounts.** With `pages1g` above 0 the VMMs need a `hugetlbfs` of 1 GiB pages at `/dev/hugepages1G`. Where
  systemd runs (3c0fe02), bcd writes the mount unit `/etc/systemd/system/dev-hugepages1G.mount` (`What=hugetlbfs`,
  `Options=pagesize=1G`, `DefaultDependencies=no`, `Before=sysinit.target`, `WantedBy=sysinit.target`), runs
  `systemctl daemon-reload` and `systemctl enable dev-hugepages1G.mount`, and `systemctl start`s it when
  `/proc/mounts` does not show it, so the mount is there from boot, before any VMM starts. Without systemd, bcd mounts
  it itself (`mount -t hugetlbfs -o pagesize=1G`), again after each boot. 2 MiB pages use systemd's `/dev/hugepages`,
  which bcd does not manage.

The configured pools are what inventory reports (`memory.hugepages`, and per NUMA node since badb8c2), what the
capacity module books as `hugepages2Mi` and `hugepages1Gi` (and takes out of `memoryGiB`), and what bcd's own
[admission of instance starts and holds](/docs/daemon/delivery#instances) checks free pages against.

## cpu [#cpu]

The CPU frequency governor, and CPUs isolated from the kernel scheduler for pinned vCPUs and holds (1cad924).

| Setting    | Type                                                        | Rules                                                                                                                             |
| ---------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `governor` | `'performance' \| 'schedutil' \| 'ondemand' \| 'powersave'` | Set live on every CPU that has cpufreq                                                                                            |
| `isolate`  | string                                                      | A kernel cpulist such as `4-63,68-127`, at most 256 characters, CPUs up to 8191; never CPU 0, which keeps the host's housekeeping |

A pool's `policy.cpu: { governor?, isolate? }` becomes this unit (09669d3), and a profile's `hostConfig` may name it;
the pool's wins. Provision checks the same rules (`hostConfig.cpu.isolate must not include CPU 0 ...`), and bcd
checks them again, terminally.

* **Governor.** For every `cpu<N>` under `/sys/devices/system/cpu` with a `cpufreq/scaling_governor`, bcd writes
  the governor when it differs. A CPU whose driver does not list the governor in `scaling_available_governors` fails
  the unit terminally (`intel_pstate` offers only `performance` and `powersave`). A VM without cpufreq has nothing to
  set, and the unit converges.
* **Isolation.** `isolate` becomes kernel arguments, `isolcpus=managed_irq,domain,<list> nohz_full=<list>
  rcu_nocbs=<list>`, on the boot loader's command line: with `update-grub`, the drop-in
  `/etc/default/grub.d/90-bcd-cpu.cfg` (`GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX ..."`) and `update-grub`; with
  `grubby`, `--remove-args` of the previous arguments, then `--args`; with neither, the step fails (retried). The
  arguments bcd last set are recorded in `/var/lib/bcd-hostconfig/bootargs/cpu`, so an unchanged list plans nothing.
* **Reboot.** Until `/sys/devices/system/cpu/isolated` shows exactly the list, the unit reports
  `cpu: CPUs <list> are isolated from the next boot (now: "<what the kernel shows>"): reboot the node`, a
  [`RebootRequired`](#reboots).
* **Removal.** Leaving the spec (or an empty `isolate`) removes the arguments; the governor stays as it is.

The inventory reports the isolated CPUs, and the node label `cpu.isolated` counts them (see
[labels](/docs/guides/nodes/inventory-and-health#labels)). The control plane uses them (b6b804e): a pinned guest takes
isolated CPUs first (see [flavor placement](/docs/guides/vps/flavors#placement)), while hold cpusets and guest CPUs
never include the host's reserved CPUs, the lowest non-isolated, `policy.hostReserve.vcpu`.

## iommu [#iommu]

The IOMMU on, for GPUs passed through to VMs and sandboxes with VFIO: the runtimes bind each instance's GPUs to
`vfio-pci`, which needs IOMMU groups (3fc3628). Load `vfio-pci` itself with [`kernel-modules`](#kernel-modules).

| Setting       | Type    | Rules                                                                                                         |
| ------------- | ------- | ------------------------------------------------------------------------------------------------------------- |
| `passthrough` | boolean | Default `true`: adds `iommu=pt`, so only devices given to guests are translated and the host's own DMA is not |

A pool's `policy.iommu: { passthrough? }` becomes this unit (e4b0c01); set it on `vps-host` pools with GPU flavors.
A profile's `hostConfig` may name it too, and the pool's wins.

* **Arguments.** From the CPU vendor in `/proc/cpuinfo`: `GenuineIntel` gets `intel_iommu=on`, `AuthenticAMD` and
  `HygonGenuine` get `amd_iommu=on`, a CPU with no `vendor_id` line (Arm, whose SMMU is on by default) gets only
  `iommu=pt`, and any other vendor fails the unit terminally. They go on the boot loader's command line the same way
  as `cpu`'s, in `/etc/default/grub.d/90-bcd-iommu.cfg` or through `grubby`, recorded in
  `/var/lib/bcd-hostconfig/bootargs/iommu`.
* **Reboot.** Until `/sys/class/iommu` lists an IOMMU, the unit reports `iommu: the IOMMU is on from the next boot
  (<arguments>): reboot the node; check the firmware enables VT-d or AMD-Vi`, a [`RebootRequired`](#reboots). A
  machine whose firmware has VT-d or AMD-Vi off keeps reporting it after the reboot.
* **Removal.** Leaving the spec removes the arguments.

`bcd doctor` warns when a host with an active IOMMU has no `vfio-pci` driver loaded (see
[bcd doctor](/docs/daemon#modes)).

## nvidia-driver [#nvidia-driver]

The NVIDIA driver branch, installed and loaded.

| Setting  | Type    | Rules                                                                                                      |
| -------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `branch` | string  | Three digits, such as `"580"`                                                                              |
| `open`   | boolean | `true` for the open kernel modules, `false` for the proprietary ones; unset accepts whichever is installed |

A loaded driver of the branch and flavor is converged; bcd then only keeps `nvidia-persistenced` running where it is
installed (`systemctl enable --now`), so the GPUs stay initialized between jobs. Otherwise:

1. **Install.** When `modinfo nvidia` does not show the branch and flavor, bcd installs the branch's packages with
   the distribution's package manager (from `/etc/os-release`). Unset `open` installs the proprietary flavor. The
   repositories are the OS image's business: a package they do not carry fails the step, which is retried.

   | Distribution                                    | Packages                                                                                                                                              |
   | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
   | Ubuntu                                          | `nvidia-headless-<branch>-server` (`-open` for the open flavor) and `nvidia-utils-<branch>-server`                                                    |
   | Other Debian family                             | `nvidia-open-<branch>`, or `cuda-drivers-<branch>`                                                                                                    |
   | RHEL family (RHEL, Rocky, Alma, Fedora, CentOS) | the DNF module stream `@nvidia-driver:<branch>-open`, or `<branch>-dkms`                                                                              |
   | SUSE family                                     | `nvidia-open-driver-G06-signed-kmp-default` or `nvidia-driver-G06-kmp-default`, with `nvidia-compute-utils-G06` (these names do not carry the branch) |
   | Anything else                                   | Refused (terminal)                                                                                                                                    |

   `apt-get` runs with `DEBIAN_FRONTEND=noninteractive`, `--no-install-recommends` and `--force-confold` after an
   `apt-get update`; `dnf -y install` and `zypper --non-interactive install` elsewhere.
2. **Swap a loaded driver.** When another driver is loaded, bcd stops `nvidia-fabricmanager`, `nvidia-persistenced`
   and `nvidia-dcgm`, then unloads `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` and `nvidia`. `modprobe -r` refuses
   while anything holds the GPUs, and the step fails with `driver <version> is in use; reboot the node to load
   branch <branch>`.
3. **Load.** `modprobe nvidia` and `modprobe nvidia_uvm`, then a check of the loaded driver. If the loaded one is
   still not the branch and flavor, the step fails with `reboot the node`.

Both reboot cases are [`RebootRequired`](#reboots): retried, with the section `PROGRESSING` and the node condition
set, until someone reboots the machine.

## fabric-manager [#fabric-manager]

NVIDIA Fabric Manager, installed and running, on NVSwitch systems. It has no settings (`{}`).

* A host without NVSwitches (nothing under `/proc/driver/nvidia-nvswitch/devices`) has nothing to do, so a profile
  can name the unit for every node of a mixed pool.
* On an NVSwitch host it needs a loaded driver (until then it is retried) and installs Fabric Manager when
  `nv-fabricmanager` is missing: `nvidia-fabricmanager-<branch of the loaded driver>` on the Debian family,
  `nvidia-fabric-manager` elsewhere. It then runs `systemctl enable --now nvidia-fabricmanager` when it installed it
  or the service is not active.
* While an NVSwitch host runs without it, the unit reports `FabricManagerDown`: NVLink between its GPUs is down, and
  CUDA fails to initialize multi-GPU jobs. The node is NoSchedule while it lasts.

## gpu-settings [#gpu-settings]

Settings every GPU of the host shares, read and set with `nvidia-smi` (3fd1c41). It needs the driver: until
`nvidia-smi` answers, the unit is retried. With neither setting it does nothing.

| Setting           | Type    | Rules                                                                                                                                                 |
| ----------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `powerLimitWatts` | number  | Each GPU's board power cap; a whole number from 1 to 10,000 in provision's manifest, and within each GPU's own `power.min_limit` to `power.max_limit` |
| `ecc`             | boolean | ECC on or off                                                                                                                                         |

A pool's `policy.gpu: { powerLimitWatts?, ecc? }` becomes this unit (e4b0c01), and a profile's `hostConfig` may name
it; the pool's wins.

* **Power limit.** bcd queries each GPU's limit and range. A limit outside a GPU's range fails the unit terminally
  (`GPU <n> takes <min> to <max> W, not <limit>`). A GPU whose limit differs by a watt or more is set with
  `nvidia-smi -i <n> -pl <watts>`. The driver forgets the limit when it reloads, so the next pass sets it again. A GPU
  that reports no power management (`[N/A]`) is skipped.
* **ECC.** On each GPU whose current ECC mode is `Enabled` or `Disabled` (a GPU without ECC is skipped), bcd sets the
  pending mode with `nvidia-smi -i <n> -e 1` or `-e 0` when it differs. The mode changes at the GPU's next reset:
  until the current mode matches, the unit reports `gpu-settings: the ECC mode of GPU(s) <indexes> changes at their
  next reset: reboot the node`, a [`RebootRequired`](#reboots).

## mig-layout [#mig-layout]

Every MIG-capable GPU of the host partitioned to one layout.

| Setting  | Type                  | Rules                                                                              |
| -------- | --------------------- | ---------------------------------------------------------------------------------- |
| `from`   | `'pool' \| 'profile'` | Where the control plane took the layout from; bcd does not use it                  |
| `layout` | string                | The layout; empty means the control plane resolved none, and the unit does nothing |

The layout grammar:

| Layout                              | Meaning                                                                                                             |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `all-disabled`                      | MIG mode off                                                                                                        |
| `all-<profile>`                     | As many instances of one profile as fit on an empty GPU (`all-1g.10gb`), from `nvidia-smi mig -lgip`                |
| `mixed:<n>x<profile>+<profile>+...` | These GPU instances on each GPU (`mixed:3g.40gb+2x2g.20gb`); a count is 1 to 8, and a layout holds 1 to 8 instances |

A profile is `<n>g.<n>gb`. Profiles that carry a `+` of their own (such as `1g.10gb+me`) cannot be expressed. A
layout that does not parse, or a profile the GPU does not have, is terminal.

For each GPU that `nvidia-smi --query-gpu` shows as MIG-capable (current mode `Enabled` or `Disabled`):

* **Mode.** bcd turns MIG mode on (`nvidia-smi -i <n> -mig 1`) or off (`-mig 0`). When the change waits for a GPU
  reset, as on Ampere, it runs `nvidia-smi --gpu-reset -i <n>`; a failed reset fails the step with `MIG mode ...
  waits for a GPU reset, which failed (reboot the node)`, a [`RebootRequired`](#reboots).
* **Instances.** When the GPU's instances (`nvidia-smi mig -lgi`) are not the layout's, bcd destroys the compute
  instances, then the GPU instances ("not found" counts as done), and creates the layout's GPU instances, largest
  first, each with a compute instance filling it (`nvidia-smi mig -cgi <profiles> -C`).
* **Busy GPUs.** A GPU that a live hold uses (a hold's device id names it, or a MIG instance or slot of it,
  `<gpu>/gi<g>/ci<c>` or `<gpu>/s<k>`; 0f68c9a) is left alone. The
  other GPUs are repartitioned, and the unit then fails, retryably, with `mig-layout: <uuids> hold allocations; they
  are repartitioned once those end`. The control plane changes a pool's layout only once the GPUs' atoms are free
  (the [MIG layout gate](/docs/guides/capacity/fractional-gpus#the-mig-layout-gate)); this is bcd's own guard.

A hold on a MIG instance the layout has not created yet fails with `device-missing` until this unit has made it (see
[device assignments](/docs/daemon/delivery#device-assignments)).

## gpu-sharing [#gpu-sharing]

How a `timeslice` or `mps` pool's GPUs are shared.

| Setting    | Type                   | Rules                                                       |
| ---------- | ---------------------- | ----------------------------------------------------------- |
| `mode`     | `'mps' \| 'timeslice'` | Required                                                    |
| `replicas` | integer                | 1 to 48 (MPS serves at most 48 clients per GPU); default 10 |

* **`mps`** needs `nvidia-cuda-mps-control`, which ships with the driver's utilities (until it is installed the
  unit is retried). bcd writes `/etc/systemd/system/bcd-mps.service` and, when it changed, runs `systemctl
  daemon-reload`, `enable` and `restart`; otherwise it starts the service when it is not active. The service runs
  the MPS control daemon with `CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps` and `CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps`,
  sets every GPU's compute mode to `EXCLUSIVE_PROCESS` before it starts and back to `DEFAULT` after it stops, and
  limits each client by default to `100 / replicas` percent of the SMs (at least 1), with
  `set_default_active_thread_percentage`. It restarts on failure after 5 seconds. While the service is not active
  the unit reports `MPSDown`, since its clients fail, and the node is NoSchedule. A slot hold gets
  `CUDA_MPS_PIPE_DIRECTORY` in its env when `/run/nvidia-mps` exists.
* **`timeslice`** uses the GPUs' default time slicing: the replicas are the slots the control plane hands out. bcd
  stops and removes `bcd-mps.service` if it installed one.

Provision renders this unit from the pool (1f0f683): a pool whose `sharing` is `mps` or `timeslice` gets
`gpu-sharing { mode, replicas }`, with `replicas` from `policy.sharing.replicas`, by default 10 for MPS and 4 for
time slicing, the same numbers of slots the capacity module derives. The limits agree end to end: `pools.create` and
`pools.update` refuse more than 48 replicas (37dcd02), and since a708aa6 the inventory makes at most 48 slot atoms
per GPU even for a pool that bypassed the check. A profile may also name the unit.

## container-toolkit [#container-toolkit]

The NVIDIA container toolkit, the container runtimes configured with it, and the vendor CDI spec.

| Setting | Type    | Rules                                                       |
| ------- | ------- | ----------------------------------------------------------- |
| `cdi`   | boolean | Default `true` (in the control plane's manifest and in bcd) |

* **Install.** `nvidia-container-toolkit` with the distribution's package manager when `nvidia-ctk` is missing.
* **Runtimes.** For each runtime installed on the host, containerd (`containerd`, `/etc/containerd/config.toml`)
  and Docker (`dockerd`, `/etc/docker/daemon.json`), whose configuration does not mention
  `nvidia-container-runtime` yet: `nvidia-ctk runtime configure --runtime=<name>` (with `--cdi.enabled` when `cdi`
  is on), then `systemctl restart containerd` or `docker`. A runtime already configured is not configured again, so
  turning `cdi` on later does not rewrite it.
* **CDI only.** With `cdi`, since b61b9c5 (security review 2) the unit sets
  `accept-nvidia-visible-devices-envvar-when-unprivileged` and `accept-nvidia-visible-devices-as-volume-mounts` to
  `false` in `/etc/nvidia-container-runtime/config.toml` (`nvidia-ctk config --in-place --set <key>=false`) unless
  the file already says so. `nvidia-ctk runtime configure --cdi.enabled` points containerd at the legacy
  `nvidia-container-runtime`, and the packaged configuration leaves the first key commented out, at its permissive
  default: any unprivileged container, any pod, could then choose GPUs, other tenants' included, with
  `NVIDIA_VISIBLE_DEVICES`, and on a `k8s-join` node bcd's own
  [`ForeignGpuExposure` check](/docs/guides/capacity/kubernetes#gpu-isolation) withdrew every GPU. The runtime reads
  its configuration per container, so nothing restarts. `cdi: false`, the legacy mode that hands GPUs out through the
  variable, changes neither key.
* **CDI.** With `cdi`, bcd generates `/etc/cdi/nvidia.json` with `nvidia-ctk cdi generate --format=json --device-name-strategy=uuid` (replacing the `/etc/cdi/nvidia.yaml` older versions wrote) and records
  what it describes in `/var/lib/bcd-hostconfig/cdi-nvidia.stamp`: a SHA-256 of the loaded driver version and
  `nvidia-smi -L`, which lists every GPU and MIG device. The spec is generated again whenever that changes, so a
  driver update or a new MIG layout gets a new spec in the same pass. It needs a loaded driver; until then the unit is
  retried. `cdi: false` generates nothing and removes nothing.

This spec is NVIDIA's. bcd's own hold specs are `/etc/cdi/bcd-<allocation>.json`, written by the `devices`
converger; the [Kubernetes device plugin](/docs/guides/capacity/kubernetes#the-device-plugin) names both.

## k8s-join [#k8s-join]

Joins the machine to a Kubernetes cluster as a worker, and keeps its kubelet and labels (88ecb07, wired into
the agent, the helper and the configuration in 7579184). Compute never runs a control plane and never schedules
pods; it hands a cluster machines and GPUs. The whole flow is on [Kubernetes workers](/docs/guides/capacity/kubernetes).

| Setting             | Type                       | Rules                                                                                                                                                                                          |
| ------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `server`            | string                     | The API server, `https://host[:port]` (port 443 when left out), no credentials, path or query                                                                                                  |
| `caCertSha256`      | string                     | The SHA-256 of the cluster CA's public key, 64 hex digits, optionally `sha256:` (kubeadm's `--discovery-token-ca-cert-hash`)                                                                   |
| `tokenRef`          | string                     | A `secret://` reference to a kubeadm bootstrap token (since e4b0c01 provision refuses `vault://` and `env://`, which the broker would not release); the token never travels in the spec        |
| `kubernetesVersion` | string                     | A release such as `1.34.1`; picks the package repository when kubeadm and the kubelet are missing                                                                                              |
| `nodeLabels`        | `Record<string, string>`   | At most 64 Kubernetes labels; the `kubernetes.io` and `k8s.io` prefixes are refused                                                                                                            |
| `taints`            | `string[]`                 | `key[=value]:NoSchedule\|PreferNoSchedule\|NoExecute`, no key and effect twice                                                                                                                 |
| `gpu`               | boolean                    | Default `true`: compute-bound GPUs reach the kubelet                                                                                                                                           |
| `gpuApi`            | `'device-plugin' \| 'dra'` | How they reach it: bcd's device plugins (the default) or its DRA driver `gpu.bcd.dev` (7d24836), never both; see [device plugin or DRA](/docs/guides/capacity/kubernetes#device-plugin-or-dra) |

A pool's `policy.kubernetes: { apiServer, caCertSha256, joinTokenRef, labels?, taints?, kubernetesVersion?, gpu?, gpuApi? }`
becomes this unit when it names the cluster: `apiServer`, `caCertSha256` and `joinTokenRef` go together, and a
partial set is `INVALID_INPUT` (7db110a). A pool without them is one whose nodes already run bcd as a DaemonSet
inside the cluster, and needs no unit (see [on Kubernetes](/docs/daemon/kubernetes)). bcd checks the settings again on
the node, since labels, taints and the fetched token end up in YAML and on command lines. Each pass converges, in
order:

1. **Packages.** When `kubeadm`, `kubelet` or `kubectl` is missing, bcd adds the `pkgs.k8s.io` repository of
   `kubernetesVersion`'s minor release and installs them: on the Debian family the source
   `/etc/apt/sources.list.d/bcd-kubernetes.list` with its key fetched over https (downloaded to
   `/var/lib/bcd-hostconfig/k8s-release.key`) into `/etc/apt/keyrings/bcd-kubernetes.gpg`, `apt-get install`, then
   `apt-mark hold`; on the RHEL family `/etc/yum.repos.d/bcd-kubernetes.repo` and `dnf install --disableexcludes=bcd-kubernetes`. It installs `containerd` too when it is missing. Missing packages without
   `kubernetesVersion`, or another distribution, are terminal. A pass that installed packages ends there, and the
   next one plans the rest.
2. **Kernel.** `overlay` and `br_netfilter` in `/etc/modules-load.d/90-bcd-k8s.conf`, loaded now; then
   `net.bridge.bridge-nf-call-iptables`, `net.bridge.bridge-nf-call-ip6tables` and `net.ipv4.ip_forward` at 1 in
   `/etc/sysctl.d/90-bcd-k8s.conf` and live; and `swapoff -a` whenever `/proc/swaps` lists swap, since the kubelet
   refuses to run with it. Do not set those `net.bridge.*` keys to 0 in the `sysctl` unit on the same machine: the two
   units would set them back and forth.
3. **containerd.** `/etc/containerd/config.toml` imports `/etc/containerd/conf.d/*.toml` (a new file when there is
   none; an `imports` list without it is terminal, for the operator to fix), and bcd's drop-in
   `/etc/containerd/conf.d/90-bcd-k8s.toml` turns CDI on (`/etc/cdi`, `/var/run/cdi`) and the systemd cgroup driver
   for runc, in config version 2 or 3 as the installed containerd reads. containerd is restarted when either
   changed, and started when it is not running.
4. **Join.** While `/etc/kubernetes/kubelet.conf` does not exist, bcd fetches the bootstrap token, writes a
   `JoinConfiguration` (API `kubeadm.k8s.io/v1beta4` from kubeadm 1.31, `v1beta3` before) with the endpoint, the
   token, the CA pin, the Node name, containerd's socket, the labels and the taints to
   `/var/lib/bcd-hostconfig/k8s-join.yaml` (0600), runs `kubeadm join --config`, and removes the file whatever
   happened. The Node name is the host name, lower-cased with `_` and spaces as `-`; one that is not a DNS subdomain
   is terminal. It records what it manages in `/var/lib/bcd-hostconfig/k8s-join.json`.
5. **Kept.** Once joined: a kubelet whose kubeconfig names another API server is terminal (`reset it by hand
   (kubeadm reset) to move it`): bcd never resets a node joined elsewhere. It starts the kubelet when it is not
   active, and keeps the Node's labels with the kubelet's own credentials (`kubectl --kubeconfig
   /etc/kubernetes/kubelet.conf label`), removing only labels it set before or labels under `bcd.dev/`. Besides the
   spec's labels, bcd sets `bcd.dev/node-id`, `bcd.dev/site`, `bcd.dev/gpu.count`, `bcd.dev/gpu.product` and
   `bcd.dev/gpu.driver` from its identity and inventory.

**Taints are set once** (6ef198b). The NodeRestriction admission plugin forbids a kubelet to change its own Node's
taints, so bcd sets them only at registration, in the `JoinConfiguration`, and records them in `k8s-join.json` as
registered. When the unit's taints later differ from the Node's (the pool's `taints` changed), bcd does not retry: the
unit reports the condition `KubernetesTaintsDrift` (reason `NodeRestriction`) while they differ, and its message
carries the command a cluster admin runs, `kubectl taint node <node> --overwrite <taints>`, which sets the unit's
taints and removes those bcd registered that the unit no longer names. The gateway records it as an informational
node condition. Labels are not affected: the kubelet may label its own Node outside `kubernetes.io` and `k8s.io`.

**The token.** `bcd node` fetches it with a `SecretRequest` that names workflow `host:k8s-join` and step `tokenRef`.
The secret broker releases it only while the node's current spec names that reference in that field (binding rule 3,
e93bf9c; see [sealed secrets](/docs/operations/security#sealed-secrets)). The token is removed once the join
succeeded.

**Which clusters.** Joining a cluster gives it root on the machine (privileged pods), so root decides which clusters
are allowed, in `config.toml`:

```toml title="/etc/bcd/config.toml"
[kubernetes]
allowed_clusters = "https://10.0.0.1:6443#sha256:<64 hex digits>,https://k8s.example.com#sha256:<64 hex digits>"
```

Each entry is an API server and its CA pin; a malformed entry stops bcd from starting. Since 9f860ad the list is
required whether `bcd node` runs as root or under bcd helper, where the helper builds the unit with root's list: an
empty list refuses every join (`joining a Kubernetes cluster needs [kubernetes] allowed_clusters in bcd's config`),
and `bcd doctor` fails a joined node without one. Since the helper cannot ask the control plane for secrets, the node
fetches the token first and stages it at `/var/lib/bcd-hostconfig/k8s/bootstrap-token` (helper file kind
`k8s-token`, mode 0600, in root's directory, so the node cannot point it at another file), where the helper's join
reads it (1bb78d6).

## Examples [#examples]

### A sandbox host [#a-sandbox-host]

A pool of VM hosts for [sandboxes](/docs/guides/vps/sandboxes): KVM with nested virtualization, `vhost_net` and
`tun` for the guests' NICs, 1 GiB pages per NUMA node for pinned guests, and CPUs isolated for them. The modules,
pages and CPUs belong to the pool, so they go in its policy, here in a [FleetConfig](/docs/guides/fleet-config)
document:

```json title="fleet.json"
{
  "version": 1,
  "pools": [
    {
      "name": "sandbox-fra",
      "site": "fra",
      "class": "vps-host",
      "policy": {
        "kernelModules": {
          "load": ["kvm_intel", "vhost_net", "tun", "br_netfilter"],
          "options": { "kvm_intel": "nested=1" }
        },
        "hugepages": { "size": "1Gi", "count": 64, "perNuma": true },
        "cpu": { "governor": "performance", "isolate": "4-63,68-127" }
      }
    }
  ]
}
```

Kernel parameters come only from a profile's `hostConfig`, so the machines' deploy profile carries them:

```ts title="profiles/sandbox-host.ts"
hostConfig: {
  sysctl: {
    // Keep bridged guest traffic out of the host's IP firewall. bcd's instance rules live in the
    // bridge family and do not need br_netfilter, but the keys exist only once it is loaded.
    'net.bridge.bridge-nf-call-iptables': 0,
    'net.bridge.bridge-nf-call-ip6tables': 0,
  },
},
```

On such a host, `kernel-modules` writes both drop-ins and loads the modules; `kvm_intel` loaded before without
`nested=1` is reloaded only if no VM holds it. `sysctl` then finds the `net.bridge.*` keys in the same pass. The
`hugepages` unit sets 64 pages on each NUMA node and mounts `/dev/hugepages1G`; on a host that has been up for a while
it may instead set up the boot loader and report `RebootRequired`. The `cpu` unit sets the governor at once and adds
the isolation arguments, which also wait for that reboot; one reboot serves both. An Intel host needs `kvm_intel`, an
AMD one `kvm_amd`; use a `when:` expression on a profile unit, or separate pools, for mixed hardware.

### A GPU host [#a-gpu-host]

A MIG pool of H100 hosts with the open driver of branch 580, Fabric Manager where there are NVSwitches, the pool's
layout, and the container toolkit with CDI:

```ts title="profiles/ubuntu-gpu-host.ts"
hostConfig: {
  'nvidia-driver': { branch: '580', open: true },
  'fabric-manager': {},
  'mig-layout': { from: 'pool' },
  'container-toolkit': { cdi: true },
},
```

with the pool's `policy.migLayout`, for example `"all-1g.10gb"`. In one pass bcd installs and loads the driver (or
reports that a reboot is needed, and still converges the units after it that do not need the new driver), installs
and starts Fabric Manager on an NVSwitch board, turns MIG mode on and creates seven `1g.10gb` instances on each idle
GPU, configures containerd with the toolkit, and generates `/etc/cdi/nvidia.json` for the new MIG devices. The
inventory refresh that follows reports them, and the control plane derives the pool's `gpu.mig` atoms from it.

## What the tests cover [#what-the-tests-cover]

| Test                                                                                                                                                                                      | Checks                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/hostconfig` (22 tests, against `FakeSystem`)                                                                                                                                    | sysctl converges, is idempotent on tab-separated values, corrects drift and cleans up; refused keys and object values are terminal, a missing key is retried; kernel modules before the sysctl they provide, in one pass, and no reload of a module in use; unknown units and refused settings do not block other units, and a unit waiting for a reboot does not hold back the rest (c21ef84); 2 MiB and 1 GiB pages, the mount unit under systemd, the GRUB and grubby fallbacks, per-NUMA pools and the host-total boot reservation; the `cpu` unit's cpulists, governor and isolation; the `iommu` arguments; `gpu-settings`' power limit and ECC; the driver installed and loaded on Ubuntu, a flavor mismatch, and a driver in use asking for a reboot; the toolkit's runtime configuration and CDI regeneration after a MIG change; Fabric Manager only with NVSwitches, and `FabricManagerDown`; the MIG layout grammar, a mixed layout largest first, a busy GPU left alone, and `all-disabled`; MPS with its unit file and `MPSDown`, then time slicing removing it; the RHEL family's driver stream |
| `internal/hostconfig` `TestRealKernel` (Linux, `BCD_HOSTCONFIG_REAL=1` as root)                                                                                                           | On the real kernel: a sysctl set and put back, a tab-separated multi-value tunable converging in one pass, the transparent huge page selector, two more 2 MiB pages and back, and root refusing to write under `/tmp`, a world-writable parent or a symlinked one (60143bf, 02485b3). Passes on WSL Ubuntu                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `internal/agent` `TestHostSectionConverges`, `TestHostSectionReportsRebootRequired`, `TestHostDriftRepairedWithoutANewSpec`, `TestExternalHostAppliesNoUnit`                              | A real `Agent` under a fake root (programs are recorded, not run): a sysctl applied and its drop-in written, `FabricManagerDown` in the status report, the `RebootRequired` condition, drift repaired by the periodic pass with no new spec (3c0fe02), and `[host] units = false` applying nothing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `internal/helper` `TestHelperHostUnit`, `TestHostUnitNeedsTheSignedSpec`; `internal/agent` `TestSignedHostSectionKeptForTheHelper`                                                        | `host.unit` through a real helper socket (e0fe5ec), and the signed spec checks (03c023d); see [under bcd helper](#under-bcd-helper)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `internal/kube` `TestJoinFromBareHost`, `TestJoinRefusesAnotherCluster`, `TestJoinSettingsAreCheckedAgain`, `TestJoinUnderHelper`, `TestContainerdImport`, `TestJoinConfigurationV1beta3` | The join unit against `FakeSystem`: from a bare host to a joined Node with its labels and taints, the refusal to move a node joined elsewhere, settings checked again, the helper's allowlist and staged token, containerd's import line                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `tests/e2e/bcd-host.test.ts` (`BCD_E2E=1` on Windows)                                                                                                                                     | A `metal-host` pool with `policy.hugepages` (2 MiB, 64) and `policy.kernelModules` (`tun`, `vhost_net`): provision renders the units, the gateway sends them, and a fakehw bcd applies them under its fake root (`nr_hugepages` 64, `/etc/modules-load.d/bcd.conf`, `modprobe` recorded) (91f5e81)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

Apart from `TestRealKernel`, nothing here has run against a real machine.

  - [Host config units](/docs/reference/host-units): Each unit's settings manifest, generated from provision.

  - [Delivery](/docs/daemon/delivery): Desired state, sections, device assignments and instances.

  - [Profiles and images](/docs/guides/bare-metal/profiles-and-images#host-config): Where a profile's hostConfig is declared.

  - [Kubernetes workers](/docs/guides/capacity/kubernetes): From policy.kubernetes to GPUs in pods.

  - [Fractional GPUs](/docs/guides/capacity/fractional-gpus): MIG layouts, slots and the layout gate.


# Node identity (/docs/daemon/identity)

> What makes up a bcd node's identity on disk, how bcd join creates it with each attestor, how renewal replaces the key every day, what revocation does on the node, how bcd join --reset recovers, and how node keys live in the TPM.



A node's identity is a P-256 key and a short-lived certificate the control plane's JoinService issued for it. bcd
creates it once with `bcd join`, replaces the key and certificate at 60% of every certificate's lifetime, and uses
the key for every TLS handshake and every statement the node signs. This page describes the node's side: the files,
the join, renewal, revocation, recovery with `bcd join --reset`, and node keys held in the host's TPM.

The control plane's side is elsewhere: [joining nodes](/docs/guides/nodes/joining) covers tokens, the join
transaction and certificates, [attestation](/docs/guides/nodes/attestation) covers each attestor and the TPM proofs,
and [PKI](/docs/operations/pki) covers the keys and pins behind it all.

> **Real joins, software TPMs only.** 
  `tests/e2e/bcd.test.ts` joins real bcd binaries (fakehw builds) to a real control plane and gateway over mTLS, with
  renewal. `tests/e2e/bcd-tpm.test.ts` runs a real `bcd join --attestor tpm` and a renewal certified by the AK against
  the dev stack, with `tpmsim`, bcd's software TPM. Every TPM test uses a software TPM: none has run against a hardware
  TPM, a real manufacturer's EK certificate or a real `/dev/tpmrm0`. The round 6 development host has no TPM.

## What the identity is [#what-the-identity-is]

Everything lives in the data directory (`data_dir`, default `/var/lib/bcd`):

| File            | Mode | What                                                                                                                                                                                     |
| --------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `identity.json` | 0600 | The identity the verified `JoinResponse` and later renewals established (below)                                                                                                          |
| `node.key`      | 0600 | The node key: a PKCS#8 PEM (`PRIVATE KEY`), or, for a key in the TPM, only its record, a PEM block of type `BCD TPM NODE KEY` whose bytes are the 32-byte unique the key is derived from |
| `node.key.next` | 0600 | The key a renewal is certifying, until its certificate is stored                                                                                                                         |
| `journal.db`    | 0600 | The command journal; it belongs to one node id, and `bcd node` refuses another node's                                                                                                    |

`identity.json` holds:

| Field                                       | Meaning                                                                                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nodeId`, `siteId`, `trustDomain`, `role`   | Read from the leaf certificate's SPIFFE ID and its `OU=bc-role:<role>` attribute (`node`, `site` or `ramdisk`)                                                                                     |
| `chain`                                     | The certificate chain, leaf first, DER                                                                                                                                                             |
| `notAfter`                                  | The leaf's expiry                                                                                                                                                                                  |
| `trustVersion`, `trustRoots`, `trustBundle` | The trust bundle the node holds: its version, command-signing roots and the whole serialized bundle (approver keys for the destructive-batch quorum). No roots means signed intent is not enforced |
| `signedTrustBundle`                         | The same bundle as the JoinService key signed it, from a `Welcome`; [bcd helper](/docs/daemon/helper#hostunit) takes host units' roots from it                                                     |
| `serverTime`                                | The server time high-water mark, mirrored from the journal at each `Welcome`, so a lost journal does not reset it                                                                                  |

The identity outlives a lost journal: a corrupt or wiped journal starts over with a new journal id, but the node keeps
its key and certificate (see [journal loss](/docs/daemon/failure-modes#journal-loss)).

### Loading it [#loading-it]

Every agent mode loads the identity with `LoadState` at start:

1. No `identity.json` is `this node has not joined (run bcd join)`.
2. A `node.key.next` whose public key the stored leaf certifies is promoted to `node.key` (a renewal stopped after it
   stored the certificate); any other `node.key.next` is removed (a renewal stopped before).
3. `node.key` must be the leaf's key. If it is not: `the node key does not match the node certificate; join again:
   bcd join --reset --yes, then bcd join`. For a TPM key the message adds `the TPM derives another key from node.key
   (was the TPM cleared?)`, since a cleared owner hierarchy derives another key from the same record.

`bcd doctor`'s `identity` check reports the same: a warning when the node has not joined, a failure for an identity
that does not load or an expired certificate, each ending with the way back.

## Joining [#joining]

`bcd join` creates the identity. It runs as root, takes the join credential from one source, and never from a
command-line argument (a positional argument is refused without being echoed).

| Credential                | How                                                                                                                                                                                   | Attestor      |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| A join token              | `BCD_JOIN_TOKEN` (unset once read), else the first line of standard input (at most 4 KiB)                                                                                             | `join-token`  |
| A token file              | `--token-file PATH`: a regular file, mode 0600 or 0400, at most 4 KiB; deleted once the join succeeds or the machine had already joined (a read-only mount logs a warning instead)    | `join-token`  |
| A provisioned token       | `--node-id <node>` with a token: the OS a deploy installed joins as that node, and the join fails unless the issued identity is that node                                             | `provisioned` |
| The TPM                   | `--attestor tpm`: no token is read or sent (`BCD_JOIN_TOKEN` is dropped; `--token-file` is a usage error)                                                                             | `tpm`         |
| The EC2 instance identity | `--attestor cloud-iid` on EC2: the instance identity document and its RSA-SHA256 signature from IMDSv2, sent as `{"provider":"aws","document","signature"}`; no token is read or sent | `cloud-iid`   |
| The kernel command line   | `bcd ramdisk` only: the `bc.token` and the site agent's `bc.assertion`                                                                                                                | `discovery`   |

`--role site` joins a site agent with a site token; the token decides what is allowed. `--server`, the three pin flags
and `--data-dir` bootstrap a machine that has no `config.toml` from the control plane's well-known document (see
[joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document)). bcd produces no
`x509pop` evidence, and `cloud-iid` evidence only on AWS EC2 (the GCP variant, whose token must name the CSR, is not
implemented in bcd). A bcd node whose certificate expired offline re-joins with a token, the TPM or `--reset`;
`cloud-iid` admits an instance once, within 30 minutes of its first boot, so it is a first join only.

### What the join does [#what-the-join-does]

`agent.Join` (`daemon/internal/agent/join.go`):

1. **The key.** For a `tpm` join on a host whose node keys use the TPM, the key is made in the attestor's own TPM
   connection so the AK can certify it; otherwise a key in the host's TPM when there is one, else a PKCS#8 file key
   (see [node keys in the TPM](#node-keys-in-the-tpm)). Then a PKCS#10 CSR.
2. **The seal.** A token is sealed to the JoinService encryption key, which must match `join_encryption_key_pin`
   (from `join-encryption.jwk` beside the config, or the well-known document with `--server` when that file is
   absent). The seal is bound to the CSR's SHA-256.
3. **`Join`.** Sent over TLS pinned to the serving CA (`serving_ca_spki_pin`), with the CSR, the hardware claims
   (product UUID and serials, permanent MACs, GPU UUIDs, hostname, boot id), the attestor, its evidence, the daemon
   version and the role. Retryable failures back off, up to 20 attempts; a gateway's `grpc-retry-pushback-ms` is
   waited out exactly.
4. **The answer.** `pki.OpenJoinResponse` verifies the signature under `join_signing_key_pin`, that `csr_sha256` is
   this request's, that the leaf certifies this key, the SPIFFE ID and the role. A deferral is answered with the same
   CSR: a `JoinChallenge` for this attestor (at most 3 rounds, refused once its `not_after` passed), or a `JoinPending`
   approval, waited out (at least one second) and, for an attestor with a challenge, followed by a fresh round.
5. **Root's record.** On a node with `[helper]`, when `bcd join` runs as root, the verified `JoinResponse` is written
   as signed to `/var/lib/bcd-hostconfig/join-response.pb` (0600) before the identity (a966cd4). A failed record fails
   the join and leaves no identity. bcd helper takes the node id and the command-signing roots for host units from it.
6. **Persist.** If the data directory held another node's identity, its `journal.db` is moved aside as
   `journal.db.<oldNode>-<unix time>`. A stale `node.key.next` is removed, then `node.key` and `identity.json` are
   written.
7. **Hand over.** Under `[helper]`, root `Lchown`s `identity.json` and `node.key` to the `[helper] user`, so the
   unprivileged `bcd node` can read them (b4fcc2e); a name swapped for a symlink changes only the link.

It prints `joined as <node> (site <site>, role <role>); certificate valid until <time>`. On a machine whose identity
loads, `bcd join` reads no credential, answers `already joined as <node> (site <site>, role <role>)` and exits 0, so a
join unit can run on every boot; a different `--node-id` is refused. Every refusal from the control plane is the same
`ACCESS_DENIED` ("Join refused"); the reason is only server-side (see [refusals](/docs/guides/nodes/joining#refusals)).

### The pins [#the-pins]

| `config.toml` key                | Protects                                                                                                          |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `serving_ca_spki_pin`            | The TLS connection to the gateway: never system roots                                                             |
| `join_encryption_key_pin`        | The key a token is sealed to                                                                                      |
| `join_signing_key_pin`           | The `JoinResponse` signature, trust bundles, and root's join record under the helper                              |
| `command_root_sha256` (optional) | Command-signing roots for host units under the helper (see [signed specs](/docs/daemon/host-config#signed-specs)) |

The install script writes them from the PKI bundle; a malformed pin stops bcd from starting.

## Renewal [#renewal]

A node certificate lasts at most 24 hours (a ramdisk's at most 2). The agent renews at 60% of the leaf's lifetime
(`renewFraction`), with `Renew` on the gateway, which relays to JoinService. `renewOnce`:

1. **Chooses the next key.** If the leaf names a TPM AK (`urn:bc:tpm-ak:`, written after a certified `tpm` join;
   d94f1f3), bcd loads that AK alone, checks its TPM name is the one the certificate names, and makes the next key in
   that TPM. Otherwise the next key goes into the TPM when the host has one, so a file-key node moves into the TPM
   here, else it is a file.
2. **Writes `node.key.next` first**, then builds the request: a new CSR, the current chain, the client time and a
   proof of possession by the current key (purpose `renew-pop`).
3. **Attaches the AK's proof** for a certificate that names one: `key_attestation` with the AK's public area, the new
   key's public area, and the AK's `TPM2_Certify` of it over `sha256(csr)`.
4. **Checks the answer**: the new leaf names the same node, site, trust domain and role, and certifies the new key.
5. **Stores in order**: `node.key.next` again (in case something removed it), then `identity.json`, then renames
   `node.key.next` over `node.key`. A crash at any point leaves a key and certificate that match. The new state keeps
   the trust version, roots, bundle and server time. The old key is unloaded 5 minutes later, once nothing can still
   be signing with it.

A failed renewal is logged as `renewal failed` with the attempt and `expiresIn`, and retried with backoff from 30
seconds to 10 minutes, or exactly when a busy gateway says (it allows 6 renewals per node and hour). There is no
downgrade for a certificate that names an AK: when the TPM is unavailable, or derives another AK, the renewal fails
before anything is sent, and keeps failing until the certificate expires.

| At renewal                     | Message                                                                                                                                                                                                              |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The TPM cannot be opened       | `renewal: the certificate's node key is in the TPM, which is unavailable: ...`                                                                                                                                       |
| Another AK                     | `renewal: this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join` |
| No next key in the TPM         | `renewal: the next node key in the TPM: ...`                                                                                                                                                                         |
| No certification               | `renewal: the TPM's attestation key certifying the next key: ...`                                                                                                                                                    |
| An answer for another identity | `renewal: the certificate is for <site>/<node> (<role>), not ...`                                                                                                                                                    |
| An answer for another key      | `renewal: the certificate does not certify the new key`                                                                                                                                                              |

The control plane's checks and refusal reasons are on [renewal](/docs/guides/nodes/joining#renewal) and [renewing a
TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key). A certificate that expired cannot be renewed;
see [joining again](#joining-again-with---reset).

## Revocation [#revocation]

The control plane revokes a node's certificates with `nodes.revokeIdentity`, by deleting the node, or on its own for a
certificate that looks cloned (see [revoking a node's
identity](/docs/guides/nodes/operating-nodes#revoking-a-nodes-identity)). A gateway with a revocation feed then sends
the node's live session a `RevocationNotice` and closes it.

On the node, the notice is logged as `certificate revoked` with its reason, the agent stops (`ErrRevoked`), and
`bcd node` exits 1 with `agent: this node's certificate was revoked`. It does not redial with that certificate, and
nothing on the node deletes or renames the identity. The revoked certificate cannot renew, re-join or open a session
again (the gateway answers `UNAUTHENTICATED`), and a later identity generation means older certificates stay dead. The
machine comes back only by joining again with fresh evidence: a new token, or the TPM, after `bcd join --reset`.

## Joining again with --reset [#joining-again-with---reset]

An identity that loads but no longer works (expired, revoked, refused at renewal, naming an AK this TPM no longer has)
makes `bcd join` answer `already joined` forever. Since a1c1a46, `--reset` moves it aside first:

```bash
# as root: show what would move (moves nothing, exits 2)
bcd join --config /etc/bcd/config.toml --reset
# move it aside and join again, with a new token on standard input
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt
# or with the TPM
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm
```

| Behaviour       | Detail                                                                                                                                                                                     |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Root only       | Outside fakehw builds a non-root `--reset` fails with `bcd join --reset: run it as root`                                                                                                   |
| What moves      | `identity.json`, `node.key`, `node.key.next` and `journal.db`, whichever exist, each renamed `<name>.reset-<UTC time>` (for example `node.key.reset-20260926T114200Z`). Nothing is deleted |
| Without `--yes` | Prints the files, that the following join makes a new key and joins again (a new node unless the attestor proves this machine and reclaims it), and exits 2                                |
| Then            | The usual join, with the usual credential; if it fails, the identity stays moved aside and a later `bcd join` joins                                                                        |

Whether the machine comes back as the same node depends on the attestor. `tpm` with the same EK and `provisioned`
reclaim their node; a token join of a known machine needs a token with `allowReclaim`, or the old node deleted first.
The cases are on [joining again with --reset](/docs/daemon/install#joining-again-with---reset) and [recovering a node
whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed). Under `[helper]`, restart
`bcd-helper.service` and `bcd.service` afterwards; the join also rewrites root's join record, so the helper binds host
units to the new node.

## Node keys in the TPM [#node-keys-in-the-tpm]

bcd keeps the node key in the host's TPM when there is one (965c14d, 50e18f9, 0ba2a79). Whatever holds it, the key is
a Go `crypto.Signer`, and nothing on the wire changes: the same P1363 signatures and the same `kid`.

| Property         | Value                                                                                                                                                                                                      |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Template         | `tpm.NodeKeyTemplate`: an unrestricted P-256 signing key, `fixedTPM`, `fixedParent`, `sensitiveDataOrigin`, `userWithAuth`, `noDA`, `sign`, a primary of the owner hierarchy derived from a 32-byte unique |
| Storage          | Nothing is persisted in the TPM. `node.key` keeps the unique, and the key is derived again at every load                                                                                                   |
| Signatures       | ECDSA with the digest's hash (SHA-256, -384 or -512)                                                                                                                                                       |
| Device           | `/dev/tpmrm0`, the kernel's resource manager, opened once per process so the keys' handles stay valid together. A fakehw build uses its software TPM only with `BCD_FAKEHW_TPM_KEYS=1`                     |
| When             | Every new key: at `bcd join` and at every renewal. When the TPM cannot hold a key, bcd logs a warning and makes a file key instead, except at the renewal of a certificate that names an AK, which fails   |
| What it protects | The key cannot be copied off the machine, but root on the node can derive and use it: non-exportable, not root-proof                                                                                       |

`internal/tpm` is bcd's own TPM 2.0 client (f61b32d): it builds the TPM's command format directly over the transport,
with no library, and sends only `CreatePrimary`, `ReadPublic`, `NV_ReadPublic`, `NV_Read`, `StartAuthSession`,
`PolicySecret`, `ActivateCredential`, `Quote`, `PCR_Read`, `Sign`, `Certify` and `FlushContext`. The same package
builds the `tpm` attestor's evidence (EK, AK, the EK certificate from NV index `0x01C00002`) and its round-2 answer;
see [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm).

### Through bcd helper [#through-bcd-helper]

The kernel's resource manager filters no command, so whoever can open `/dev/tpmrm0` could change the hierarchies'
authorization, clear the TPM or trip its lockout. Since 8870b8d (security review 3) an unprivileged `bcd node` never
opens it:

| Process                            | Opens the TPM                                                           |
| ---------------------------------- | ----------------------------------------------------------------------- |
| `bcd join` (root)                  | Itself, `[helper]` or not                                               |
| `bcd node` as root (no `[helper]`) | Itself                                                                  |
| `bcd node` under `[helper]`        | Never: `agent.TPM` is `helper.TPMKeys`, the helper's `tpm.*` operations |
| `bcd helper` (root)                | Itself, for the node                                                    |

The node sends `tpm.nodekey`, `tpm.sign`, `tpm.ak`, `tpm.certify` and `tpm.release`, each naming a key by its unique,
with a 30-second timeout. It checks every answer: the derived key must have the node-key attributes, every signature
must verify against the key's public half (`the helper's signature is not the node key's` otherwise), and a
certification must come from the AK the helper announced (`the helper certified with another AK`). What the helper
checks is on [bcd helper](/docs/daemon/helper#tpm). The install script no longer puts `bcd` in the `tss` group, and
`bcd doctor`'s `tpm` check fails while it is still a member.

### Seeing where the key is [#seeing-where-the-key-is]

| Tool                    | Shows                                                                                                                                                                                                                    |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `bcd status`            | `node key in the TPM` or `node key a file` on its identity line; `nodeKey` (`tpm` or `file`) in `--json`                                                                                                                 |
| `bcd doctor` `node key` | `ok` in the TPM; `ok` for a file key on a host without a TPM; `warn` for a file key on a host with one ("the next renewal moves it into the TPM")                                                                        |
| The control plane       | `nodeKey: 'tpm'` in `node.joined` and the audit row only for a certified `tpm` join, and the `urn:bc:tpm-ak:` statement in its certificates; every other node is `unattested`, and the control plane has only bcd's word |

## What the tests cover [#what-the-tests-cover]

| Test                                                             | Covers                                                                                                                                                                         |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `internal/agent` join, state and renewal tests                   | Deferrals, challenges, verification refusals, the next-key promotion and discard, `TestRenewCertifiedByTheAK` against `tpmsim` (no TPM or another AK fails before any request) |
| `TestJoinReset`, `TestJoinHandsTheIdentityToTheNode` (`cmd/bcd`) | The dry run, the move (journal included), no identity loading afterwards, and the hand-over to the helper's user (as root)                                                     |
| `internal/tpm` `TestTSContract`                                  | bcd's evidence and answer against the TypeScript `tpmAttestor` through fixtures, both directions                                                                               |
| `internal/helper` `TestHelperTPMKeys`                            | The `tpm.*` operations against `tpmsim`                                                                                                                                        |
| `tests/e2e/bcd.test.ts`                                          | Real joins, sessions and renewal of fakehw bcd binaries against the real gateway                                                                                               |
| `tests/e2e/bcd-tpm.test.ts`                                      | A tokenless `bcd join --attestor tpm` with an EK certificate from a test manufacturer root, a rejoin as the same node, and a renewal certified by the AK, all on `tpmsim`      |


# Overview (/docs/daemon)

> bcd is one Go binary for servers, site agents and the PXE ramdisk. It dials out to a gateway, runs typed commands and converges desired state.



bcd is the Better Compute daemon. One Go binary runs on every server you manage, on the site
agents that boot bare metal, and inside the PXE ramdisk. It connects out to a regional
gateway, reports what the machine is, runs the typed commands
the control plane sends, and converges the machine toward its NodeSpec.

The daemon holds no policy. It never decides who gets a GPU or when a machine is wiped; the TypeScript control plane
does. What bcd decides on its own is about safety: refusing a command meant for another node or past its deadline,
never running a side effect twice, and never dropping a device assignment because it lost its connection.

> **Built and tested, not released.** 
  The v0.1 node agent is implemented: `bcd join`, `bcd node`, `bcd status`, `bcd doctor` and `bcd version` work, and
  `tests/e2e/bcd.test.ts` runs the Linux binary (fake-hardware build) against a real control plane and gateway over
  mTLS. `bcd site` and `bcd ramdisk` now run too, with their command kinds, the site's boot server (proxy DHCP, TFTP,
  iPXE scripts, the image cache) and the ramdisk's discovery join, but only the Go unit tests exercise them: no test
  runs them against the control plane or on a real machine. A node agent also runs VPS instances, instance tunnels
  and the metadata service, and `tests/e2e/bcd-vps.test.ts` creates, starts and deletes an instance on it through the
  control plane on a fakehw build's file-backed runtime. A host build runs instances on the `libvirt-kvm` runtime when
  `virsh` is installed (362317c) and on the `cloud-hypervisor` runtime when `cloud-hypervisor` is (2598f0a); both so
  far have been tested only against fake tools (a fake `virsh`, a fake cloud-hypervisor API), never on a real
  hypervisor. bcd is not released: the release
  pipeline exists but no tag has been cut, so the install script has nothing to download yet. `bcd sim` runs many
  real node agents on fake hardware in one process (fakehw builds only; see [bcd sim](#bcd-sim)). Self-update with A/B
  slots (de9f380; the end-to-end test updates a real bcd to a second signed build) and the WebSocket fallback (6cd48be;
  tested end to end against the real gateway, renewal included, 4b2d5aa) are written, and the whole bcd end-to-end
  suite runs under signed intent. `bcd helper`, the root half of [privilege separation](#privilege-separation), is
  written (28dc2ae) and tested end to end with an unprivileged `bcd node`, self-update included (2d67b25), and
  `daemon install-script --unprivileged` sets it up (169c459). A site agent drives IPMI BMCs through the `bcx-ipmi`
  plugin (51806ab), tested against a fake `ipmitool` only. GPU details come from `nvidia-smi` rather than the NVML
  library (cb987e5), and NVML events are missing (see [what bcd does today](#what-bcd-does-today)). The
  [host converger](/docs/daemon/host-config) (6c2343f), MIG and slot holds with their cgroup slices (46a61db,
  260e33c) and Kubernetes workers (88ecb07, 7579184) are written and tested against fakes only, and bcd can run as a
  [DaemonSet](/docs/daemon/kubernetes) on an existing cluster (8df8c70): CI builds its image and has a kind cluster's
  API server dry-run its manifests (ac1ca4b), but no test has deployed it. The TypeScript `FakeDaemon` covers those in tests, the simulator and
  `better-compute dev --sim N`.

## Where bcd sits [#where-bcd-sits]

<Mermaid
  chart="flowchart LR
  subgraph machines[&#x22;Your machines&#x22;]
    N[&#x22;bcd node&#x22;]
    S[&#x22;bcd site&#x22;]
    R[&#x22;bcd ramdisk&#x22;]
  end
  LB[&#x22;L4 load balancer (TLS passthrough)&#x22;]
  G[&#x22;gateway&#x22;]
  DB[(&#x22;PostgreSQL&#x22;)]
  A[&#x22;api and workers&#x22;]
  N -->|&#x22;mTLS, port 443&#x22;| LB
  S --> LB
  R --> LB
  LB --> G
  G --> DB
  A --> DB
  A -.->|&#x22;pokes, tunnels&#x22;| G"
/>

Each daemon holds one long-lived gRPC session to one gateway. The gateway keeps sessions,
heartbeats and lane counters in memory and writes what matters to PostgreSQL, which stays the only source of truth.
Losing a gateway loses open streams and nothing else: the daemon redials, the next gateway replays what was in
flight, and the daemon's journal makes sure nothing runs twice. See
[sessions](/docs/daemon/sessions) and [delivery](/docs/daemon/delivery).

## Modes [#modes]

The same binary runs in several modes. The mode decides which command kinds the daemon accepts and which role its
certificate carries.

| Mode      | Runs on                                         | Does (PLAN.md)                                                                                                                                                                                                                    | Certificate            | Planned | In the code                                                                                                                                                                                                                                                                                       |
| --------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node`    | every server                                    | the session, the journal, inventory, health (NVML events, kernel XIDs, EDAC, SMART, from v0.2), NodeSpec convergence (CDI holds, MIG layout, host config, runtimes), commands, tunnels (v0.3), OTLP telemetry, self-update (v0.4) | `node`, 24 h           | v0.1    | works for the v0.1 scope; see below                                                                                                                                                                                                                                                               |
| `site`    | one or two per site, as a node with role `site` | proxy DHCP, TFTP and iPXE over HTTPS, and discovery vouching while it holds the site boot lease; the image cache and BMC access at all times                                                                                      | `site`, 24 h           | v0.2    | works: BMC operations over Redfish, or through a hosted bcx driver such as `bcx-ipmi` (51806ab), `site.httpProxy`, boot entries, and the boot server (proxy DHCP, TFTP, iPXE scripts with signed discovery assertions under the boot lease, the image cache)                                      |
| `ramdisk` | the PXE initramfs                               | inspection, DCGM diagnostics, wipes (`quick`, `secure-erase`, `crypto-erase`), image streaming, `boot.kexec`                                                                                                                      | `ramdisk`, at most 2 h | v0.2    | works: joins with the `discovery` attestor from its kernel command line, then runs `disk.wipe`, `deploy.image`, `boot.kexec`, `diag.dcgm` and the shared kinds. No ramdisk image is built in the repository                                                                                       |
| `sim`     | test hosts                                      | N daemons on fake hardware with real TLS and gRPC                                                                                                                                                                                 | a test CA              | v0.4    | works in fakehw builds: N real node agents in one process against the configured gateway, each with its own identity (a `node` certificate from a normal join); see [bcd sim](#bcd-sim)                                                                                                           |
| `helper`  | beside `node`, as root                          | typed privileged operations over a local socket                                                                                                                                                                                   | none                   | v0.4    | works on Linux (28dc2ae): `bcd helper --config` serves an unprivileged `bcd node` on `[helper] socket`, self-update's privileged half included (2d67b25); `daemon install-script --unprivileged` installs it as `bcd-helper.service` (169c459); see [privilege separation](#privilege-separation) |

A site agent and the ramdisk speak the same session protocol
as a node and run the same agent (`runAgent`: session, journal, dispatcher, convergers, status socket); only their
command kinds and what runs beside the agent differ. [Sites](/docs/guides/bare-metal/sites) and
[deploying](/docs/guides/bare-metal/deploying) cover what they do for bare metal.

The three agent modes take these flags:

| Mode          | Flags                                                                    | Identity                                                                                                                                                                    |
| ------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bcd node`    | `--config` (default `/etc/bcd/config.toml`)                              | joined by `bcd join`, kept in `data_dir`                                                                                                                                    |
| `bcd site`    | `--config`                                                               | joined by `bcd join --role site` with a site token; refuses a data directory that holds another role's identity. With a `[boot]` section it also runs the boot server       |
| `bcd ramdisk` | `--cmdline` (default `/proc/cmdline`), `--data-dir` (default `/run/bcd`) | no config file: the `bc.*` boot parameters give the gateway, pins and credentials, and the key, identity and journal live in tmpfs, so every ramdisk boot is a new identity |

Beside the modes, bcd has five local commands, all of which work:

| Command       | Does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bcd join`    | Joins the fleet with a join token read from `BCD_JOIN_TOKEN`, standard input or `--token-file` (deleted once the join succeeds), never from arguments. `--role` is `node` (the default) or `site`; the token decides what is allowed. It seals the token to the key in `join-encryption.jwk` next to the config, or, when that file is absent and `--server https://…` is given, to `join.encryptionJwk` from the control plane's `/.well-known/better-compute`; either key is refused unless it matches `join_encryption_key_pin`. `--node-id` joins an installed OS with the `provisioned` attestor and fails unless the joined node is that one; `--attestor` names another. It verifies the signed `JoinResponse`, writes `node.key` (a PKCS#8 key, or the record of a key it made in the host's TPM; see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)) and `identity.json` under `data_dir`, and prints `joined as <node> (site <site>, role <role>)`. With `--server` and no `config.toml` at all, it reads the gateway (`gateway.url`, `serverName`, which the server publishes when its options set `gateway.url`) from that well-known document, checks the document's pins against `--serving-ca-pin`, `--join-signing-pin` and `--join-encryption-pin` (all three required on a host build; a fakehw build may take them from the document, which is trust on first use), joins, then writes `config.toml` (0600, naming the data directory from `--data-dir`, default `/var/lib/bcd`) and `join-encryption.jwk` (0644); see [joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document). With a `config.toml`, pin flags given must match it. A node that already joined exits 0 without a token, so a join unit may run on every boot; a different `--node-id` is refused. `--reset --yes` (a1c1a46, as root) first moves an identity that still loads but no longer works aside, so the machine joins again; see [joining again with --reset](/docs/daemon/install#joining-again-with---reset) |
| `bcd status`  | Asks the running agent (node, site or ramdisk) over its root-only unix socket (`--socket`, default `/run/bcd/bcd.sock`; `--json` for JSON) for its node, site and role, boot, session epoch and clock offset, applied and observed generations with each section's state, running and deferred commands, the event backlog, certificate expiry, what holds the node key (`in the TPM` or `a file`; `nodeKey` in JSON, 50e18f9), trust bundle version and whether signed intent is enforced. Since ad8791f also the machine: one line per GPU read now, a `kubernetes` line and a `telemetry` line (see [below](#bcd-status))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `bcd doctor`  | Checks the configuration file, NVML, the hardware backend, `/dev/kvm`, IOMMU units, a TCP connection to the gateway and the node's identity                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `bcd update`  | Shows the self-update slots, the release floor, a pending update and the versions that failed to start (`--config`); `--clear-failed` lets a failed version be tried again. See [self-update](/docs/daemon/install#self-update)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `bcd version` | Prints the version, platform, Go version and build tags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

Exit codes are 0 for success, 1 for a failure and 2 for a usage error. bcd
never echoes a positional argument, so a join token pasted into the command line does not end up in logs:

```bash
$ bcd join bcjt_jt_01...
bcd join: takes no positional arguments
bcd join reads the join token from BCD_JOIN_TOKEN or stdin, never from arguments
```

`bcd doctor` prints one line per check, with a status of `ok`, `warn` or `fail`. The `nvml` check asks `nvidia-smi`
for the driver version and GPU count (cb987e5); a host without `nvidia-smi` warns
(`nvml: nvidia-smi not found (no NVIDIA driver on this host)`), and a build with `-tags nonvml` warns that NVML is
not available in this build:

```text
ok    config    /etc/bcd/config.toml: gateway https://gw.eu-1.example.com, site fsn1
ok    nvml      driver …, 8 GPUs
ok    hardware  host: 8 GPUs, … inventory sections hashed
ok    kvm       /dev/kvm present
ok    iommu     16 IOMMU units
ok    gateway   gw.eu-1.example.com:443 reachable in 4ms
ok    identity  node node_01k5w… (site fsn1, role node), certificate valid for 14h12m0s
```

Since f9d7af9 doctor also prints one `gpu` line per GPU from `nvidia-smi`: its UUID and name, how busy it is, its
temperature, power draw of its limit and memory used of its total. A pending row remap or a MIG mode change waiting
for a reset is a warning; failed row remapping or uncorrectable ECC errors since the driver loaded is a failure. A
host without the NVIDIA driver prints no `gpu` line. With a configuration file it adds the Kubernetes checks:
`kubernetes` fails when `[kubernetes] allowed_clusters` does not parse, reports the number of allowed clusters, and
fails a node bcd joined to a cluster without any, since every `k8s-join` then refuses (9f860ad); on a node bcd joined to a cluster,
`kubelet` fails when the kubelet is not running, and `device-plugin` warns when `/var/lib/kubelet/device-plugins` is
missing. `gpu-isolation` fails whenever bcd would report `ForeignGpuExposure`: on a joined worker for a toolkit
configuration that lets unprivileged pods pick GPUs and for another GPU DRA driver in the node's ResourceSlices, and,
since f88bcdb, on a node with `[host] units = false` (a DaemonSet) for the toolkit and for a container runtime
configuration it cannot read (see [GPU isolation](/docs/daemon/kubernetes#gpu-isolation)). As root with `[helper]`,
`helper` says what the helper binds host units to: root's signed join record (`ok`), trust on first use for a node
joined before that record (`warn`), or a record that names another node or fails its signature (`fail`; f77c6f9), and
fails when neither `command_root_sha256` nor the record names command-signing roots, so host units would go unsigned
(71ccf71; see [signed specs](/docs/daemon/host-config#signed-specs)).

```text
ok    gpu       GPU-5f1c… NVIDIA H100 80GB HBM3: 87% busy, 61 C, 540/700 W, 71.2/79.6 GiB
warn  gpu       GPU-9a02… NVIDIA H100 80GB HBM3: 0% busy, 34 C, 72/700 W, 0.0/79.6 GiB; a row remap waits for a GPU reset
ok    kubernetes 1 allowed clusters
```

A missing `/dev/kvm` or IOMMU is a warning (the host cannot run VPS instances, or pass GPUs through to them), and so
is a node that has not joined. An identity that is there but does not load (a node key that no longer matches its
certificate, an unreadable `identity.json`) is a failure since a1c1a46 (it was a warning); its detail, like that of
an expired certificate, ends with the way back: `join again as root: bcd join --reset --yes, then bcd join`. A host with neither `virsh` nor `cloud-hypervisor` gets a warning that it runs no VPS
instances. With either, doctor reports the runtime it found and checks a VPS host's tools: it fails one missing
`qemu-img` (no instance disks), both `genisoimage` and `xorriso` (no NoCloud user data seed) or `nft` (no
anti-spoofing or network policies, so the metadata service and instance dials stay off) (4a0165f). For
`cloud-hypervisor` it also warns when `/usr/share/cloud-hypervisor/CLOUDHV.fd` is missing (its instances cannot
boot) and fails without `systemd-run` (its VMMs could not outlive bcd) (2598f0a). Since 2845f41 it also warns when
`[update] release_pins` is missing (self-update is refused) and otherwise reports the pin count and threshold, fails
when the slots are in use but the entrypoint does not link to the slots directory's `current`, warns about a pending
update and each failed version (pointing at `bcd update --clear-failed`), and warns when a site agent has no
`[site] bmc_networks`. It finds the slots and records where `bcd update` does (8eca2b1): `[update] slots_dir`, else
`/usr/local/lib/bcd` under the helper, else `<data_dir>/bin`. Run as root on a helper node, it fails the `helper`
check for each directory on the way to the slots, the helper socket or the entrypoint that is not root's alone, the
condition that keeps `bcd helper` from starting (9e71b1f). A
gateway that does not accept a TCP connection within 5
seconds, or an expired node certificate, is a failure. It also warns when others can read `config.toml` or its group can write it (group read is the `--unprivileged`
install's `root:bcd` 0640), and exits 1 when any
check fails. Since e2a0cf0 it also checks that the host clock is synchronized (`clock`), and `bcd hardware`
reports what the host really is; see [running bcd on real hardware](/docs/daemon/real-hardware).

Since ab26e87 doctor also checks what enforcing booked host resources needs from the host:

| Check              | Fails or warns when                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cgroups`          | Fails without cgroup v2 (`/sys/fs/cgroup/cgroup.controllers`) or without its `cpuset`, `cpu` and `memory` controllers, since holds with limits cannot be confined; warns without `hugetlb`, since a hold's huge page limits are not enforced                                                                                                                                                                                                                                                                                                                                      |
| `hugepages`        | Reports the free and total 2 MiB and 1 GiB pages, or that no pool exists and huge-page-backed flavors and limits are refused here                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `vfio`             | On a host with an active IOMMU, warns when `vfio-pci` is not loaded: GPU passthrough cannot rebind devices                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `systemd`          | With `cloud-hypervisor` installed, fails below systemd 247 (its confined units need it) and warns when the version cannot be read                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `libvirt`          | Under `[helper]` with `virsh` installed (ed3c7d2), fails when the helper's user is in the `libvirt` group, which is root-equivalent (`gpasswd -d bcd libvirt`), when `/etc/libvirt/qemu.conf` lacks `dynamic_ownership = 0`, and when QEMU's user (`libvirt-qemu`, else `qemu`) is not in the node account's primary group (`usermod -aG`); warns when neither QEMU user exists. See [libvirt through the helper](#libvirt-through-the-helper)                                                                                                                                    |
| `cloud-hypervisor` | Under `[helper]` with `cloud-hypervisor` installed (7fe74a1), warns when `[helper] instance_bridges` is empty, since the helper then refuses every instance tap; otherwise `ok`: its taps, VFIO and VMM units go through `bcd helper`                                                                                                                                                                                                                                                                                                                                             |
| `tpm`              | Under `[helper]` (both `socket` and `user` set; e6c3f53), fails when the helper's user is in the `tss` group, which lets it send the TPM any command although `bcd helper` holds the node keys: "`<user>` is in the tss group: it can send the TPM any command (lock or clear its hierarchies); bcd helper holds the node keys, so remove it (gpasswd -d `<user>` tss) and restart bcd.service". Warns when the membership cannot be read; prints nothing on a host without a `tss` group or a user outside it. See [node keys through the helper](#node-keys-through-the-helper) |
| `node key`         | Once the node has joined (50e18f9): `ok` when the key is in the TPM, `ok` for a file key on a host without a TPM, and warns for a file key on a host with one: "a file key although this host has a TPM: the next renewal moves it into the TPM (through bcd helper when bcd node runs unprivileged)" (a1c1a46)                                                                                                                                                                                                                                                                   |

## bcd status [#bcd-status]

`bcd status` prints the agent's state first, then, since ad8791f, the machine's:

```text
gpus       2
  0 GPU-5f1c…   87%   61 C   540/700 W  71.2/79.6 GiB  held by alloc_01k…
  1 GPU-9a02…    0%   34 C    72/700 W  0.0/79.6 GiB  UNHEALTHY
kubernetes hosted node gpu-node-7, offering 1 bcd.dev/gpu
telemetry  https://otel.example.internal:4318, last export 2026-09-26T08:12:00Z
```

* **gpus.** One line per GPU, read from `nvidia-smi` at that moment: index, UUID, how busy, temperature, power draw of
  its limit, memory used of its total, the allocation holding it, and `UNHEALTHY` when it may take no work (a failed
  or pending row remap, uncorrectable ECC errors, an XID). `--json` also carries the model and the processes.
* **kubernetes.** On a Kubernetes worker: `joined` or `hosted`, the Node and, when joined, its API server, and how
  many devices each `bcd.dev` resource offers the kubelet (`no devices` when none).
* **telemetry.** With a `telemetry` section: the OTLP endpoint, the last export and the last error.

`--json` has `gpus`, `kubernetes` and `telemetry` objects for the same.

## bcd sim [#bcd-sim]

`bcd sim` (aa7501c) runs many real node agents in one process against the gateway of a configuration file, so the
load a gateway sees is the real daemon's, not a model of it. It exists only in fakehw builds (`go build -tags
fakehw`); a host build exits 2 with `bcd sim: needs a fakehw build`.

| Flag       | Default                | Meaning                                                                                                                                        |
| ---------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `--config` | `/etc/bcd/config.toml` | The gateway URL, `gateway_server_name`, pins, journal limits and logging, as for `bcd node`. The simulated nodes live under its `data_dir/sim` |
| `--nodes`  | `1`                    | How many daemons to run; below 1 is a usage error (exit 2)                                                                                     |
| `--seed`   | `1`                    | The fakehw seed of the first daemon; daemon `i` uses `seed + i`, so each is a distinct machine with its own hardware key                       |

Each simulated machine `i` gets its own data directory `<data_dir>/sim/<i>` holding its `node.key`, `identity.json`,
bbolt journal, fakehw boot id and the file-backed `libvirt-kvm` instance runtime, and runs the same agent as
`bcd node` over its own session. A machine that has not joined yet joins as role `node` (claims hostname
`sim-<i>`) with the one token `bcd sim` reads at start from `BCD_JOIN_TOKEN` or standard input; the token must
allow as many uses as there are unjoined machines, and at most 16 joins are in flight at once. The token is only
read when some machine still needs it, and `join-encryption.jwk` must sit next to the configuration file (there is
no `--server` fallback). Machines that joined before simply reconnect on the next run.

A `node.reboot` writes a new boot id and restarts only that machine's agent, not the process; a machine whose agent
fails is restarted after a back-off of one second per consecutive failure, capped at a minute. The process runs
until SIGINT or SIGTERM, then prints how many machines joined and how many reboots it saw:

```bash
printf '%s\n' "$SIM_TOKEN" | bcd sim --config ./sim.toml --nodes 5 --seed 100
# bcd sim: 5 nodes (fakehw seeds 100-104) against <gateway_url>
# ... one "simulated node joined" log line per machine ...
# bcd sim: stopped; 5 joined, 0 reboots
```

`tests/e2e/bcd.test.ts` runs five of them with one five-use join token and checks that five distinct machines
(five distinct hardware keys) connect and that the token appears nowhere in the output. `better-compute sim` is a
different tool: the deterministic, single-process fleet simulator built on the TypeScript
[FakeDaemon](#the-fakedaemon), with chaos and invariant checks and no real gateway. `bcd sim` is the one to point at
a real gateway and PostgreSQL. PLAN.md's nightly run of `bcd sim --nodes 10000` through Toxiproxy (v0.4) is not
wired up.

## What bcd does today [#what-bcd-does-today]

`bcd node` is the v0.1 node agent (`daemon/internal/agent`); `bcd site` and `bcd ramdisk` run the same agent with
their own kinds. What it implements and what is still missing, as of commit 2598f0a:

| Area                        | Implemented                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Missing                                                                                                                                                                                                                                                                                                                                                                                                                    |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identity                    | Join with a sealed `join-token` as role `node` or `site`; a `provisioned` join for an OS a deploy installed (`--token-file`, `--node-id`, `--server` for the well-known join key); with no `config.toml`, the configuration bootstrapped from the well-known document and checked against the install's three pins (69aaba9); the ramdisk's `discovery` join (the token and the site agent's signed assertion from the kernel command line); a retried deferral with the same CSR, `JoinResponse` verification (pinned signer, CSR binding, leaf key, SPIFFE id, role); a `tpm` join with no token, `bcd join --attestor tpm` over `/dev/tpmrm0` (8545688; `internal/tpm`, up to 3 challenge rounds, checked against the TypeScript verifier by a fixture contract in both directions and end to end with the software TPM `tpmsim`, db9bd46; see [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm)); renewal at 60% of the lifetime with proof of possession and a new key; the node key in the host's TPM when there is one (an unprivileged `bcd node` reaching it only through `bcd helper`'s `tpm.*` operations, 8870b8d), else a 0600 PKCS#8 file, a file-key node moving into the TPM at its next renewal (965c14d, 50e18f9, 0ba2a79; see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | A proof of where the key lives for any join other than a certified `tpm` join (those certificates name the AK, and every renewal sends the AK's certification of the next key, cf24876 and d94f1f3), a join or TPM node key on a hardware TPM (every test uses `tpmsim`), the `.well-known` fallback for a stale key file (bcd reads the well-known key only when `join-encryption.jwk` is absent and `--server` is given) |
| Transport                   | gRPC over HTTP/2 with connect-go (`Join`, `Renew`, `Session`, `Tunnel`), TLS 1.3 pinned to the serving CA's SPKI (never system roots), mTLS with the node certificate, HTTP/2 pings every 20 s; `gateway_server_name` for a gateway reached by IP; with `[websocket] url`, the WebSocket fallback for proxies that strip client certificates: the session over `wss://.../v1/session` with a node proof and a checked gateway proof, renewals over `POST /v1/renew`, no tunnels, command-signing roots required (6cd48be, tested end to end in 4b2d5aa; see [the fallback](/docs/daemon/protocol#websocket-fallback-v04))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Session                     | Hello from the journal (with the trust bundle version it holds), Welcome (epoch, event ack, clock offset, requested sections, signed trust bundle), heartbeats with lane credits (inventory hashes every 10th and right after changed sections, 427787a), `Ack`, `Cancel`, `Drain`, `RevocationNotice`, `OpenTunnel`, secret and workload-token answers, full-jitter redial, or exactly the gateway's `grpc-retry-pushback-ms` when a busy refusal carries one (joins too; 2237be7); a Welcome more than 5 minutes before the server time mark is refused, and one within that allowance never moves the working clock back; a newer trust bundle is acknowledged once, in the next `StatusReport` (`trust_version` is 0 in every other report)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Journal                     | bbolt, every write fsynced; the command rules, results with their `boot_id`, the `cancelled` outcome, the TTL sweep, spent exec-ticket ids (kept until 5 minutes after the ticket expires); a corrupt file is moved aside                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Commands                    | `inventory.refresh`, `diag.nvml`, `logs.collect` (redacted bundle to a presigned https URL), `gpu.reset` (refuses a GPU with a live hold, reads health back), `node.reboot` (self-terminating); on a site agent `bmc.*`, `site.httpProxy`, `boot.prepareRamdisk` and `boot.cleanUp`; in the ramdisk `disk.wipe`, `deploy.image`, `boot.kexec` and `diag.dcgm` (also on nodes); on a VPS host `instance.snapshot` (disk, and optionally memory, uploaded to a presigned https URL with its SHA-256); `driver.invoke` (node, site and ramdisk) into a hosted bcx driver plugin; the acceptance order; signed commands verified once the trust bundle holds command-signing roots; with approver keys in it, the v1.0 destructive-batch quorum (`BATCH_UNAPPROVED`, each batch leaf run at most once, recorded in `data_dir/batch-leaves.json`, and `UNBATCHED_LIMIT` counted per rolling hour of the host clock in `data_dir`; a quorum change in a newer trust bundle only at a higher approver-set epoch with the held approvers' signatures, a refused set keeping the held quorum while roots and version apply, a keyless node taking only its pinned `approver_set_sha256` set and refusing destructive commands until it has it, the held set reported in `Hello` and `StatusReport` (e171f0d), and a `site.httpProxy` write only under a payload-bound leaf; e7ddb38, 8b0f50c, see [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)); attested results node-signed; `daemon.upgrade` (node and site, self-terminating; see [self-update](/docs/daemon/install#self-update))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Every other registered kind (answered `UNSUPPORTED`)                                                                                                                                                                                                                                                                                                                                                                       |
| Driver plugins              | The bcx host (`internal/bcx`): the plugins of `[plugins] dir` named in its `pins` file, each run only if its binary's SHA-256 is its pin, over the go-plugin style handshake and gRPC with pinned mTLS on a unix socket; their drivers are advertised in `Hello.drivers`. `bcx.Serve` is the plugin side for Go plugins. On Linux each plugin runs from the file that was hashed (`/proc/self/fd/3`), under its own uid from a block bcd keeps (`0x7FFE0000` to `0x7FFEFFFF`, assigned in name order) when bcd is root (or `[plugins] user`; root only for plugins named in `[plugins] privileged`), through bcd's `__bcx-exec` shim with `no_new_privs` and a seccomp filter on every thread (no namespaces, mounts or `io_uring`), in its own process group and, with `[plugins] cgroup`, its own cgroup with `memory_max` and `pids_max`, in a delegated cgroup bcd prepares itself (d132fef, 3872c6b, 6827df8, d902063); `daemon install-script --plugins` sets a node up for it. A plugin that exits is restarted (9006eee): back-off from 1 s to 1 min, re-hashed against its pin and sandboxed again, with operations meanwhile failing with a retryable `unavailable`. See [drivers](/docs/extending/drivers#the-plugin-sandbox)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | A plugin that fails its first start is only logged and skipped, never retried; plugins beside an unprivileged `bcd node`                                                                                                                                                                                                                                                                                                   |
| Desired state               | Full specs and patches, signature checks with command-signing roots, stored and enforced offline; `host` (the twelve config units `kernel-modules`, `sysctl`, `hugepages`, `cpu`, `iommu`, `nvidia-driver`, `fabric-manager`, `gpu-settings`, `mig-layout`, `gpu-sharing`, `container-toolkit` and `k8s-join`, each re-planned in order against the host, a failing one never holding back the rest (c21ef84), through `bcd helper`'s `host.unit` when unprivileged (only from signed specs, 03c023d, 2d18327), with the conditions `FabricManagerDown`, `MPSDown` and `RebootRequired`; drift repaired by a pass every 10 minutes (3c0fe02); 6c2343f, 7579184; `[host] units = false` turns it off, a193249, and under the helper root's own copy of the setting refuses every `host.unit`, 3fa9021; see [host configuration](/docs/daemon/host-config)); `devices` (CDI holds and env files for whole GPUs, MIG instances and slots with parent-aware conflicts and no hold beside a GPU a VM has, 46a61db; `BINDING_KUBERNETES` holds for the kubelet device plugin, d02ba4f; a hold with limits in its own cgroup slice, 260e33c, except a Kubernetes hold, e01d7ff; host admission of holds, `resources-exhausted`, 31555bf; node-signed `AllocationBound` and `AllocationReleased`, `AllocationFailed`); the kubelet device plugins or the DRA driver `gpu.bcd.dev` and the release reaper on a node bcd joined to a cluster or runs on as a DaemonSet (88ecb07, 7579184, 1e5f96b, 7d24836; see [Kubernetes workers](/docs/guides/capacity/kubernetes)); `access` (break-glass SSH: the sshd drop-in, CA, node-bound principals and a KRL; the exec-ticket JWKS; a CA key or principal that could inject an sshd line makes the whole spec invalid, signed or not); `instances` (below); `telemetry` (bcd's own metrics, `bcd.gpu.*`, `bcd.host.*` and the instances' `bcd.instance*`, over OTLP/HTTP to the section's collector, 27c7ee2, 3111284, 0b21b4b; see [node metrics](/docs/operations/observability#node-metrics-from-bcd)); `site` (the boot lease state, and `cache_images` pulled into the image cache: `PROGRESSING` while they warm); `daemon` (self-update to a signed release under `[update] release_pins`: A/B slots in `data_dir/bin`, `upgrade.json` with a 120 s deadline, confirmation on the first `Welcome`, a swap back without one, a crash-loop rollback unit under systemd; de9f380; once confirmed, no older binary is taken, the other slot's included, c139b9c)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | A non-empty `daemon` section is `SectionUnsupported` in `bcd sim` and `bcd ramdisk`; no host unit or Kubernetes path has run on a real machine (all tested against fakes, apart from `TestRealKernel`'s sysctl, THP and 2 MiB pages); see [the roadmap](/docs/reference/roadmap#cpu-memory-and-kubernetes-v11)                                                                                                             |
| Instances                   | The `instances` converger over a `Runtime` interface (`List`, `Ensure`, `SetState`, `Destroy`), on node agents: the section is the whole desired set; a domain bcd did not create is only reported; user data is fetched with a `SecretRequest` bound to the allocation; `RUNNING`, `PAUSED` and `STOPPED`; `restore_from` boots a new instance from a snapshot, downloaded over https only to `data_dir/snapshots/<sha256>` and verified first, in the background while the converger goes on (another runtime, or a memory snapshot from another CPU model, fails it), and deleted once no spec restores from it; transfers end when a 2-minute window moves less than 1 MiB; `instance.snapshot` stages its upload under `data_dir/snapshots` too, bounded by the command's deadline; a failing instance fails alone (`InstanceFailed`); `StatusReport.instances` lists every domain with `instances_complete`, and `Hello.drivers` lists each runtime (kind `runtime`). A host build with `virsh` on its `PATH` runs `libvirt-kvm` (362317c; see [delivery](/docs/daemon/delivery#instances)): `virsh` on `qemu:///system` (under `bcd helper`, through its checked `libvirt.*` operations; ed3c7d2) and `qemu-img`, qcow2 overlays on sha256-verified https base images, a NoCloud seed ISO, taps named before the domain starts, VFIO GPU passthrough, host-passthrough CPU, and every domain it defines marked in its metadata. A host build with `cloud-hypervisor` on its `PATH` runs that runtime too (2598f0a): one VMM per instance in a confined `bcd-ch-<id>.service` transient unit in `bcd-instances.slice` (since b12c217 as the instance's own uid with no capabilities and no network, its taps handed over as file descriptors; only KVM and VFIO devices, only its own instance directory and base image visible, `/run` and `/var` hidden, memory, tasks and CPU bounded by the flavor) driven over its REST API socket, the same verified qcow2 overlays and NoCloud seed, persistent taps bridged before the VMM starts, VFIO GPUs, memory snapshots and restores (only into the same vCPUs and memory), a serial console socket, and `instances/<id>/vm.json` as bcd's record of what it owns. Both runtimes read qcow2 headers in Go, with no tool probing a tenant image, and refuse a base image or restored snapshot disk that names a backing file or an external data file, is encrypted or is not a qcow2, and an instance disk that names anything but its recorded base; `raw` and `iso` images are never probed (security review 12, 933c51b, and its verification, b12c217; see [tenant images](/docs/daemon/delivery#tenant-images)). A memory restore's guest gets its own SSH identity on a hot-added disk (ca08d5c; see [identity rebind](/docs/daemon/delivery#identity-rebind)). Both runtimes' tests run against fakes (a fake `virsh`, a fake cloud-hypervisor API, fake `qemu-img`), and neither has run on real hardware. fakehw builds run a file-backed runtime under the same name; `tests/e2e/bcd-vps.test.ts` runs that one against the control plane. A bcx plugin whose driver lists the kind `runtime` is also a runtime on `bcd node` (`list`, `apply`, `remove` over `Invoke`; no snapshots; see [walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx)). The instance `network_policy` as host nftables rules on the taps, in the `bcd_instances` bridge table, replaced atomically before any domain starts together with the `inet bcd_taps` backstop that drops traffic from or to a tap off its bridge (933c51b): `PRIVATE` keeps a NIC to its IPAM subnets and the host, `NONE` to the host only, and a policy that cannot be expressed or rules that fail to load fail the instance (6db49c8); a host with runtimes advertises the capability `instance.network-policy` in `Hello`, since an older bcd drops the field. The same table guards every tap with a MAC and IPAM addresses, whatever its policy: only the NIC's own MAC, IP addresses, EUI-64 link-local address and ARP (278296e, 6827df8), each guarded address bound to its tap against every other port, and no foreign neighbour adverts (6639335); every instance tap, guarded or not, drops DHCP server replies, router advertisements and redirects and may not use or claim the host's own bridge addresses (6827df8); the metadata service and instance tunnel dials trust only addresses the loaded table binds, never one two instances claim, a check every 5 seconds counts both tables' rules and, if the table or a chain was flushed, ends that trust at once and loads the ruleset again, and a host whose nft cannot load it keeps egress instances running with both off. The metadata service on `[imds] listen`: `PUT /latest/api/token`, then `GET /latest/identity-token?audience=`, for the running instance that holds the request's source address | A run of `libvirt-kvm` or `cloud-hypervisor` on a real KVM host; `iso` images on `cloud-hypervisor` (refused). A host build with neither tool and without a runtime plugin fails a non-empty `instances` section (`this host runs no instance runtime`)                                                                                                                                                                    |
| Tunnels                     | `OpenTunnel` with the exec ticket re-verified against the pinned JWKS and the journaled `jti` set; for node tickets `console` on a real pseudo-terminal, `ssh` and `tcp` to loopback ports; for instance tickets `ssh` (port 22) and `tcp` to the instance's own address as its runtime reports it, never a loopback, link-local or host address, and `console` on the serial console of a running, managed instance of this host, from its runtime (`libvirt-kvm` opens the pty its live domain names for the serial port, `cloud-hypervisor` dials the VM's serial socket; the fakehw stand-in echoes; f8273ef), never the host's shell, which `tunnels.open` issues since 798e710 (`better-compute console --instance`); `tunnel.instance-dial` advertised when the host runs a runtime; at most 16 at once                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `exec` into a guest (it needs a guest agent: the tunnel closes with exit status 127)                                                                                                                                                                                                                                                                                                                                       |
| Secrets and workload tokens | `SecretRequest` and `WorkloadTokenRequest`, node-signed, answered with a JWE to a fresh one-time P-256 key, matched by request id within 10 s. Callers: BMC credentials and `site.httpProxy` secret headers (bound to the workflow step), the boot server's discovery tokens (bound to the workflow), `deploy.image` user data (bound to the step), instance user data (bound to the allocation), the `k8s-join` unit's bootstrap token and the OTLP exporter's headers (bound to the node's own spec, workflow `host:<unit>`), the metadata service's workload tokens                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Health                      | NVIDIA XIDs from `/dev/kmsg` as `Xid<n>` device conditions for the rest of the boot, kept in `data_dir/faults.json` with the boot id so a bcd restart in the same boot keeps them, or until a `gpu.reset` leaves the GPU healthy (662fb55); PSI pressure and CPU steal every 30 s in `StatusReport.pressure`; `diag.nvml` checks from `nvidia-smi` data: `nvml.init` needs the driver's management interface, and per GPU `ecc.uncorrectable`, `row-remap.pending`, `row-remap.failure`, `pcie.link` (a link narrower than its maximum width) and `xid.recent` (this boot's XIDs), each failure with its reason; the inventory collected again every 10 minutes and about 2 seconds after an XID burst (662fb55)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | NVML events, EDAC, SMART                                                                                                                                                                                                                                                                                                                                                                                                   |
| Inventory                   | `inventory.Collector` from procfs and sysfs without cgo: DMI, UEFI and Secure Boot, TPM presence, CPU and NUMA, memory, disks, NICs with SR-IOV and RDMA, NVIDIA GPUs from `/proc/driver/nvidia`, OS, cgroups, KVM, IOMMU; on a site agent an `images` section listing its image cache (digest, source URL, size, `READY` or `PULLING`), re-sent when a pull starts or ends; on a host build the GPUs completed by `nvidia-smi` and merged by UUID (cb987e5): MIG mode and instances, NVLink, topology, fabric clique, ECC counters and row remapping, power limit, persistence and compute mode, with live samples (memory, utilization, temperature, power, clocks, processes) kept out of the inventory hash                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | A go-nvml backend; netlink                                                                                                                                                                                                                                                                                                                                                                                                 |
| Site agent                  | Redfish BMC operations (power, boot device, CD virtual media, inspection, firmware inventory, credential rotation, BIOS settings, RAID layouts, and firmware updates served to the BMC from the verified image cache; 944221c), each only under its own `bmc.*` kind, pinned to the enrolled SPKI, on the system a BMC address names (`/redfish/v1/Systems/<id>`) or else the first, with every resource path kept under `/redfish/v1` on that BMC; a `bmc.*` operation whose driver is not `redfish` goes to the hosted bcx driver with that id that implements the operation's concern (`power.setPower`: concern `power`), with the address and TLS pin as its config and the credentials, a rotation's new password included, as secrets, never arguments; the kind's operation allowlist, the quorum and the BMC memory apply as for Redfish, and a driver the agent does not host answers `unsupported-driver` (51806ab). `bcx-ipmi` is that driver for IPMI (`ipmi`: `power.getPower`, `power.setPower`, `management.setBootDevice` and, since 7818f19, `inspect.inspect` from the FRU, BMC and sensor data, without the host's MACs but with the system GUID as its UUID (7cd1dd2), through `ipmitool -I lanplus` with cipher suite 17, or 3, 8 or 12 from the BMC's enrollment driver options (`driver_options_json`, handed to the plugin as `config.options`, 7753df0), and the password over a pipe; a BMC with a TLS key pin is refused, since IPMI has no TLS (933c51b); see [drivers](/docs/extending/drivers#bcx-ipmi-the-ipmi-driver)); `site.httpProxy` to site-internal addresses only; boot entries persisted in `data_dir/boot-entries.json`; the boot server: proxy DHCP (UDP 67 and 4011) and read-only TFTP for the iPXE builds, `GET /boot/ipxe` scripts with a signed discovery assertion each, all answering only prepared machines while the agent holds the boot lease and hears from the gateway, and listing the unknown ones in `StatusReport.discovered` (0056e1e; well-formed fields only, 0442561), which metal stores and proposes as the MAC of a machine enrolled by product UUID, for an operator to confirm (3d26428, 8de6e2f; see [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines)); request headers capped at 8 KiB and TFTP at 64 transfers, 4 per source (0442561, 4d8b493); `sanboot` of a cached ISO for a boot entry with `sanboot_sha256`, which metal sets for a catalog ISO on a driver without virtual media (0056e1e, 3d26428); the image cache (`data_dir/cache/<sha256>`, served only under grants at `GET /cache/t/<grant>/<sha256>`, b7ed8a2, trimmed least recently used first to `[boot] cache_max_bytes` without evicting an image `cache_images` still names)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Vendor BMC drivers; IPMI beyond power, boot device and inspection (virtual media, firmware, BIOS, RAID and rotation are Redfish only); a run of `bcx-ipmi` against a real BMC (its tests use a fake `ipmitool`); matching `site.httpProxy` targets to the node's own addresses; the iPXE builds and the ramdisk itself, which you supply                                                                                   |
| Ramdisk                     | Boot parameters from `/proc/cmdline`, the `discovery` join as role `ramdisk`, state in tmpfs; quick wipes read back, secure and crypto erase with `nvme format`, `hdparm` or `blkdiscard --secure`; image streaming (from the site cache by digest, else the source) through SHA-256 and gzip or zstd; cloud-init or Ignition user data; kexec into the installed kernel; `dcgmi diag`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | A ramdisk image (kernel and initramfs with bcd and its tools) built in the repository; qcow2 images; a run on a real machine                                                                                                                                                                                                                                                                                               |
| Privilege separation        | `bcd helper` (28dc2ae, `internal/helper`): with `[helper] socket` and `user` set, `bcd node` runs as that account and sends its privileged work to the helper, which runs as root: the files bcd owns (CDI specs, allocation env files, the sshd drop-in, CA, KRL and principals), `firewall.apply` and `firewall.check` on the `bcd_instances` and `bcd_taps` tables, `gpu.reset`, `reboot` and `ssh.reload`, and self-update's `update.*` operations with the slots in root's `[update] slots_dir` (2d67b25), the `host.unit` operation that plans and applies one config unit as root (6c2343f), and `slice.apply` and `slice.remove` for GPU holds' cgroup slices (260e33c), and the `libvirt.*` operations that define and run `libvirt-kvm` domains after a default-deny check of each definition, and hand the node its console as a descriptor (c305a05, af09411, 8c0cf9e, fea4d2e, used since ed3c7d2), and the `ch.*` operations for `cloud-hypervisor` (its taps, VFIO groups and VMM units, built by root from the instance record it checks; 7fe74a1), and the `tpm.*` operations that derive, sign with and certify the node's keys in root's TPM, which the node never opens (8870b8d). `daemon install-script --unprivileged` sets it up (169c459; without the `libvirt` group since d418820, without the `tss` group since 8870b8d). Peers are checked with `SO_PEERCRED` and `/proc/<pid>/exe`; requests are versioned JSON, strictly decoded, and each is logged (a successful `tpm.sign` excepted). `tests/e2e/bcd.test.ts` runs an unprivileged node beside a real helper that writes and removes a GPU hold's CDI spec, and a signed update reaching an unprivileged node through the helper                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | PLAN.md's `runtime.*` operations (a MIG layout goes through `host.unit` rather than `mig.apply`); bcx plugins beside an unprivileged node (`--unprivileged` refuses `--plugins`); only `bcd node` uses the helper (`bcd site` ignores `[helper]`, and the ramdisk has no configuration file)                                                                                                                               |
| Local                       | `bcd status` over the unix socket, with the GPUs, the Kubernetes worker and the OTLP export (ad8791f); `bcd doctor`'s host checks (KVM, IOMMU, gateway reachability, identity, cgroup v2 controllers, huge page pools, `vfio-pci`, systemd for cloud-hypervisor, the libvirt and cloud-hypervisor setup under the helper, what holds the node key); `bcd sim`, many fakehw node agents in one process (aa7501c)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `doctor` and `support-bundle` against the running agent; a clock check                                                                                                                                                                                                                                                                                                                                                     |

## Outbound only [#outbound-only]

Nodes only dial out. A daemon connects to `gw.<region>:443` through an L4 load balancer that passes TLS through to
the gateway, so a server needs no inbound port, no public address and no firewall exception. Everything the control
plane wants from a node travels down the session the node opened: commands, desired state, requests to open a
[tunnel](/docs/guides/access/tunnels), revocation notices.

`bcd ramdisk` opens no TCP listener, and neither does `bcd node` unless it serves the metadata service. The only
local endpoint is a root-only unix socket (mode 0600 in a 0700 directory) that serves `bcd status`; PLAN.md also plans
`doctor` and `support-bundle` against the running agent on it. If the socket cannot be created, the agent logs it and
runs without. Under [privilege separation](#privilege-separation) there is a second unix socket, `bcd helper`'s at
`[helper] socket`, which only the node's account and root may use. A VPS host with `[imds] listen` set serves its instances' metadata service there (for example
`169.254.169.254:80` on the instances' bridge), answering only requests whose source address belongs to one of its
running instances. The other exception is a site agent with a `[boot]` section: its boot server listens on plain HTTP at `[boot] listen`, and optionally for proxy DHCP (UDP 67
and 4011) and TFTP, for the PXE machines of its own provisioning network. It answers a booting machine only while it
holds the site's boot lease (see
[sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site)).

## Privilege separation [#privilege-separation]

By default `bcd node` runs as root under systemd hardening: `ProtectSystem=strict`, `NoNewPrivileges`, a narrow
set of writable paths and a bounded capability set. [Installing bcd](/docs/daemon/install#the-systemd-unit) shows
the unit.

PLAN.md §6 (v0.4) splits the daemon in two, and since 28dc2ae bcd can run that way. With `[helper] socket` and
`[helper] user` in `config.toml`, `bcd node` runs as that unprivileged account, and `bcd helper --config
/etc/bcd/config.toml` runs as root beside it and performs a few typed operations on the node's behalf.
`daemon install-script --unprivileged` (169c459) sets this up: the `bcd` account, `bcd-helper.service`, and a
`bcd.service` drop-in that runs the node as `bcd` (see
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)). The session,
the journal, and every command, spec and frame that comes from the network are then parsed by the unprivileged
process; the helper parses only its own small requests, and trusts nothing in them.

<Mermaid
  chart="flowchart LR
  G[&#x22;gateway&#x22;] -->|&#x22;session, mTLS&#x22;| N[&#x22;bcd node (helper.user)&#x22;]
  N -->|&#x22;unix socket, JSON v1&#x22;| H[&#x22;bcd helper (root)&#x22;]
  H --> F[&#x22;/etc/cdi, /run/bcd/allocations, /etc/ssh&#x22;]
  H --> T[&#x22;nft: bridge bcd_instances, inet and arp bcd_taps&#x22;]
  H --> R[&#x22;gpu.reset, reboot, sshd reload&#x22;]
  H --> U[&#x22;self-update: slots_dir, swap, confirm&#x22;]
  H --> L[&#x22;libvirt: checked define, start, stop, save, console&#x22;]
  H --> C[&#x22;cloud-hypervisor: taps, VFIO, VMM units&#x22;]
  H --> K[&#x22;TPM: node keys, sign, certify&#x22;]"
/>

### What the helper does [#what-the-helper-does]

| Operation                                                                                                       | Sent by the node for                                                                                             | Checked by the helper                                                                                                                                                                                                                                                                                                                                                                                                 |
| --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `file.write`, `file.remove`                                                                                     | The `devices` converger's CDI specs and env files, the `access` converger's sshd files                           | A file kind and an id, never a path (below); at most 1 MiB                                                                                                                                                                                                                                                                                                                                                            |
| `firewall.apply`                                                                                                | The instances' nftables ruleset                                                                                  | Only the shape bcd renders: at the top level, `table …`, `delete table …`, `flush table …` and `table … {` blocks of exactly three tables, `bridge bcd_instances`, `inet bcd_taps` (933c51b) and `arp bcd_taps` (b12c217), the same name in another family refused; inside a block no nested `table`; no `include` or `define` line and no line naming `ruleset`; balanced braces                                     |
| `firewall.check`                                                                                                | The 5-second check that the tables are still loaded                                                              | A rule count from 0 to 1,000,000, compared with the rules of the three tables (an absent `inet` or `arp bcd_taps` counts as none)                                                                                                                                                                                                                                                                                     |
| `gpu.reset`                                                                                                     | The `gpu.reset` command                                                                                          | A `GPU-…` UUID of a GPU in the host's own inventory                                                                                                                                                                                                                                                                                                                                                                   |
| `reboot`                                                                                                        | `node.reboot`                                                                                                    | The reason is cut to 200 bytes                                                                                                                                                                                                                                                                                                                                                                                        |
| `ssh.reload`                                                                                                    | The `access` converger, when the sshd drop-in changed                                                            | none                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `host.unit`                                                                                                     | The host converger, for each config unit that is not converged (6c2343f)                                         | A unit name of `[a-z0-9-]`; at most 1024 busy GPUs; the signed spec that asks for exactly these settings, under roots pinned in `command_root_sha256` or from the trust bundle the pinned JoinService key signed (03c023d, 2d18327); the helper plans the unit again as root with its own units, which check the settings, and gives it 30 minutes. See [under bcd helper](/docs/daemon/host-config#under-bcd-helper) |
| `slice.apply`, `slice.remove`                                                                                   | The `devices` converger, for a hold with limits (260e33c)                                                        | Only a `bcd-alloc-<id>.slice` name and cpulists reach `systemctl` and cgroupfs; `slice.remove` stops a slice only once it is empty                                                                                                                                                                                                                                                                                    |
| `update.status`, `update.prepare`, `update.swap`, `update.confirm`, `update.rollback`, `update.settle`          | [Self-update](/docs/daemon/install#self-update): the `daemon` section and `daemon.upgrade` (2d67b25)             | Everything that matters, with the helper's own configuration: the manifest against its release pins and threshold, the release floor, the expiry against its own clock, and the staged binary's size and SHA-256 (below)                                                                                                                                                                                              |
| `libvirt.define`                                                                                                | The `libvirt-kvm` runtime, defining an instance's domain (af09411)                                               | The domain XML against the default-deny policy, and never over a domain bcd did not define (below)                                                                                                                                                                                                                                                                                                                    |
| `libvirt.start`, `libvirt.shutdown`, `libvirt.destroy`, `libvirt.suspend`, `libvirt.resume`, `libvirt.undefine` | The runtime's state changes                                                                                      | An instance id, and a domain whose bcd metadata names it (for every `libvirt.*` operation but `define`); `start` may ask for `--paused` (d418820), `undefine` adds `--nvram --managed-save`                                                                                                                                                                                                                           |
| `libvirt.blockresize`                                                                                           | Growing a disk                                                                                                   | A `vdX` target and a size in bytes                                                                                                                                                                                                                                                                                                                                                                                    |
| `libvirt.setmem`                                                                                                | The [balloon controller](/docs/daemon/delivery#memory-ballooning) (12e9ced)                                      | A size of at least 1 MiB, applied live; libvirt caps it at the domain's memory                                                                                                                                                                                                                                                                                                                                        |
| `libvirt.attach-disk`, `libvirt.detach-disk`                                                                    | The [identity rebind](/docs/daemon/delivery#identity-rebind)'s disk                                              | Only `vdz`, live; the attached file passes the same file checks as a domain's disks                                                                                                                                                                                                                                                                                                                                   |
| `libvirt.save`, `libvirt.restore`                                                                               | Memory snapshots and restores                                                                                    | `save` writes into root's staging directory first, then copies into the node's instance directory through an `os.Root` and hands the file to the node's uid; `restore` checks the state file, and a `--xml` definition against the same policy and for the same domain                                                                                                                                                |
| `libvirt.console`                                                                                               | [Console tunnels](/docs/guides/access/tunnels#the-node-half) to an instance (8c0cf9e)                            | A bcd domain's pty console, opened as root (`O_NOCTTY`, a Unix98 pty slave) and sent back as a file descriptor (`SCM_RIGHTS`); the node gets that descriptor, never the path's permissions                                                                                                                                                                                                                            |
| `ch.taps`, `ch.untaps`, `ch.vfio`, `ch.unvfio`, `ch.launch`, `ch.stop`                                          | The `cloud-hypervisor` runtime's taps, GPUs and VMM units (7fe74a1)                                              | An instance id, and what root reads and checks itself from that instance's record; see [cloud-hypervisor through the helper](#cloud-hypervisor-through-the-helper)                                                                                                                                                                                                                                                    |
| `tpm.nodekey`, `tpm.sign`, `tpm.ak`, `tpm.certify`, `tpm.release`                                               | The node key in the TPM: loading it, every signature it makes, and a renewal's certification by the AK (8870b8d) | A 32-byte unique naming the key, a SHA-256, -384 or -512 digest, 32 bytes of qualifying data; root builds every TPM command itself; see [node keys through the helper](#node-keys-through-the-helper)                                                                                                                                                                                                                 |

The helper builds each file's path from its own table:

| Kind             | Path                                            | Id                                                                                                               |
| ---------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `cdi`            | `/etc/cdi/bcd-<id>.json`                        | `[A-Za-z0-9_-]`, 1 to 128 characters                                                                             |
| `allocation-env` | `/run/bcd/allocations/<id>.env`                 | the same                                                                                                         |
| `ssh-dropin`     | `/etc/ssh/sshd_config.d/10-better-compute.conf` | none                                                                                                             |
| `ssh-ca`         | `/etc/ssh/bc_ca.pub`                            | none                                                                                                             |
| `ssh-krl`        | `/etc/ssh/bc_krl`                               | none                                                                                                             |
| `ssh-principals` | `/etc/ssh/bc_principals/<id>`                   | `[A-Za-z0-9._@-]`, 1 to 64 characters                                                                            |
| `k8s-token`      | `/var/lib/bcd-hostconfig/k8s/bootstrap-token`   | none; the kubeadm bootstrap token staged for `host.unit` `k8s-join` (7579184; in root's directory since 1bb78d6) |

Every file is written mode 0644, except `k8s-token`, which is 0600. The helper walks each directory down from `/` with `os.Root`, checking every
component with `Lstat` and refusing a symlink or anything that is not a directory (a missing one is created 0755).
Running as root on the real filesystem, it also refuses a directory on the way that is not root's or that group or
others can write (02485b3).
It writes a temporary file created exclusively in that directory, then renames it over the target, so a symlink
planted at the target is replaced, never followed. The node side (`helper.Client`) maps a path to its kind and id and
refuses any path outside the table before it sends anything.

### libvirt through the helper [#libvirt-through-the-helper]

The `libvirt` group gives root: whoever can write to `qemu:///system` can define a storage pool over `/etc` or a
domain with a host disk. So since ed3c7d2 an unprivileged `bcd node` is not in that group. Its `libvirt-kvm` runtime
(`runtime.Libvirt` with a `Hypervisor`) sends every privileged `virsh` verb to the helper's `libvirt.*` operations
above, reads through libvirt's read-only socket (`virsh -r`) for `list`, `domstate`, `dumpxml` and `domstats` only,
and refuses any other verb ("is not an operation bcd helper does"). A paused start or restore passes the helper's
`Paused` flag (`--paused`; f3c6a65). `libvirt.restore` needs a definition, checked like a `define` and passed as
`--xml`: a restore without one is refused before anything runs, since `virsh restore` alone would start the domain
the node-written state file embeds (b6df6f1). The helper runs `virsh -c qemu:///system` as root, stages each definition in its own directory,
`/var/lib/bcd-hostconfig/libvirt`, and acts only on domains whose bcd metadata names them, never on one bcd did not
define.

`libvirt.define` checks the domain XML default-deny (c305a05, fea4d2e): only what the runtime renders is admitted, and
anything else is refused, such as `qemu:commandline`, filesystems, block, USB or direct devices, graphics, a custom
emulator or kernel, a `seclabel`, a DOCTYPE or entities, and foreign namespaces or attributes.

| Part of the domain | Admitted                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Domain             | `type="kvm"`, `os` type `hvm`, at most 1 MiB and 4096 elements; the name is an instance id, and bcd's metadata (its own namespace only) names that instance exactly once                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Disks              | File disks and cdroms, driver `qemu`, `qcow2` or `raw`, under `data_dir`'s `libvirt/instances`, `libvirt/images` or `volumes` only, as a clean absolute path. As root the helper checks that no component from that directory down is a symlink and that the file is a regular file of the node's uid with one link, then re-runs the image checks (414bfef): an instance's qcow2 disk must be an overlay of `<images>/<image>.<format>` as bcd's metadata names it (flat when restored), and a cdrom must pass the base image check. Since 3fcd1a1 the overlay's base is checked as root too, since QEMU opens it: under bcd's directories, the node's own regular file with one link and no symlink on the way, a plain image naming no backing or data file, and named by its SHA-256 (64 lowercase hex digits) |
| Interfaces         | `type="bridge"` on a bridge listed in `[helper] instance_bridges`, with the tap exactly `runtime.TapName(domain, i)`; with no bridges configured, no domain with a NIC is defined                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Host devices       | Managed PCI hostdevs of this host's GPUs, from its inventory at each request                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Others             | `memballoon` `none` or `virtio`; `pty` serial and console; vCPU, `cputune` pinning, `numatune` and huge page backing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

libvirt then runs with `dynamic_ownership = 0`, so it never chowns a path the node names: QEMU reaches the node's files
through the node's group instead. The runtime makes instance disks 0660 (read-only ones 0640) and gives directories on
the way group search rather than others' search. The file checks are the second line, since the node owns those
directories and could race them. `daemon install-script --unprivileged` sets the host up (d418820; see
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)), and `bcd doctor`'s `libvirt`
check fails a host that is not.

`internal/helper` tests the validator (`domain_test.go`), the operations against a fake `virsh`
(`libvirt_linux_test.go`, a foreign domain refused, a paused start) and the file check on a real file system; the
runtime's side is `internal/runtime` `hypervisor_test.go`. No test has run it against a real libvirtd.

### cloud-hypervisor through the helper [#cloud-hypervisor-through-the-helper]

Until 7fe74a1 an unprivileged node dropped `cloud-hypervisor` with a warning and did not advertise it, since its VMMs
need root for their `systemd-run` units, taps and VFIO devices. Since then (runtime side d86fcbe, helper side
7fe74a1 and 94843ec) it runs under `[helper]` like `libvirt-kvm`, and sandboxes, which default to it, can use such a host. The
node keeps owning every file and shares the instance's with the VMM's own uid through POSIX ACL entries (`aclGrant`,
55e0069; the unit's `UMask` becomes `0007`), and root does only the steps that need root, each from what it checks
itself:

| Operation              | What root does and checks                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ch.taps`, `ch.untaps` | Creates exactly `runtime.TapName(instance, i)` (at most 16), owned by the node, on a bridge of `[helper] instance_bridges`, and deletes only those names; an existing interface of that name that is not a tap is refused                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `ch.vfio`, `ch.unvfio` | Binds the IOMMU groups of the GPUs the instance record names to `vfio-pci` and gives `/dev/vfio/<group>` (0600) to the VMM's uid: only this host's GPUs (from its inventory at each request), none that root sees holding work, none whose `/dev/vfio` group is owned by another running VMM unit's user (94843ec: a passed-through GPU is on `vfio-pci`, where neither `nvidia-smi` nor a CDI hold sees it), and only a group whose other devices are functions of this host's GPUs or PCI bridges. `ch.unvfio` gives groups back to root and unbinds them, skipping any group a running VMM unit's user owns. Who holds a group is root's own observation of `/dev/vfio`, never a node-written record |
| `ch.launch`            | Reads and checks the instance's `vm.json` as root (`runtime.CHRecord`: the node's regular files, no symlink in the instance directory, the base image and overlay re-checked, volumes in the volume store, the VMM uid in bcd's range and no other running unit's, vCPU and memory bounds), then runs `systemd-run` with the confined unit it builds itself (`runtime.CHLaunch`) and root's own `cloud-hypervisor` binary, one launch at a time                                                                                                                                                                                                                                                         |
| `ch.stop`              | Reads the VMM's user from the unit before stopping `bcd-ch-<id>.service`, answers its `ActiveState`, and once it is `inactive` or `failed` gives back to root exactly the `/dev/vfio` groups that user owns                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

The node still talks to each VMM over its API socket, which reaches only what the VMM's own uid can open inside its
unit. Each call names the instance by the domain-name pattern and is logged with it (`instance=` in the helper's
log). Without a `cloud-hypervisor` binary on root's `PATH` the helper refuses every `ch.*`. `bcd doctor` warns when
`[helper] instance_bridges` is empty. See [delivery](/docs/daemon/delivery#the-runtimes) for the runtime side.
`internal/helper`'s `ch_test.go` (taps on allowed bridges only, a launch never taking another unit's uid, VFIO for
whole groups of this host's GPUs only) and `internal/runtime`'s `ch_privileged_linux_test.go` run against fakes; no
real VMM has run under the helper.

### Node keys through the helper [#node-keys-through-the-helper]

bcd keeps the node key in the host's TPM (see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)).
Until 8870b8d an unprivileged `bcd node` opened `/dev/tpmrm0` itself, through the `tss` group the install script put
it in. Security review 3 found that too much: the kernel's resource manager filters no command, so a compromised
`bcd` user could change the hierarchies' authorization, clear the TPM through the (default empty) lockout
authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys enrolled with
`systemd-cryptenroll`) and attestation. Now the unprivileged node never opens the TPM:

* **Who opens it.** Processes that run as root open it themselves: `bcd join`, `bcd helper`, and `bcd node` without
  `[helper]`. When `bcd node` runs unprivileged with `[helper] socket` set on a host with a TPM, every node-key
  operation goes to the helper instead (`helper.TPMKeys`).
* **What the node can ask.** Only the five operations below, each naming a node key by the 32-byte unique that
  `node.key` records. The helper derives the key from `tpm.NodeKeyTemplate` itself, with the empty password the
  template sets, and uses its own AK, so the node chooses no template, handle, hierarchy or authorization.
* **What the node checks.** The key the helper derives must have the node-key attributes, every signature must
  verify against the key's public half, and a certification must come from the AK the helper announced.

| Operation     | Does                                                                                                                                                               | Refuses                                                       |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- |
| `tpm.nodekey` | Derives the node key of a unique and answers its public area (`TPMT_PUBLIC`)                                                                                       | A unique that is not 32 bytes                                 |
| `tpm.sign`    | Signs a digest with that key (ASN.1 ECDSA)                                                                                                                         | A digest that is not 32, 48 or 64 bytes (SHA-256, -384, -512) |
| `tpm.ak`      | Answers the AK's public area                                                                                                                                       |                                                               |
| `tpm.certify` | Has the AK certify the key (`TPM2_Certify`) over the qualifying data, a renewal's `sha256(csr)`; answers the `TPMS_ATTEST`, its signature and the AK's public area | Qualifying data that is not 32 bytes                          |
| `tpm.release` | Unloads the key; it is derived again at its next use                                                                                                               | A unique that is not 32 bytes                                 |

At most 8 node keys stay loaded in the helper (TPM object slots are few); the least recently used one is unloaded to
make room, and an unloaded key is derived again whenever it is used, so a helper restart costs nothing. The node's
calls time out after 30 seconds. A helper on a host without a TPM refuses every `tpm.*` operation ("node keys in the
TPM are not served by this helper (the host has no TPM)"), and any other `tpm.` verb is an unknown operation. Every
TLS handshake and node statement signs, so a successful `tpm.sign` is not logged; its refusals and every other
`tpm.*` call are.

`daemon install-script --unprivileged` no longer adds `bcd` to `tss`, and an upgrade run takes it out (see
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)). Until that run, a host
installed between 0ba2a79 and 8870b8d keeps `bcd` in `tss`, and since e6c3f53 `bcd doctor` fails its `tpm` check
there (`gpasswd -d bcd tss`, then restart `bcd.service`). `internal/helper`'s
`TestHelperTPMKeys` checks the operations against `tpmsim`; no hardware TPM has run them.

### Self-update through the helper [#self-update-through-the-helper]

Self-update's privileged half is `selfupdate.Manager` (2d67b25): `Prepare` (verify only), `Swap`, `Confirm`,
`RollBack`, `Settle`, `Status`, `Expire` and `ClearFailed`. A node without a helper runs it in process; under the
helper the helper runs it, and the node only downloads and restarts:

1. **The slots are root's.** The binary slots and the upgrade and release records (`upgrade.json`, `update.json`)
   live in `[update] slots_dir`, by default `/usr/local/lib/bcd` under the helper (`<data_dir>/bin` otherwise, with
   the records in `data_dir`). The helper creates it, each missing directory on the way 0755 whatever its umask,
   never re-moding one that exists. A `slots_dir` inside `data_dir` (or containing it), or an `[update] entrypoint`
   inside `data_dir`, is refused under the helper, since the node owns that directory and could plant the binary root
   trusts (9e71b1f). Running as root, the helper adopts the entrypoint's binary into a slot only when it is a
   root-owned regular file that group and others cannot write.
2. **The node downloads.** It asks `update.prepare` which artifact to fetch and writes it to
   `<data_dir>/staging/bcd-<version>`.
3. **The helper verifies and swaps.** `update.swap` takes no path from the node. The helper verifies the signed
   manifest again under its own `[update]` pins, threshold and release floor, and the expiry against its own clock
   (the node's server time can only make that check stricter). It opens the staged file beneath the staging
   directory's parent without following a symlink or blocking on a FIFO, takes only a regular file with one link
   owned by the node's uid (c139b9c), and hashes it as it copies it into the other slot, so what is checked is what
   runs, then records `upgrade.json` and swaps.
4. **The node confirms.** The node restarts into the entrypoint, and on its first `Welcome` asks the helper for
   `update.confirm`, which moves the release floor up. The helper confirms only while `current` points at the
   pending slot.
5. **The helper's own watchdog.** An update still unconfirmed 30 seconds after its deadline is swapped back by the
   helper, which then restarts `bcd.service`, whatever the node does. The watchdog is re-armed when the helper starts.

A terminal refusal (a manifest that does not verify, a version below the floor) comes back flagged as such, so the
node treats it as final rather than retrying.

The helper trusts nothing the node claims (c139b9c, security review 11). The running version is the current slot's
record in `slots.json` (before the first update, the helper's own version), not the version the node reports.
`update.settle` decides by where `current` points, so a node that falsely claims to run the old version cannot get
the new one recorded as failed and the watchdog disarmed. `slots.json` records each slot's version, manifest
sequence and written SHA-256, and the binary in the other slot is reused without a manifest only when its recorded
sequence is known and not below the floor. Once a release is confirmed, nothing takes the node below it, so a
compromised node cannot swap back to an older binary that root would then run. `tests/e2e/bcd.test.ts` delivers a build signed under the pinned key to
an unprivileged node through a real helper, with the slots outside the data directory, and sees it confirmed.

### Who it serves [#who-it-serves]

* **The peer.** For every connection the helper reads the caller's pid, uid and gid with `SO_PEERCRED`. The uid and
  gid must both be `[helper] user`'s (its primary group), or the uid must be 0. `/proc/<pid>/exe` must be the same
  file as an installed bcd: `[update] entrypoint`, `bcd-a` or `bcd-b` in `[update] slots_dir`, or the helper's own
  binary. A refused caller gets only "not allowed"; the helper logs its pid, uid and the reason.
* **The protocol.** One request per connection: JSON `{"v":1,"op":…}` of at most 4 MiB, answered `{"v":1,"ok":…}`
  with an `error` when it failed. Both sides refuse unknown fields and another version, so a node and helper of
  different versions refuse each other rather than guess. Each side gives a request 2 minutes, and the helper serves
  at most 32 requests at once (c139b9c); a connection over that cap gets a retryable `busy` answer (9e71b1f).
* **The socket.** Mode 0660, owned by root and the node account's primary group. Its directory (created 0755 if
  missing) must be root's and closed to group and other writes, or the helper refuses to start. Run as root, it
  also requires every existing directory from `/` down to the slots, the socket and the entrypoint to be root's
  alone (c139b9c, 9e71b1f), and checks that before it touches its update records; `bcd doctor` as root reports a
  directory that is not as a failed `helper` check. The helper binds,
  chmods and chowns the socket inside a private 0700 directory and only then renames it into place, so no path the
  node could swap for a symlink is ever changed by root (2d67b25, after a security review found the earlier
  chmod-after-bind could be redirected). The install script uses `/run/bcd-helper/helper.sock`, not `/run/bcd`, which
  `bcd` owns.
* **The log.** Every operation is logged with the peer's pid, the operation, the file kind and id or the GPU, and
  the outcome (`helper: operation … op=file.write kind=cdi name=<allocation>` in text logs).

`bcd helper` refuses to start without `[helper] socket`, when `[helper] user` names no account, or when it is not
root (a fakehw build excepted). The helper is Linux only, since it needs `SO_PEERCRED`. The configuration check
requires an absolute socket path and a user that is not `root`. See the keys in
[config.toml](/docs/daemon/install#configtoml).

Both units start the same entrypoint, which a swap points at the new slot. A self-update restarts only the node,
so the helper keeps running the binary it started with until it restarts; that works as long as both speak the same
helper protocol version (1 today), since a node and a helper of different versions refuse each other. A
[rerun of the install script](/docs/daemon/install#before-v04-rerun-the-script) restarts both units.

What the helper does not cover still runs with the node account's own rights: the journal, key and status socket
in `data_dir` and `socket_path`, the `libvirt-kvm` runtime's reads on libvirt's read-only socket and its files under
`data_dir/libvirt`, the `cloud-hypervisor` runtime's files and ACL grants and its VMMs' API sockets, and reading
`/dev/kmsg` for XIDs. From 7dde6a3 until 7fe74a1 `cloud-hypervisor` was not offered under the helper; now its taps,
VFIO and VMM units go through `ch.*` (see [above](#cloud-hypervisor-through-the-helper)). bcx plugins need root to
start (they get their own uids only when bcd is root), so `--unprivileged` refuses `--plugins`. PLAN.md's
`runtime.*` operations are not helper operations; the MIG layout, like every config unit, goes through `host.unit`. The end-to-end tests cover the CDI path (an
unprivileged fakehw `bcd node` beside a real `bcd helper` binds and releases a GPU hold, and the helper's log shows
the `file.write` and `file.remove` of its CDI spec) and self-update. Only `bcd node` uses the helper.

> **Hosts installed before d418820.** 
  Until ed3c7d2 and d418820 the install script put the `bcd` account in the `libvirt` group on VPS and sandbox hosts,
  which made `[helper]` separate nothing there. Such a host keeps that group membership until you run the install
  script again (see [upgrading an unprivileged libvirt host](/docs/daemon/install#upgrading-an-unprivileged-libvirt-host));
  `bcd doctor` fails its `libvirt` check until then.

Other limits hold with or without the helper:

* **Keys.** The node key is in the host's TPM when there is one (50e18f9, 0ba2a79; for an unprivileged node, held
  by the helper, 8870b8d): `/var/lib/bcd/node.key` then
  holds only the record the key is derived from again, and the key cannot be copied off the machine, though root on
  the node can derive and use it. Otherwise it is a 0600 PKCS#8 file, which the next renewal moves into the TPM once
  one is usable. A `bcd join --attestor tpm` also proves it to the control plane: the TPM's attestation key certifies
  the join's key (cf24876), and the certificate then names that AK, so every renewal has the AK certify the next key
  in the TPM or fails loudly (d94f1f3; see
  [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key)). bcd never exports it. It signs only statements of the form "this node says": the mTLS handshake,
  renewal proofs of possession, attested results, allocation reports, and secret and workload-token requests.
* **Allowlists.** Each role accepts only its own command kinds; anything else is rejected `UNSUPPORTED`.
* **Bounds.** A command body is at most 1 MiB and a desired-state body at most 4 MiB. The journal is capped at 64
  MiB and 10,000 unacked events.
* **Secrets.** Commands carry references, never values. bcd fetches a value with a node-signed `SecretRequest` and
  opens the answer, a JWE sealed to a fresh one-time key, only for that request id; the design keeps the value in
  memory only and zeroes it after use. The site agent asks for BMC credentials, `site.httpProxy` secret headers and
  each PXE machine's discovery token this way, and the ramdisk for `deploy.image` user data. `logs.collect` redacts
  tokens, JWTs, private keys and URL passwords from its bundle before the upload.

## Inside bcd [#inside-bcd]

The Go module (`github.com/better-compute/better-compute/daemon`, Go 1.26 or later per `go.mod`) is laid out by
concern. Its third-party modules are connect-go, protobuf, bbolt and klauspost/compress (zstd for image streaming);
go-nvml is not added: GPUs are read through `nvidia-smi` instead (cb987e5).

| Package               | What                                                                                                                                                                                                                                                                                             | State                                                                                                                                                 |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cmd/bcd`             | subcommands and flags                                                                                                                                                                                                                                                                            | every subcommand works; `sim` only in fakehw builds (aa7501c)                                                                                         |
| `cmd/bcx-ipmi`        | the IPMI BMC driver plugin a site agent hosts (`ipmitool`, lanplus), shipped as its own archive (348ff56)                                                                                                                                                                                        | done (51806ab; tested against a fake `ipmitool`)                                                                                                      |
| `internal/agent`      | the agent of every mode: join, session, dispatcher, the `devices`, `access`, `instances` and `site` convergers, the kinds, tunnels, secret and workload-token requests, the health watch, renewal, status; the site agent's kinds and boot server; the ramdisk's kinds and their host operations | done for v0.1; the v0.2 site and ramdisk parts in progress                                                                                            |
| `internal/bootparams` | the `bc.*` kernel command line between `bcd site` and `bcd ramdisk` (`Render`, `Parse`)                                                                                                                                                                                                          | done                                                                                                                                                  |
| `internal/redfish`    | the site agent's Redfish client                                                                                                                                                                                                                                                                  | done for the operations listed above                                                                                                                  |
| `internal/pxe`        | the boot server's first stage: a proxy DHCP server that never hands out addresses, and a read-only TFTP server                                                                                                                                                                                   | done                                                                                                                                                  |
| `internal/bcx`        | the out-of-process driver protocol: `Start` hosts a sha256-pinned plugin (handshake, pinned mTLS, `Describe`, `Invoke`, `Health`), `Serve` is the plugin side                                                                                                                                    | done                                                                                                                                                  |
| `internal/imds`       | the VPS host's metadata service: session tokens bound to the calling instance, workload tokens per audience, forwarded requests refused                                                                                                                                                          | done                                                                                                                                                  |
| `internal/runtime`    | the `Runtime` interface instance runtimes implement; `Libvirt`, the `libvirt-kvm` runtime over `virsh` and `qemu-img`; `CloudHypervisor`, one cloud-hypervisor VMM per instance over its REST API; and `Fake`, a file-backed runtime whose domains outlive the process and stop on a new boot id | done: `libvirt-kvm` (362317c) and `cloud-hypervisor` (2598f0a), both tested against fakes only                                                        |
| `internal/hostconfig` | the config units of `NodeSpec.host` over a `System` port (`OSSystem`, `FakeSystem`), the registry that orders and re-plans them                                                                                                                                                                  | done (6c2343f), tested against `FakeSystem`, and `TestRealKernel` on a real kernel for sysctl, THP and 2 MiB pages                                    |
| `internal/kube`       | the `k8s-join` unit, the kubelet device plugins (`deviceplugin/v1beta1` over connect's gRPC, hand-encoded messages checked against the upstream descriptors), the DRA driver `gpu.bcd.dev` (7d24836), hosted mode (1e5f96b) and the release reaper                                               | done (88ecb07, wired in 7579184), tested against a fake host and a fake kubelet                                                                       |
| `internal/slice`      | the cgroup-v2 slices of GPU holds with limits                                                                                                                                                                                                                                                    | done (1d03c28, used since 260e33c)                                                                                                                    |
| `internal/config`     | `/etc/bcd/config.toml` in a strict TOML subset                                                                                                                                                                                                                                                   | done                                                                                                                                                  |
| `internal/journal`    | the journal contract and the command rules (duplicates, crash recovery, self-terminating kinds), an in-memory store and the bbolt store                                                                                                                                                          | done                                                                                                                                                  |
| `internal/dispatch`   | lanes with credits and worker pools, per-resource locks, the kind registry, the acceptance checks                                                                                                                                                                                                | done                                                                                                                                                  |
| `internal/session`    | Hello and Welcome, the epoch, resume, reconnect back-off                                                                                                                                                                                                                                         | done                                                                                                                                                  |
| `internal/converge`   | the NodeSpec sections and the observed-generation rule                                                                                                                                                                                                                                           | done; section convergers live in `internal/agent`                                                                                                     |
| `internal/pki`        | purpose-framed ES256, keys and pins, the join JWE, CSRs, `JoinResponse` and signed-intent verification, trust bundles, exec tickets, secret and workload-token requests                                                                                                                          | done; interop fixtures from `@better-compute/pki`                                                                                                     |
| `internal/health`     | the XID parser for `/dev/kmsg`, PSI pressure and CPU steal                                                                                                                                                                                                                                       | done                                                                                                                                                  |
| `internal/transport`  | the `AgentGateway` client over connect-go, and `ws.go`, a dependency-free RFC 6455 client for the WebSocket fallback with renewal over `POST /v1/renew`                                                                                                                                          | done                                                                                                                                                  |
| `internal/selfupdate` | the A/B binary slots, `upgrade.json`, the release floor and the crash-loop rollback units                                                                                                                                                                                                        | done (de9f380)                                                                                                                                        |
| `internal/helper`     | privilege separation: `bcd helper`'s server (peer checks, the file table, the ruleset check, the `update.*` operations) and the node's clients                                                                                                                                                   | done (28dc2ae, 2d67b25)                                                                                                                               |
| `internal/inventory`  | NodeInventory v1, canonicalization, section encoding and hashes, the procfs/sysfs `Collector`                                                                                                                                                                                                    | done                                                                                                                                                  |
| `internal/support`    | the secret scanner of `@better-compute/pki` for log bundles                                                                                                                                                                                                                                      | done                                                                                                                                                  |
| `internal/fakehw`     | a deterministic 8x H100 SXM5 80GB HGX node                                                                                                                                                                                                                                                       | done                                                                                                                                                  |
| `internal/nvml`       | the NVML boundary: `nvml.SMI`, a cgo-free backend over `nvidia-smi` (`-q -x`, `-L`, `--query-gpu compute_cap`, `topo -m`, `nvlink -s`, `nvlink -R` for each link's peer (5026330), `mig -lgip`; each run bounded to 30 s), and `Probe` for doctor                                                | done without cgo (cb987e5), tested against captured `nvidia-smi` output; `-tags nonvml` builds a stub (`ErrNotBuilt`); a go-nvml backend is not added |
| `gen/`                | Go code generated from `proto/` (protoc-gen-go, protoc-gen-connect-go)                                                                                                                                                                                                                           | done; `node scripts/daemon-proto-gen.mjs [--check]`                                                                                                   |

Two numbers shape everything the daemon runs. Commands go to one of three lanes, and each lane has a fixed worker
pool, never a goroutine per command: `control` 4 workers, `provision` 2, `bulk` 1. The converger runs NodeSpec
sections in a fixed order (`host`, `devices`, `instances`, `access`, `telemetry`, `daemon`), each as an idempotent
plan and apply, and advances its observed generation only when every section has converged or failed terminally.

Contributors build and test the module from `daemon/`:

```bash
go test -tags nonvml,fakehw ./...   # logic tests: no cgo, no NVIDIA driver, fake HGX H100 hardware
go vet -tags nonvml,fakehw ./...
go build -ldflags "-X main.version=0.1.0" -o bin/bcd ./cmd/bcd
```

From the repository root, `node scripts/daemon-test.mjs` runs the same tests on any OS.

The `nonvml` tag builds without cgo and never loads `libnvidia-ml`; `fakehw` reports the fake HGX node instead of
reading the host, keeps its boot id in `data_dir`, "reboots" by re-executing itself, and is the only build that
accepts `dev_insecure_h2c`. bcd targets Linux on amd64 and arm64 (for Grace and GB200). On Windows,
`scripts/daemon-test.mjs` cross-compiles each package's test binary for linux/amd64 and runs it in WSL, since
Application Control may refuse locally built executables. See [testing](/docs/operations/testing#the-go-daemon).

## The FakeDaemon [#the-fakedaemon]

`FakeDaemon` in `@better-compute/testing` is the daemon side of the protocol in TypeScript. It mirrors the Go
packages rule for rule: the same journal rules, the same acceptance checks, the same lane credits and worker pools,
the same converger order, the same server time mark. Its fake hardware produces the same inventory section hashes as
Go's `fakehw` for the same seed. It runs over any frame transport: an in-memory link with fault injection, real
HTTP/2 to a real gateway, or the WebSocket fallback.

It is what runs wherever features bcd lacks, or many daemons without a real gateway, are needed:

* `better-compute dev --sim N` joins N simulated machines through a loopback gateway, the way bcd does: a join token
  sealed to JoinService's key, the `Join` RPC, a session, `diag.nvml` acceptance and adoption. See
  [local development](/docs/operations/local-development).
* `better-compute sim` runs the deterministic fleet simulator with crashes, reboots, journal loss, partitions and
  gateway kills, then checks invariants I1, I2, I3, I4, I5 and I8.
* `createFleetSim`, `createControlPlaneSim` and the real-time harness drive fleets from tests and benchmarks. See
  [testing](/docs/operations/testing).

```bash
better-compute dev --memory --sim 8
better-compute sim --nodes 100 --chaos heavy --seed 7
```

Both now speak the WebSocket fallback and self-update (bcd with a real A/B slot swap, the FakeDaemon's opt-in
`selfUpdate` in simulation). bcd matches it on joining, sessions, the journal, signed intent, the five v0.1 kinds, `driver.invoke` (the same driver
selection, secret handling and idempotency key), the `devices` section, the
`instances` rules (on a fakehw build; the FakeDaemon's `FakeHypervisor` and bcd's fake runtime follow the same ones),
the metadata service's contract, tunnels (both advertise `tunnel.instance-dial`: the FakeDaemon when its tunnels are
configured, bcd when it runs a runtime), secret and workload-token requests, XID conditions and host pressure. bcd is ahead on the `access` and `host`
sections, which the FakeDaemon still reports as `SectionUnsupported`, its tunnels run a real shell where the
FakeDaemon's answer with an echo shell, and its site agent and ramdisk act on real BMCs and disks (disk image files
under the `fakehw` build) where the FakeDaemon's simulate them.

  - [Installing bcd](/docs/daemon/install): The install script, the files it writes and upgrades.

  - [Protocol](/docs/daemon/protocol): Transport, proto packages, frames and version skew.

  - [Sessions](/docs/daemon/sessions): Hello, epochs, resume rules and liveness.

  - [Delivery](/docs/daemon/delivery): Desired state, commands, credits and the journal.

  - [Host configuration](/docs/daemon/host-config): The config units bcd applies from NodeSpec.host.

  - [On Kubernetes](/docs/daemon/kubernetes): bcd as a DaemonSet on an existing cluster.

  - [Gateway](/docs/daemon/gateway): The process that holds the sessions.

  - [Failure modes](/docs/daemon/failure-modes): What happens when each piece fails.


# Installing bcd (/docs/daemon/install)

> The install script that puts bcd on a server, the files and hardened systemd unit it writes, joining with BCD_JOIN_TOKEN, and upgrades.



A server becomes a node in two moves: an operator generates an install script once per pool, then runs it on each
machine with a single-use join token. The script downloads a pinned
bcd binary, checks its digest, writes a small configuration, installs a hardened systemd unit,
joins the fleet and starts the daemon. The script contains no secret, so you can keep it in a provisioning
repository; the token travels separately.

> **The pieces are not all runnable yet.** 
  The script generator (`better-compute daemon install-script`) is built and tested, and so is the
  `GET {basePath}/downloads/bcd/{version}/linux-{arch}` route it fetches from (see [serving the
  binaries](#serving-the-binaries)). bcd itself is built and tested: `bcd join` and `bcd node` work, and
  `tests/e2e/bcd.test.ts` runs them against a real gateway (see [the overview](/docs/daemon#what-bcd-does-today)). But
  bcd is not released: the release workflow exists, but no tag has been released, so there are no published bare
  binaries and digests for the script to pin, and no test yet runs the script itself on a host. You can build a binary from `daemon/` and serve it yourself; otherwise
  `better-compute dev --sim N` shows nodes joining through the same protocol with the TypeScript FakeDaemon. See
  [local development](/docs/operations/local-development).

## Serving the binaries [#serving-the-binaries]

The api serves the daemon binaries itself when you configure `downloads`:

```ts
betterCompute({
  // …
  downloads: { dir: '/srv/bcd' }, // or { blob: blobStore }, or both (the blob store is tried first)
});
```

`GET {basePath}/downloads/bcd/{version}/{os}-{arch}` answers goreleaser's bare binary `bcd_{version}_{os}_{arch}`,
read from `downloads.blob` under the key `bcd/{version}/{file}` or from `{dir}/{file}`. The route is public: the
install script and self-update check the sha256 from the verified release manifest, not the transport. Responses are
`application/octet-stream` with `cache-control: public, max-age=31536000, immutable`. The path must match a strict
pattern (a semantic version with an optional leading `v`, `linux`, `amd64` or `arm64`), so nothing else on disk can
be reached; anything else, or a binary the deployment does not have, is 404. `downloads` needs `dir` or `blob`, or
`betterCompute()` throws `INVALID_CONFIG`.

The same route serves the `bcx-ipmi` plugin (97b2ac2): `GET {basePath}/downloads/bcx-ipmi/{version}/{os}-{arch}`
answers goreleaser's archive `bcx-ipmi_{version}_{os}_{arch}.tar.gz`, from the blob key `bcx-ipmi/{version}/{file}`
or `{dir}/{file}`. The products are a fixed list, `bcd` and `bcx-ipmi`; any other name is 404. The route stays public
because the archive's digest is in the same signed release manifest, and the install script checks the download
against it ([installing plugins from the release](#installing-plugins-from-the-release)).

> **Where the binaries come from.** 
  The route only serves files you put there. Copy each release's bare binaries (from goreleaser's output) into the
  directory or blob store before you roll the version out.

## Generate the script [#generate-the-script]

`daemon install-script` reads the public half of the PKI bundle for the pins, so it needs no secret. Create the
bundle first with `better-compute pki init` (see [PKI](/docs/operations/pki)).

```bash
better-compute daemon install-script \
  --pool pool_h100 --site fsn1 \
  --gateway-url https://gw.eu.example.com \
  --base-url https://compute.example.com \
  --bcd-version 0.1.0 \
  --sha256 amd64=<hex>,arm64=<hex> > install-bcd.sh
```

| Flag                            | Required        | Environment                  | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------- | --------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--pool ID`                     | yes             |                              | The pool the node joins; named in the script's header                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `--site ID`                     | yes             |                              | The pool's site; written to `config.toml` as `site`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `--gateway-url URL`             | yes             | `BETTER_COMPUTE_GATEWAY_URL` | The regional gateway bcd dials, `https://...`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `--base-url URL`                | yes             | `BETTER_COMPUTE_BASE_URL`    | The control plane that serves the downloads                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `--base-path PATH`              | no              |                              | The API base path, default `/api/compute`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `--bcd-version VERSION`         | yes             | `BETTER_COMPUTE_BCD_VERSION` | The release to install, e.g. `0.1.0` (a leading `v` is dropped)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `--sha256 ARCH=HEX,...`         | one of two      |                              | SHA-256 of each architecture's bare binary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `--release-manifest PATH`       | one of two      |                              | A signed release manifest; its verified digests are pinned instead                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `--release-pins PINS`           | with a manifest | `BCD_RELEASE_PINS`           | Comma-separated `sha256/...` pins of the release keys; also written to `config.toml` as `[update] release_pins`, which [self-update](#self-update) needs                                                                                                                                                                                                                                                                                                                                                                                                             |
| `--release-threshold N`         | no              |                              | How many release keys must have signed (1 to 16, default 1); `[update] release_threshold`                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `--release-sequence N`          | no              |                              | The node's release floor, `[update] release_sequence`: bcd never installs an older release; needs `--release-pins`                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `--websocket-url URL`           | no              |                              | For networks whose TLS-inspecting proxies break mTLS: a `wss://.../v1/session` URL, written as the `[websocket]` section so bcd runs its session and renewals over the [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04) (0ea4ab4). A configuration that already has a `[websocket]` section is left alone, and the script says so                                                                                                                                                                                                                  |
| `--websocket-audience HOST`     | no              |                              | `[websocket] audience`, the gateway's public host the node's proof names; needs `--websocket-url`                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `--proxy-ca-file PATH`          | no              |                              | The TLS-inspecting proxy's CA, read from the local file and embedded only when it holds PEM certificates and nothing else; the script writes it to `/etc/bcd/websocket-proxy-ca.pem` and sets `[websocket] proxy_ca_file`. It is trusted for the WebSocket's TLS only: the daemon still proves its identity inside the session, and the gateway's proof must chain to the pinned serving CA. Needs `--websocket-url`                                                                                                                                                 |
| `--bmc-networks CIDRS`          | no              |                              | A site agent's BMC networks (comma-separated), written as `[site] bmc_networks`: `site.httpProxy` never reaches into them                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `--plugins`                     | no              |                              | Prepare the node for bcx driver plugins: a `[plugins]` section and the systemd drop-in; see [the plugin drop-in](#the-plugin-drop-in)                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `--privileged-plugins NAMES`    | no              |                              | Comma-separated plugins that run as root without the seccomp filter, such as a Kata runtime; implies `--plugins`                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `--install-plugins NAMES`       | no              |                              | Comma-separated bcx plugins to install from the release, each checked against its archive digest in the verified `--release-manifest` (adf2ae9). Only plugins a release ships are accepted: `bcx-ipmi`. Implies `--plugins` and needs `--release-manifest`; see [installing plugins from the release](#installing-plugins-from-the-release)                                                                                                                                                                                                                          |
| `--unprivileged`                | no              |                              | Run `bcd node` as the unprivileged `bcd` user beside a root `bcd helper` (169c459): a `[helper]` section, the `bcd` account, `bcd-helper.service` and a `bcd.service` drop-in; see [running bcd node unprivileged](#running-bcd-node-unprivileged). Refused with `--plugins`, `--privileged-plugins` or `--install-plugins`. Self-update works in this mode: the helper does the privileged half (2d67b25). On a `libvirt-kvm` host the helper defines and runs the domains, and the `bcd` account stays out of the `libvirt` group (d418820)                        |
| `--attestor tpm`                | no              |                              | Join with the host's TPM instead of a join token (a006d9f): the script reads no token, fails early without `/dev/tpmrm0`, and runs `bcd join --attestor tpm`; see [joining with the TPM](#joining-with-the-tpm). `tpm` is the only value; any other is `INVALID_INPUT`                                                                                                                                                                                                                                                                                               |
| `--instance-bridges BRIDGES`    | no              |                              | With `--unprivileged`, on VPS and sandbox hosts: the bridges an instance's NIC may join, comma-separated interface names (1 to 15 of letters, digits, `.`, `_` and `-`), written as `[helper] instance_bridges` unless the configuration already sets it (d418820). Without `--unprivileged` it is `INVALID_INPUT`                                                                                                                                                                                                                                                   |
| `--approver-set-sha256 HEX`     | no              |                              | Writes `approver_set_sha256` (64 lower-case hex digits, the `approverSetSha256` that `pki trust-version` prints) into `config.toml`, pinning the node's first approver set (9b7a199): a keyless bcd takes only that set, and runs nothing destructive until it has it; see [pinning the first set](/docs/operations/pki#pinning-the-first-set). An upgrade run without the flag keeps the node's existing pin (b6d401f)                                                                                                                                              |
| `--command-root-sha256 HEX,...` | no              |                              | Writes `command_root_sha256`, 1 to 16 lower-case hex SHA-256 digests of the command-signing root certificates (`pki trust-version` prints them as `commandRootSha256`), so `bcd helper` checks host config units against specs signed under them from its first request (59985ae; see [signed specs](/docs/daemon/host-config#signed-specs)). Without the flag the script pins the PKI bundle's command-signing root and, during a rotation, the added one, and nothing when the bundle has no command signing; an upgrade by a script without pins keeps the node's |
| `--pki PATH`                    | no              |                              | The PKI bundle to read the pins from                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `--out PATH`                    | no              |                              | Write a new file with mode 0755 instead of printing; refuses to overwrite (`FILE_EXISTS`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |

Give exactly one of `--sha256` and `--release-manifest`. With a manifest, the CLI verifies its signatures under
the pins and threshold, checks that it is for product `bcd` and for `--bcd-version` (`RELEASE_MISMATCH`
otherwise), and pins the one bare `linux-amd64` and `linux-arm64` binary it lists; archives, checksums and
metadata files are skipped. See [rollouts](/docs/guides/rollouts) for signing releases.

Every value embedded in the script is checked against a narrow pattern before it is written, so nothing a caller
passes can break out of the shell or TOML quoting:

| Value                                      | Must be                                                                                    |
| ------------------------------------------ | ------------------------------------------------------------------------------------------ |
| pool and site ids, privileged plugin names | letters, digits, `.`, `_` and `-`, starting with a letter or digit, at most 128 characters |
| version                                    | a semantic version such as `0.1.0` or `v0.2.0-rc.1`                                        |
| pins                                       | `sha256/` followed by the standard base64 of 32 bytes                                      |
| digests                                    | 64 lower-case hex digits, for `amd64` and/or `arm64` only                                  |
| URLs                                       | `https://` with letters, digits and `. - _ ~ /` only; trailing slashes are dropped         |
| join encryption key                        | a P-256 public JWK (`kty: EC`, `crv: P-256`)                                               |

A bad value fails with `INVALID_INPUT` naming the flag.

## What the script does [#what-the-script-does]

The output is a POSIX `sh` script that runs with `set -eu` and `umask 077`.

      ### Check the host [#check-the-host]

    It stops unless the host runs Linux, the caller is root, and `systemctl` and `sha256sum` exist. It maps
    `uname -m` to an architecture (`x86_64` or `amd64` to `amd64`, `aarch64` or `arm64` to `arm64`) and refuses an
    architecture the script carries no digest for.
  
      ### Read the token [#read-the-token]

    The token comes from `BCD_JOIN_TOKEN`, or else from the first line of standard input when standard input is not a
    terminal. The variable is unset right after. Anything that does not look like `bcjt_<id>.<secret>` is refused with
    exit code 2. Without a token, the script continues only when `/etc/bcd/config.toml` already exists; that is an
    upgrade (below).
  
      ### Download and verify [#download-and-verify]

    It fetches `{downloads}/bcd/{version}/linux-{arch}` into a temporary directory with `curl` (HTTPS only, TLS 1.2 or
    later, three retries) or `wget --https-only`, and compares the file with the pinned SHA-256. A mismatch stops the
    script before anything is installed.
  
      ### Install the files [#install-the-files]

    It creates `/etc/bcd` and `/var/lib/bcd` with mode 0700 and `/etc/cdi` with mode 0755 (and, with `--plugins`,
    `/usr/local/lib/bcd/plugins` with mode 0755). Each file is written next to its target as `*.new` and renamed over
    it, so a replacement is atomic even while bcd runs. When a `config.toml` already exists, the script writes the
    top-level keys it sets itself (`gateway_url`, `site`, the three pins, and `approver_set_sha256` and
    `command_root_sha256` when it has them), then carries over every other top-level key of the existing file as it is
    written there (`gateway_server_name`, `data_dir`, `socket_path`, a pin of an earlier install the new script has no
    flag for), never a key twice, and then every section of it (`[plugins]`, `[boot]`, `[imds]`, ...); with `--plugins`, a
    `[plugins]` section is appended unless the file already has one, and with `--unprivileged` a `[helper]` section
    likewise. After the unit, it installs or removes the [plugin drop-in](#the-plugin-drop-in) according to whether
    `config.toml` has a `[plugins]` section, and the [helper mode](#running-bcd-node-unprivileged) (the `bcd` account,
    `bcd-helper.service`, the `unprivileged.conf` drop-in, `/etc/bcd` readable by the `bcd` group) according to whether
    it has a `[helper]` section. Then `systemctl daemon-reload`.
  
      ### Check, join and start [#check-join-and-start]

    It runs `bcd doctor --config /etc/bcd/config.toml` and prints a warning if doctor reports a problem, without
    stopping. Before the first join, doctor's `identity` check only warns that the node has not joined; a gateway it
    cannot reach over TCP is a failure. With a token, it pipes the token into `bcd join --config /etc/bcd/config.toml` with `printf`, a shell
    builtin, so the token never appears in a process listing, then runs `systemctl enable --now bcd.service`. Without
    a token, it runs `systemctl restart bcd.service`. In helper mode it first hands `/var/lib/bcd` to the `bcd` user
    (`chown -R bcd:bcd`, since `bcd join` wrote the identity as root) and enables and starts, or restarts,
    `bcd-helper.service`, so the helper is up before the node.
  
Download the script before running it. Piping it into `sh` would leave no standard input for the token.

```bash
BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh
# or, keeping the token out of the environment:
sudo sh install-bcd.sh < token-file
```

### Joining with the TPM [#joining-with-the-tpm]

A script generated with `--attestor tpm` (a006d9f) joins the machine with its TPM, through the control plane's
[`tpm` attestor](/docs/guides/nodes/attestation#tpm), and carries no token step. Its header says the attestor's
policy decides where the machine joins. It differs in two places:

* **Instead of reading a token**, it unsets `BCD_JOIN_TOKEN` and stops unless `/dev/tpmrm0`, the kernel's TPM
  resource manager, exists.
* **Instead of the token join**, it runs `bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null`, which
  returns at once on a machine that already joined, so the same script installs and upgrades. Then it enables and
  restarts `bcd-helper.service` (in helper mode, after `chown -R bcd:bcd /var/lib/bcd`) and `bcd.service`. As root on
  a node with `[helper]`, the join also keeps the signed join record the helper binds host units to (see
  [signed specs](/docs/daemon/host-config#signed-specs)).

The join makes the node key in the same TPM and sends the TPM's certification of it, which `tpmAttestor` requires by
default (`nodeKey: 'certified'`, cf24876). The join runs as root and opens the TPM itself, with or without
`[helper]`; under `[helper]` the unprivileged `bcd node` then reaches that key only through `bcd helper` (8870b8d), and
never opens `/dev/tpmrm0` ([below](#running-bcd-node-unprivileged)). If the TPM cannot make the key, the join stops
with an error. See [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm).

```bash
better-compute daemon install-script --pool pool_tpm --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex> --attestor tpm > install-bcd.sh
sudo sh install-bcd.sh
```

The control plane must run `tpmAttestor` with a `resolve` allow-list or `enrollment` for the machine's endorsement
key, else the join is refused or waits for approval; `--pool` only names the pool in the header. A pool can then
refuse token joins with `policy.joinAttestors: ['tpm']` (see
[requiring an attestor](/docs/guides/nodes/attestation#requiring-an-attestor)).

## Files on the host [#files-on-the-host]

      - `bcd (0755)`
  
      - `config.toml (0600)`

    - `join-encryption.jwk (0644, public)`
  
      - `bcd.service (0644)`

    - `bcd-helper.service (0644, only on nodes with [helper])`

          - `plugins.conf (0644, only on nodes with [plugins])`

      - `unprivileged.conf (0644, only on nodes with [helper])`
      
      - `pins (sha256sum format) and the plugin binaries, yours to add`
  
      - `node.key (0600, the node key, or its TPM record)`

    - `identity.json (certificate chain, trust bundle, server time mark)`

    - `journal.db (the bbolt journal)`

    - `host-intent.json (0600, the signed spec that last set the host section)`

    - `volumes/ID.raw, ID.json (0600, persistent host volumes)`
  
      - `bootargs/UNIT, hugepages-1g-boot, cdi-nvidia.stamp (the host units' records)`

    - `k8s-join.json, k8s/bootstrap-token (k8s-join)`

    - `helper-intent.json (bcd helper's accepted signed spec)`

    - `libvirt/ (bcd helper's staging for domain definitions and memory saves)`
  
      - `bcd.sock (0600 status socket, owned by the user bcd runs as)`
  
      - `helper.sock (0660, root:bcd)`
  
      - `bcd-ALLOCATION.json (written by the devices converger)`
  
The script writes the binary, `config.toml`, `join-encryption.jwk`, the unit and, on a plugin node, the drop-in and
the (empty) plugin directory. The other paths belong to the daemon: `bcd join` writes the key and
`identity.json` under `data_dir`, `bcd node` opens the journal there, serves the status socket at `socket_path`, and
writes CDI specs for `hold` bindings (with env files under `/run/bcd/allocations`). On a host with a TPM
(`/dev/tpmrm0`) the key is made in the TPM and `node.key` holds only its record, a PEM block `BCD TPM NODE KEY`;
otherwise it is a PKCS#8 file (see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)).
A renewal writes the next key as `node.key.next` and promotes it only once the new certificate is stored. A journal that does not open is moved aside
as `journal.db.corrupt-<unix time>` and a new one starts. The token is consumed by the join and never written
anywhere. With `--proxy-ca-file` the script also writes `/etc/bcd/websocket-proxy-ca.pem`. After its first
[self-update](#self-update) bcd owns `<data_dir>/bin/bcd-a`, `bcd-b`, `current`, `previous` and `slots.json`,
`upgrade.json` while an update awaits its `Welcome`, `update.json` (the release floor and failed versions), and,
under systemd, `bcd-rollback.service` and `bcd.service.d/50-bcd-rollback.conf`; `/usr/local/bin/bcd` then links to
`current`. Under `bcd helper` the slots and both records are in root's `[update] slots_dir` instead (default
`/usr/local/lib/bcd`), and the node downloads into `<data_dir>/staging`. A quorum node keeps `batch-leaves.json` and
`unbatched-destructive.json`, and a site agent `bmc-hosts.json`, under `data_dir`.

### config.toml [#configtoml]

```toml title="/etc/bcd/config.toml"
# Written by the better-compute install script. bcd refuses unknown keys.
gateway_url = "https://gw.eu.example.com"
site = "fsn1"
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."
```

bcd reads the file with `daemon/internal/config`, a strict subset of TOML 1.0: bare keys, basic or literal
strings, decimal integers (with `_` between digits) and booleans, in the top level or a bare `[section]`. Arrays,
inline tables, floats, dates, multi-line strings, dotted or quoted keys, unknown keys, unknown sections, duplicate
keys and wrong types are all errors with a line number, so a typo fails loudly. Files over 64 KiB are refused.

| Key                                          | Required         | Default                                                   | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------------- | ---------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `gateway_url`                                | yes              |                                                           | An `https://` URL without credentials                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `site`                                       | yes              |                                                           | The site id, no whitespace                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `serving_ca_spki_pin`                        | yes              |                                                           | Pins the gateway serving CA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `join_signing_key_pin`                       | yes              |                                                           | Pins the JoinService key that signs `JoinResponse`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `join_encryption_key_pin`                    | yes              |                                                           | Pins the JoinService key the token is sealed to                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `data_dir`                                   | no               | `/var/lib/bcd`                                            | Absolute path for the journal and key                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `socket_path`                                | no               | `/run/bcd/bcd.sock`                                       | Absolute path of the status socket                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `[journal] max_bytes`                        | no               | 67108864                                                  | May lower the 64 MiB cap, never raise it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `[journal] max_events`                       | no               | 10000                                                     | May lower the 10,000 unacked-event cap, never raise it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `[log] level`                                | no               | `info`                                                    | `debug`, `info`, `warn` or `error`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `[log] json`                                 | no               | `true`                                                    | JSON lines                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `gateway_server_name`                        | no               |                                                           | The name the gateway certificate must carry when `gateway_url` names an IP address (an L4 balancer, a dev stack reached from WSL)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `dev_insecure_h2c`                           | no               | `false`                                                   | Allows an `http://` `gateway_url` (cleartext HTTP/2 to a dev gateway). Only fakehw builds accept it; the host build refuses to start                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `approver_set_sha256`                        | no               |                                                           | 64 lower-case hex digits: the only approver set a daemon without approver keys takes first (see [pinning the first set](/docs/operations/pki#pinning-the-first-set)); `--approver-set-sha256` writes it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `command_root_sha256`                        | no               |                                                           | Comma-separated lower-case hex SHA-256 digests of the command-signing root certificates, at most 16 (03c023d); `pki trust-version` prints them as `commandRootSha256`. `bcd helper` then applies a `host.unit` only with a spec signed under these roots, which decide alone. `daemon install-script` writes it by default from the PKI bundle's current command-signing root and, during a rotation, the next one; `--command-root-sha256` overrides that. Without it the helper takes the roots from the `SignedTrustBundle` the JoinService key signed, checked against `join_signing_key_pin` (2d18327): it refuses a bundle older than one it already accepted (a rollback) and, once it has accepted a bundle with roots, any request without a signed spec; before that it checks nothing. A malformed value stops bcd from starting. See [signed specs](/docs/daemon/host-config#signed-specs) |
| `[update] release_pins`                      | no               | none                                                      | Comma-separated `sha256/...` pins of the release keys. Without them bcd refuses every self-update; see [self-update](#self-update)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `[update] release_threshold`                 | no               | `1`                                                       | How many distinct pinned keys must have signed a release                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `[update] release_sequence`                  | no               | `0`                                                       | The release the node was installed at: no release manifest with a lower sequence is taken (0: unknown)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `[update] entrypoint`                        | no               | `/usr/local/bin/bcd`                                      | The path `bcd.service` starts; the first update makes it a symlink to `current` in the slots directory. Under the helper it may not be inside `data_dir` ("under bcd helper, update.entrypoint may not be inside data\_dir"; 9e71b1f), since root adopts the binary found there into a slot                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `[update] slots_dir`                         | no               | `<data_dir>/bin`; under `bcd helper` `/usr/local/lib/bcd` | An absolute path for the binary slots and, beside them, `upgrade.json` and `update.json` (without it and without the helper, the records stay in `data_dir`; 2d67b25). Under the helper it may neither be inside `data_dir` nor contain it ("under bcd helper, update.slots\_dir may not be inside data\_dir"; both paths cleaned first, c139b9c), since the unprivileged node owns that directory; the helper creates it                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `[helper] socket`                            | no               |                                                           | An absolute path: `bcd node` then runs unprivileged and sends its privileged operations, self-update's included, to `bcd helper` on this unix socket (28dc2ae); `--unprivileged` writes `/run/bcd-helper/helper.sock`. The directory must be root's with no group or other write. See [running bcd node unprivileged](#running-bcd-node-unprivileged)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `[helper] user`                              | with `socket`    |                                                           | The unprivileged account `bcd node` runs as (`bcd` from `--unprivileged`); `root` is refused ("helper.socket needs an absolute path and helper.user, the unprivileged account bcd node runs as")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `[helper] instance_bridges`                  | no               |                                                           | Comma-separated bridges an instance's NIC may join (fea4d2e). `bcd helper` refuses to define a libvirt domain with a NIC on any other bridge, and with none set it defines no domain with a NIC at all. A name that is not an interface name stops bcd from starting ("helper.instance\_bridges: … is not an interface name"). `--instance-bridges` writes it; see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper)                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `[site] bmc_networks`                        | no               |                                                           | A site agent's BMC networks, comma-separated CIDRs: `site.httpProxy` never reaches into them, whatever the control plane asks. A site agent with it advertises the capability `site.bmc-networks`, and `better-compute doctor` warns about connected site agents without it (a019c85)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `[websocket] url`                            | no               |                                                           | A `wss://` URL (`ws://` only with `dev_insecure_h2c`), without credentials, of the gateway's WebSocket endpoint, `.../v1/session`: `bcd node` and `bcd site` then run the session and renewals over the [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `[websocket] proxy_ca_file`                  | no               |                                                           | An absolute path to a PEM file of a TLS-inspecting proxy's CA, trusted for the TLS connection only, never for the gateway's proof                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `[websocket] audience`                       | no               | `gateway_server_name`, else the URL's host                | The gateway's public host the node's proof names                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `[instances] balloon_floor_percent`          | no               | `50`                                                      | 10 to 100: the share of its memory a ballooned guest always keeps (12e9ced); bcd also never takes a guest below its use plus 10%. Outside the range bcd does not start. See [memory ballooning](/docs/daemon/delivery#memory-ballooning)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `[imds] listen`                              | no               |                                                           | Where a VPS host serves its instances' metadata service, for example `169.254.169.254:80` on the instances' bridge; empty turns it off                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `[host] units`                               | no               | `true`                                                    | `false` when something else manages the host's configuration (the NVIDIA GPU operator beside [bcd as a DaemonSet](/docs/daemon/kubernetes)): bcd applies no `NodeSpec.host` unit and reports the section `UNSUPPORTED` with reason `HostManagedExternally` (a193249). See [host configuration](/docs/daemon/host-config)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `[kubernetes] allowed_clusters`              | for a `k8s-join` |                                                           | Comma-separated `https://host[:port]#sha256:<CA pin>` entries: the only clusters the `k8s-join` unit may join this machine to (7579184). Since 9f860ad it is required as root too: without it every join is refused, and `bcd doctor` fails a joined node without one. A malformed entry stops `bcd node` and `bcd helper` from starting. See [k8s-join](/docs/daemon/host-config#k8s-join)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `[kubernetes] gpu_api`                       | no               | `device-plugin`                                           | How a node bcd did not join itself (the cluster's DaemonSet) offers bound GPUs to the kubelet: `device-plugin` or `dra`, never both (7d24836); any other value stops bcd from starting. A node `k8s-join` joined takes the unit's `gpuApi` instead. See [GPUs for pods](/docs/daemon/kubernetes#gpus-for-pods)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `[boot] ...`                                 | no               |                                                           | A site agent's boot server; see [site agents](#a-site-agent)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `[plugins] dir`                              | no               |                                                           | A directory of bcx driver plugins and their `pins` file (sha256sum format); only pinned plugins run. Empty runs none. See [driver plugins in bcd](/docs/extending/drivers#driver-plugins-in-bcd)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `[plugins] user`                             | no               |                                                           | Who plugins run as when bcd is root (no supplementary groups); unset, each plugin gets its own uid from a block bcd keeps, `0x7FFE0000` to `0x7FFEFFFF`. `root` is refused                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `[plugins] privileged`                       | no               |                                                           | Plugins (comma-separated names) that run as root without the shim or the filter, such as a Kata runtime                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `[plugins] sandbox`                          | no               | `true`                                                    | Runs unprivileged plugins through bcd's `__bcx-exec` shim: `no_new_privs` and a seccomp filter. Linux only; elsewhere set `false` or no plugin starts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `[plugins] cgroup`                           | no               |                                                           | An absolute path under `/sys/fs/cgroup` in bcd's delegated cgroup; bcd creates it and enables the memory and pids controllers, and each plugin runs in its own child `bcx-<name>`. `--plugins` writes `/sys/fs/cgroup/system.slice/bcd.service/plugins`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `[plugins] memory_max`, `[plugins] pids_max` | no               | 0 (none)                                                  | The child cgroup's `memory.max` (bytes) and `pids.max`; not negative. See [the plugin sandbox](/docs/extending/drivers#the-plugin-sandbox)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

A pin is `sha256/` plus the standard base64 of the SHA-256 of a SubjectPublicKeyInfo, the form HPKP and
`curl --pinnedpubkey` use. `bcd doctor` warns when the file is readable or writable by anyone but its owner.

### join-encryption.jwk [#join-encryptionjwk]

The pins alone cannot seal a token, so the script also writes the JoinService encryption public key as a JWK.
`bcd join` uses it only when its SPKI hash equals `join_encryption_key_pin`. When the file is absent and
`bcd join --server https://…` names the control plane, bcd fetches `<server>/.well-known/better-compute` (system
roots, a 30-second timeout, at most 64 KiB) and uses its `join.encryptionJwk`, again only when it matches the pin.
The design's fallback for a stale file is not implemented: a mismatched file, or a missing one without `--server`,
fails `bcd join` with exit code 1. See [the join key and its fallback](/docs/guides/nodes/joining#the-join-key-and-its-fallback).

An OS that `deploy.image` installed needs neither the script nor this file: the ramdisk writes its own pinned
`config.toml` (gateway, server name, site and the three pins) and `join-encryption.jwk` into the installed root, both
mode 0600, and the deploy's user data runs `bcd join --server … --token-file /etc/bcd/join-token --node-id …` (with
the three pin flags when provision is given `userData.pins`), then `bcd node`. See
[profiles and images](/docs/guides/bare-metal/profiles-and-images).

### Joining from the well-known document [#joining-from-the-well-known-document]

A machine with no `config.toml` at all, such as an OS that Hetzner Robot's `bcd-join.service` or provision's user
data set up with a token, the control plane's URL and three pins, can still join. `bcd join --server
https://… --serving-ca-pin … --join-signing-pin … --join-encryption-pin …` reads
`<server>/.well-known/better-compute` for what the pins cannot say (where the gateway is), checks the rest against the
pins, joins, and writes its own configuration.

| Field of the document                         | Becomes                                                                                                                           |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `gateway.url`                                 | `gateway_url`                                                                                                                     |
| `gateway.serverName`                          | `gateway_server_name`                                                                                                             |
| `gateway.servingCaPin`                        | compared with `--serving-ca-pin`, which becomes `serving_ca_spki_pin`                                                             |
| `join.signingKeyPin`, `join.encryptionKeyPin` | compared with `--join-signing-pin` and `--join-encryption-pin`, which become `join_signing_key_pin` and `join_encryption_key_pin` |
| `join.encryptionJwk`                          | the key the token is sealed to, used only when it matches `--join-encryption-pin`                                                 |

The pins are `sha256/<base64>`, as in `config.toml`, and a host build requires all three to bootstrap ("joining
without config.toml needs --serving-ca-pin, --join-signing-pin and --join-encryption-pin from the install"). A
document whose published pin differs from one given is refused ("refusing to join"). Only a fakehw development build
may join with no pins at all, taking them from the document (and it also accepts an `http://` server). On a machine
that already has `config.toml`, the configuration wins, and any pin flag given must match it.

The control plane publishes the `gateway` section when its options set `gateway: { url, serverName? }`, with
`servingCaPin` computed from `pki.servingCa`; the examples read `BETTER_COMPUTE_GATEWAY_URL` and
`BETTER_COMPUTE_GATEWAY_SERVER_NAME`. bcd fetches the document over the control plane's own TLS (system roots, a
30-second timeout, at most 64 KiB, redirects refused), refuses a document without `gateway.url` ("install
/etc/bcd/config.toml instead"), joins, and only then writes `config.toml` (0600, with the site the join answered)
and `join-encryption.jwk` (0644) beside it. Re-running it on a node that already joined exits 0 without a token, so a
join unit can run on every boot. The pins come from the sealed PKI bundle: `bootstrapPins(bundle)` in
`better-compute/pki` returns all three, and provision's user data (`provisionModule({ userData: { pins } })`) and a
Robot bootstrap (`vps({ bootstrap: { pins } })`) render them into the machine; see
[profiles and images](/docs/guides/bare-metal/profiles-and-images).

> **Why the pins travel with the install.** 
  The well-known document is authenticated only by the control plane's TLS certificate. Taking its pins on trust would
  be trust on first use: whoever can serve that name with a publicly trusted certificate (a DNS hijack plus a
  certificate for the name) could hand the machine their own gateway and JoinService keys, open the sealed token and
  take the node's identity. Pins that come with the install, from the sealed bundle, are stronger, which is why a host
  build refuses to bootstrap without them. A fakehw build joining with no pins is trust on first use; keep that to
  development.

## The systemd unit [#the-systemd-unit]

By default the script installs `bcd node` running as root, so the unit narrows what root can do. With
`--unprivileged` the same unit gets a drop-in that runs it as the `bcd` user beside a root `bcd helper`; see
[running bcd node unprivileged](#running-bcd-node-unprivileged).

```ini title="/etc/systemd/system/bcd.service"
[Unit]
Description=better-compute node agent (bcd)
Documentation=https://github.com/better-compute/better-compute
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=10

[Service]
Type=simple
ExecStart=/usr/local/bin/bcd node --config /etc/bcd/config.toml
Restart=always
RestartSec=5s
UMask=0077
StateDirectory=bcd
StateDirectoryMode=0700
RuntimeDirectory=bcd
RuntimeDirectoryMode=0700
ConfigurationDirectory=bcd
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=-/etc/cdi -/var/run/cdi
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
LockPersonality=yes
SystemCallArchitectures=native
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
CapabilityBoundingSet=CAP_SYS_BOOT CAP_SYS_ADMIN CAP_SYS_RAWIO CAP_DAC_READ_SEARCH CAP_NET_ADMIN CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE

[Install]
WantedBy=multi-user.target
```

| Setting                                                                           | Effect                                                                                                                       |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `ProtectSystem=strict`                                                            | The whole file system is read-only for bcd, except its state, runtime and configuration directories and the `ReadWritePaths` |
| `ReadWritePaths=-/etc/cdi -/var/run/cdi`                                          | The CDI spec directories the devices converger writes; the `-` tolerates a missing directory                                 |
| `StateDirectory`, `RuntimeDirectory`                                              | systemd creates `/var/lib/bcd` and `/run/bcd` with mode 0700                                                                 |
| `NoNewPrivileges`, `RestrictSUIDSGID`                                             | No privilege gain through exec or set-id files                                                                               |
| `ProtectHome`, `PrivateTmp`                                                       | No access to home directories; a private `/tmp`                                                                              |
| `ProtectKernelModules`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname` | Kernel modules, cgroups, the clock and the hostname are off limits                                                           |
| `RestrictAddressFamilies`                                                         | Unix sockets, IPv4, IPv6 and netlink (for NIC inventory) only                                                                |
| `CapabilityBoundingSet`                                                           | Rebooting, device and raw I/O access, network administration and file ownership; nothing else                                |
| `Restart=always`, `StartLimitBurst=10` in 300 s                                   | Restarts after 5 s, and gives up after 10 starts in five minutes                                                             |

`bcd node` opens no TCP listener; everything it does starts with an outbound connection to the gateway. See
[privilege separation](/docs/daemon#privilege-separation) for the helper mode.

### The plugin drop-in [#the-plugin-drop-in]

The unit above cannot run sandboxed [bcx plugins](/docs/extending/drivers#the-plugin-sandbox): its
`CapabilityBoundingSet` has no `CAP_SETUID` or `CAP_SETGID`, which bcd needs to start each plugin under its own uid,
and `ProtectControlGroups=yes` makes the cgroup tree read-only. The unit stays that tight; a node that runs plugins
gets a drop-in instead. The script installs `/etc/systemd/system/bcd.service.d/plugins.conf` (0644) exactly when
`/etc/bcd/config.toml` has a `[plugins]` section, and removes it otherwise, on every run, first install and upgrade
alike:

```ini title="/etc/systemd/system/bcd.service.d/plugins.conf (with a privileged plugin)"
[Service]
# bcx plugins ([plugins] in /etc/bcd/config.toml): their own uids, their own cgroups.
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_KILL
Delegate=yes
DelegateSubgroup=main
ProtectControlGroups=no
# plugins.privileged in /etc/bcd/config.toml: runtime plugins that run VMs as root (Kata).
CapabilityBoundingSet=CAP_MKNOD CAP_SYS_RESOURCE CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_RAW CAP_SYS_CHROOT CAP_FSETID CAP_SETPCAP CAP_SYS_PTRACE
RestrictNamespaces=no
RestrictAddressFamilies=AF_VSOCK AF_PACKET
ProtectKernelModules=no
```

| Setting                                                | Why                                                                                                                                         |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_KILL` | Added to the unit's set: switch each plugin to its own uid and gid, and stop it. Nothing else is loosened                                   |
| `Delegate=yes`, `DelegateSubgroup=main`                | systemd hands bcd its cgroup, `system.slice/bcd.service`, and runs bcd itself in the `main` child, so the plugins' cgroup can sit beside it |
| `ProtectControlGroups=no`                              | bcd can create the plugins' cgroups and write their `memory.max` and `pids.max`                                                             |

The last four lines are added only when `[plugins] privileged` names a plugin (3b7ae79, following d902063): a runtime
plugin such as Kata runs VMs as root, so it needs close to root's capabilities (locking VM memory, device nodes,
virtiofsd's ptrace), namespaces and mounts, vsock and packet sockets, and the vhost and vsock kernel modules:

| Setting                                                                                                                                       | Why                                                                       |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `CapabilityBoundingSet=CAP_MKNOD CAP_SYS_RESOURCE CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_RAW CAP_SYS_CHROOT CAP_FSETID CAP_SETPCAP CAP_SYS_PTRACE` | Added to the bounding set for the privileged plugin, which runs as root   |
| `RestrictNamespaces=no`                                                                                                                       | The plugin makes namespaces and mounts                                    |
| `RestrictAddressFamilies=AF_VSOCK AF_PACKET`                                                                                                  | Added to the allowed socket families: vsock to its guests, packet sockets |
| `ProtectKernelModules=no`                                                                                                                     | The vhost and vsock modules                                               |

Confined plugins are unchanged: each drops every capability when bcd switches it to its own uid, with
`no_new_privs` and the seccomp filter. A node without privileged plugins gets none of these four lines.

`--plugins` makes a node a plugin node. The script then creates `/usr/local/lib/bcd/plugins` (root-owned, 0755, so
each plugin's uid can reach its binary) and, unless the configuration already has a `[plugins]` section, appends
one:

```toml title="/etc/bcd/config.toml (with --plugins --privileged-plugins kata)"
[plugins]
dir = "/usr/local/lib/bcd/plugins"
cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins"
privileged = "kata"
```

The cgroup is a child of bcd's delegated cgroup. At start bcd creates it and enables the memory and pids
controllers in it and in its parent; on systemd older than 254, which ignores `DelegateSubgroup=main`, bcd first
moves itself into a `main` leaf, since cgroup v2 gives no controllers to the children of a cgroup that holds
processes itself (d902063). If the cgroup cannot be prepared, no plugin runs and the log says why.
`--privileged-plugins NAMES` implies `--plugins` and writes `privileged`; each name must be letters, digits, `.`,
`_` and `-`. Other plugins you put in the directory yourself, the binaries and their lines in its `pins` file (see
[driver plugins in bcd](/docs/extending/drivers#driver-plugins-in-bcd)); the plugins a release ships the script can
install (below). A section you wrote by hand works the same way: any `[plugins]` section gets the drop-in, and
removing the section removes the drop-in on the next run.

#### Installing plugins from the release [#installing-plugins-from-the-release]

`--install-plugins NAMES` (adf2ae9) installs bcx plugins that a bcd release ships, today only `bcx-ipmi`, the IPMI
BMC driver a site agent hosts (see [IPMI through bcx-ipmi](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi)). It
implies `--plugins` and needs `--release-manifest`: the CLI takes each plugin's archive digests,
`bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz`, from the verified, signed manifest (`RELEASE_INCOMPLETE` when it
lists neither), and refuses any other name.

```bash
better-compute daemon install-script --pool pool_site --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.9.1 --release-manifest dist/manifest.json --release-pins sha256/... \
  --install-plugins bcx-ipmi > install-bcd.sh
```

For each plugin the script:

1. downloads `{base-url}{base-path}/downloads/<name>/<version>/linux-<arch>` over https (the same version as bcd)
   and checks the archive against its pinned SHA-256;
2. refuses an archive that names a path outside it (a leading `/` or a `..`), that has no member named after the
   plugin, or whose member is not a regular file (a symlink or hard link), and extracts only that member;
3. installs it with mode 0755 as `/usr/local/lib/bcd/plugins/<name>` (written as `<name>.new` and renamed);
4. writes its binary's SHA-256 as its line in the directory's `pins` file (0644), replacing an older line for the
   same plugin and keeping every other line.

A rerun replaces the plugin and its pin without duplicating the line. Like `--plugins`, it is refused with
`--unprivileged`. The script itself sets up a node (`bcd node`); a site agent that reads the same `config.toml` finds
the plugin in its `[plugins] dir` (see [a site agent](#a-site-agent)). `tests/cli-pki.test.ts` runs the install
function in a real `sh` against real archives, including a wrong digest, a symlinked member and a `../` member.

### Running bcd node unprivileged [#running-bcd-node-unprivileged]

`bcd helper` (28dc2ae, PLAN.md §6, v0.4) lets `bcd node` run as an unprivileged account while a root process beside
it does the privileged work: the CDI specs, allocation env files and sshd files bcd owns, the `bcd_instances`
and `bcd_taps` nftables tables, GPU resets, reboots and sshd reloads. [Privilege separation](/docs/daemon#privilege-separation)
describes the operations and the checks. The mode is opt-in: generate the script with `--unprivileged` (169c459).

```bash
better-compute daemon install-script --pool pool_h100 --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex> --unprivileged > install-bcd.sh
```

The flag adds a `[helper]` section to `config.toml` unless the file already has one. Both processes read that file:

```toml title="/etc/bcd/config.toml ([helper] section, as --unprivileged writes it)"
[helper]
socket = "/run/bcd-helper/helper.sock"
user = "bcd"
```

| Key                         | Meaning                                                                                                                                                                                                                                                                       |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `[helper] socket`           | The absolute path of the helper's unix socket. Unset (the default), `bcd node` does its privileged work itself, as root                                                                                                                                                       |
| `[helper] user`             | The account `bcd node` runs as, and the only caller the helper serves besides root; its primary group owns the socket. Required with `socket`, and never `root`                                                                                                               |
| `[helper] instance_bridges` | On a VPS or sandbox host: the bridges instance NICs may join, which the helper checks every libvirt domain and every `cloud-hypervisor` tap (`ch.taps`, 7fe74a1) against; unset, it defines no domain with a NIC and creates no tap. `--instance-bridges br0,br-priv` adds it |

The socket is not in `/run/bcd`: the `bcd` user owns that directory, and a directory the node can write would let it
redirect root's work on the socket path with a symlink. So it lives in the helper's own root-owned runtime directory.
Since 2d67b25 the helper refuses to start unless the socket's directory is root's and closed to group and other
writes, and it binds, chmods and chowns the socket in a private 0700 directory before renaming it into place. Since
c139b9c, a helper running as root also refuses to start unless every existing directory from `/` down to the slots
directory, the socket's directory and (since 9e71b1f) the entrypoint's directory is root's and closed to group and
other writes. It checks this before it touches its update records or arms a watchdog. A host whose `/usr/local/lib` is
group-writable (`root:staff` 2775 on older Debian) fails it; `bcd doctor`, run as root on a helper node, reports each
such directory as a failed `helper` check.

The script decides the mode from the configuration, not the flag: it sets up helper mode exactly when `config.toml`
has a `[helper]` section, so an upgrade run keeps the node's mode, and deleting the section returns the node to root
on the next run (the drop-in is removed; the account and `bcd-helper.service` are left, and the helper cannot start
without the section, so disable that unit yourself). In helper mode it:

1. creates the `bcd` system account if there is none (`useradd --system --no-create-home --home-dir /var/lib/bcd --shell /usr/sbin/nologin bcd`, or busybox `adduser -S -D -H` without `useradd`);
2. installs `bcd-helper.service` and the `bcd.service` drop-in `unprivileged.conf`, both 0644;
3. creates `/usr/local/lib/bcd`, where the helper keeps the self-update slots, with mode 0755, and sets an existing
   one to 0755 (27c9551). The helper's `UMask=0077` would otherwise leave it 0700, and `bcd node`, running as `bcd`,
   could not execute the slot its entrypoint links to after the first update. The helper itself creates every
   missing directory on the way to the slots with mode 0755 whatever its umask, and never changes the mode of one
   that exists (c139b9c, 9e71b1f);
4. makes `/etc/bcd` (0750) and `config.toml` (0640) group-owned by `bcd`, so the node can read its configuration and
   nothing else of `/etc/bcd` becomes readable to it. Both stay root's: since 3fa9021 `bcd helper` refuses to start
   as root unless its `config.toml` is a regular file root owns that no one else can write, in directories only root
   can write, because that file (`[host] units`, `command_root_sha256`, the join pin) decides what root does for the
   node. Do not hand the file to `bcd` or make it group-writable;
5. on a host with `/etc/libvirt/qemu.conf` (d418820): sets `dynamic_ownership = 0` there (replacing the line,
   commented or not, or appending it), adds `libvirt-qemu` or `qemu`, whichever exists, to the `bcd` group, and
   restarts `virtqemud.service` or `libvirtd.service` if it runs. libvirt then chowns no path the node names, and
   QEMU reaches the node's disk files through the `bcd` group (see
   [libvirt through the helper](/docs/daemon#libvirt-through-the-helper));
6. keeps the `bcd` user away from the TPM (8870b8d, security review 3). The kernel's resource manager filters no
   command, so any member of the `tss` group could change the hierarchies' authorization, clear the TPM or trip its
   dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation. `bcd node` therefore
   never opens `/dev/tpmrm0`: `bcd helper` derives, signs with and certifies its node keys (see
   [node keys through the helper](/docs/daemon#node-keys-through-the-helper)). Scripts from 0ba2a79 until then added
   `bcd` to `tss` and wrote a udev rule; an upgrade run takes `bcd` out of `tss` when it is a member (`gpasswd -d bcd
   tss`, or busybox `delgroup bcd tss`) and deletes `/etc/udev/rules.d/60-bcd-tpm.rules`. It does not change the
   device's current group or mode back. Until that run, a host installed between 0ba2a79 and 8870b8d keeps `bcd` in
   `tss`, and since e6c3f53 `bcd doctor` fails its `tpm` check there: "bcd is in the tss group: it can send the TPM
   any command (lock or clear its hierarchies); bcd helper holds the node keys, so remove it (gpasswd -d bcd tss)
   and restart bcd.service". Rerun the script, or do that by hand;
7. after `bcd join` (which runs as root), gives `/var/lib/bcd` to `bcd:bcd` and starts `bcd-helper.service` before
   `bcd.service`; an upgrade run does the same with restarts.

```ini title="/etc/systemd/system/bcd-helper.service"
[Unit]
Description=better-compute node agent privileged helper (bcd helper)
Documentation=https://github.com/better-compute/better-compute
Before=bcd.service

[Service]
Type=simple
ExecStart=/usr/local/bin/bcd helper --config /etc/bcd/config.toml
Restart=always
RestartSec=2s
UMask=0077
RuntimeDirectory=bcd-helper
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
```

```ini title="/etc/systemd/system/bcd.service.d/unprivileged.conf"
[Unit]
Requires=bcd-helper.service
After=bcd-helper.service

[Service]
User=bcd
Group=bcd
SupplementaryGroups=systemd-journal adm
```

| Setting                                   | Why                                                                                                                  |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `Requires=`, `After=bcd-helper.service`   | The node starts after the helper and stops with it                                                                   |
| `User=bcd`, `Group=bcd`                   | `bcd node` runs unprivileged; systemd hands it the unit's state and runtime directories (`/var/lib/bcd`, `/run/bcd`) |
| `SupplementaryGroups=systemd-journal adm` | Read access to the system journal and logs (`logs.collect` runs `journalctl`)                                        |
| `RuntimeDirectory=bcd-helper`, mode 0755  | `/run/bcd-helper`, root-owned, holds the socket (0660, `root:bcd`)                                                   |

The helper serves a caller only when `SO_PEERCRED` shows `bcd`'s uid and gid (or root) and `/proc/<pid>/exe` is an
installed bcd: `[update] entrypoint`, `bcd-a` or `bcd-b` in `[update] slots_dir`, or the helper's own binary. A node
run from anywhere else is refused.

Self-update works in this mode (2d67b25): the node downloads a release into `<data_dir>/staging`, and the helper
verifies it under its own `[update]` pins and floor, copies it into root's `[update] slots_dir` (default
`/usr/local/lib/bcd`) and swaps; see [self-update through the helper](/docs/daemon#self-update-through-the-helper).
Nodes installed by a script generated with `--release-pins` get the same `[update]` section as a root node.

> **Plugins need root.** 
  `--unprivileged` together with `--plugins` or `--privileged-plugins` is a usage error ("--unprivileged cannot host bcx
  plugins yet: bcd node would need root to start them").

What the helper does not do runs with the `bcd` account's own rights: the journal and the node key file (on a host
with a TPM only the key's record; the key itself is used through the helper's `tpm.*` operations), the status socket, the
`libvirt-kvm` runtime's reads (`virsh -r`, libvirt's read-only socket) and its files under `data_dir/libvirt`, the
`cloud-hypervisor` runtime's files, which it shares with each VMM's user through ACL entries, and reading
`/dev/kmsg`. Defining and running libvirt domains goes through the helper's checked `libvirt.*` operations
(ed3c7d2; see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper)), and `cloud-hypervisor`'s taps,
VFIO groups and VMM units through its `ch.*` operations (7fe74a1; see
[cloud-hypervisor through the helper](/docs/daemon#cloud-hypervisor-through-the-helper)). Only
`bcd node` uses the helper; `bcd site` ignores `[helper]`. `tests/e2e/bcd.test.ts` runs both processes by hand on a
fakehw build, a GPU hold and a signed self-update included; `tests/cli-pki.test.ts` checks the script's section,
account, units and drop-in.

## Joining [#joining]

The join itself is `bcd join`, which the script runs for you:

1. bcd generates a P-256 key and a CSR. On a host with a TPM the key is made in the TPM (50e18f9) and `node.key`
   (0600) under `data_dir` keeps only its record; without one, or when the TPM cannot hold the key, `node.key` is a
   PKCS#8 file. See [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm).
2. It seals the token to the JoinService encryption key and calls `Join` on the gateway over TLS pinned to the
   serving CA, with its hardware claims and the attestor `join-token`. The gateway relays the call; it can neither
   read the token nor forge the answer. A deferral is retried with the same CSR; retryable errors back off.
3. It verifies the signed `JoinResponse` under the pinned signing key, checks that the response is for its own
   CSR, that the leaf certifies its key, the SPIFFE id and the role, and stores the certificate chain and trust
   bundle in `identity.json`. A journal left by an earlier identity is moved aside.

bcd never takes the token from a command-line argument, so a token pasted in the wrong place does not reach shell
history or logs. `bcd join` rejects positional arguments without echoing them:

```bash
$ bcd join bcjt_jt_01...
bcd join: takes no positional arguments
bcd join reads the join token from BCD_JOIN_TOKEN or stdin, never from arguments
```

`--token-file PATH` reads the token from a file instead (c29bb16). The file must be its owner's alone: mode 0600 or
0400 (a Kubernetes Secret volume with `defaultMode: 0400`, whose `..data` symlink bcd follows), a regular file, and
at most 4 KiB; anything else is refused before the token is read. Once the join succeeds, or when the machine had
already joined, bcd deletes the file, since a token is single use; on a read-only mount such as a Secret volume it
cannot, and logs a warning instead of failing. [bcd on Kubernetes](/docs/daemon/kubernetes) joins this way.

The node then opens its session (see [sessions](/docs/daemon/sessions)) and appears as `adopting`. Tokens,
attestors, refusals and certificates are covered in [joining nodes](/docs/guides/nodes/joining).

### Joining again with --reset [#joining-again-with---reset]

`bcd join` on a machine whose identity loads answers `already joined as <node>` and exits 0, whatever state the
certificate is in: expired, revoked, refused at renewal, or naming an AK this TPM no longer has. That keeps a join
unit safe to run on every boot, but it cannot bring such a node back. Since a1c1a46 `--reset` can:

```bash
# as root: see what it would move (moves nothing, exits 2)
bcd join --config /etc/bcd/config.toml --reset
# move the identity aside and join again, here with the TPM
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm
# or with a new join token on standard input
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt
```

| Flag      | Does                                                                                                                                                                                                                                                         |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--reset` | Before the join, moves the node's identity and journal aside: `identity.json`, `node.key`, a pending `node.key.next` and `journal.db`, whichever exist in `data_dir`, each renamed `<name>.reset-<UTC time>` (for example `node.key.reset-20260926T114200Z`) |
| `--yes`   | Confirms `--reset`. Without it bcd prints which files it would move, that the join that follows makes a new key (a new node, unless the attestor proves this machine and reclaims it), and exits 2                                                           |

What it does and does not do:

* **Root only.** Outside development (fakehw) builds a non-root `--reset` fails with "run it as root", exit 1.
* **Nothing is deleted.** The old files stay in `data_dir` under their new names, so an operator can still read
  them (a TPM key's record still names its key). Remove them yourself once the node is back.
* **Then the usual join.** The move happens after the configuration loads and before the join reads any
  credential, so the same run needs the join's credential as any join: a token (`BCD_JOIN_TOKEN`, standard input or
  `--token-file`), `--attestor tpm`, or `--node-id` for an installed OS. If the join then fails, the identity stays
  moved aside and a later plain `bcd join` joins. bcd's own hints spell it as two runs, `bcd join --reset --yes`,
  then `bcd join`. With nothing to move, `--reset` does nothing and the join goes on.
* **Same node or a new one.** An attestor that proves the machine (`tpm` with the same EK, `provisioned`) reclaims
  the node it is bound to. A token join of a machine whose trusted hardware key the old node still holds needs a
  token with `allowReclaim` (see
  [placeholder hardware and reclaims](/docs/guides/nodes/joining#placeholder-hardware-and-reclaims)); to join as a new
  node instead, cordon and delete the old one first (`better-compute nodes delete <node> --yes`; see
  [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host)).
* **The journal moves too** (b4fcc2e). `journal.db` is the old node's, and `bcd node` refuses a journal of another
  node ("the journal belongs to node `<old>`, not `<new>`"). A plain join moves an old journal aside only when the
  old identity still loads, which after `--reset` it does not, so `--reset` moves it with the identity and the node
  starts a new one.
* **Under `[helper]`** the join runs as root and writes the new `node.key` and `identity.json` as root. When it
  finishes, it hands both to the `[helper]` user with `Lchown` (b4fcc2e; root only, and a name swapped for a symlink
  changes only the link), for a new join and for `already joined` alike, so `bcd node` can read its key. Then
  restart `bcd-helper.service` and `bcd.service`; re-running the install script also does the restarts.

`bcd doctor` points at it: the `identity` check fails for an identity that does not load and for an expired
certificate, and both details end with "bcd join --reset --yes, then bcd join". `TestJoinReset` covers the dry run,
the move (the journal included) and that no identity loads afterwards; `TestJoinHandsTheIdentityToTheNode` covers
the hand-over (as root; skipped otherwise).

## Site agents and the ramdisk [#site-agents-and-the-ramdisk]

The install script sets up nodes only: it joins with role `node` and its unit runs `bcd node`. The other two agent
modes use the same binary but are set up differently.

### A site agent [#a-site-agent]

A site agent reads the same `config.toml` and `join-encryption.jwk`, joins with a
[site token](/docs/guides/bare-metal/sites#enroll-site-agents-with-site-tokens) and runs `bcd site`:

```bash
BCD_JOIN_TOKEN=bcjt_... bcd join --role site --config /etc/bcd/config.toml
bcd site --config /etc/bcd/config.toml
```

`bcd site` refuses a data directory whose identity has another role. Without a `[boot]` section it runs BMC
operations, `site.httpProxy` and boot entries but serves no PXE machine (it logs a warning). With one, it also runs
the boot server:

```toml title="/etc/bcd/config.toml (site agent)"
gateway_url = "https://gw.eu.example.com"
site = "site_01k5w..."
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."

[boot]
listen = "0.0.0.0:8080"
url = "http://10.20.0.2:8080"
kernel = "http://10.20.0.2:8080/ramdisk/vmlinuz"
initrd = "http://10.20.0.2:8080/ramdisk/initrd"
cmdline = "console=ttyS0,115200"
server_ip = "10.20.0.2"
proxy_dhcp = true
tftp = ":69"
tftp_root = "/srv/bcd/ipxe"
```

| `[boot]` key        | Meaning                                                                                                                                                                                                         |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `listen`            | The HTTP address of the boot server; empty turns the boot server off. Without `join-encryption.jwk` beside the config `bcd site` exits 1, and without `kernel` and `initrd` the boot server stops with an error |
| `url`               | How machines reach the boot server. iPXE chains to it, and the ramdisks it boots get `<url>/cache` as their image cache                                                                                         |
| `kernel`, `initrd`  | The URLs of the site's ramdisk kernel and initramfs, which iPXE fetches                                                                                                                                         |
| `cmdline`           | Appended to every ramdisk's kernel command line                                                                                                                                                                 |
| `server_ip`         | The agent's IPv4 address on the provisioning network: the proxy DHCP server id and TFTP next-server                                                                                                             |
| `proxy_dhcp`        | `true` answers PXE and UEFI HTTP boot clients on UDP 67 and 4011, beside the site's own DHCP server, which keeps handing out addresses                                                                          |
| `tftp`, `tftp_root` | The TFTP server's UDP address and the directory of iPXE builds it serves: `undionly.kpxe` (BIOS), `ipxe.efi` (x86-64 UEFI), `ipxe-arm64.efi` (arm64 UEFI)                                                       |
| `cache_max_bytes`   | The image cache's byte budget (0, the default, is unbounded): least recently used images are evicted first, never one the `site` section's `cache_images` still names                                           |

`proxy_dhcp` and `tftp` need `server_ip` and `url`, or `bcd site` exits 2. The iPXE builds and the ramdisk kernel and
initramfs are yours to supply: the repository builds neither. Ports 67, 69 and 4011 are privileged, and the unit the
install script writes keeps `bcd` to a capability set without `CAP_NET_BIND_SERVICE`, so a site agent needs its own
unit. What the boot server answers, and when, is on [sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site).

### The ramdisk [#the-ramdisk]

`bcd ramdisk` is not installed. It runs inside the PXE ramdisk the site agent boots, and takes everything from the
kernel command line the site agent rendered into the machine's iPXE script, so it has no `config.toml`:

| Flag         | Default         | Meaning                                                                       |
| ------------ | --------------- | ----------------------------------------------------------------------------- |
| `--cmdline`  | `/proc/cmdline` | Where to read the `bc.*` boot parameters                                      |
| `--data-dir` | `/run/bcd`      | The key, identity and journal, on tmpfs: every ramdisk boot is a new identity |

| Boot parameter   | Meaning                                                                                   |
| ---------------- | ----------------------------------------------------------------------------------------- |
| `bc.gateway`     | The gateway URL (an `http://` one only in fakehw builds)                                  |
| `bc.server_name` | Optional: the name the gateway certificate carries                                        |
| `bc.cache`       | Optional: the site image cache, tried before an image's source                            |
| `bc.site`        | The site id                                                                               |
| `bc.pins`        | The serving CA, join signing and join encryption pins, comma-separated                    |
| `bc.jwk`         | The JoinService encryption key's public JWK, base64url; refused unless it matches its pin |
| `bc.token`       | The machine's single-use, MAC-bound `discovery` join token                                |
| `bc.assertion`   | The site agent's `SignedDiscoveryAssertion`, base64url                                    |

A missing parameter exits 2. The ramdisk joins with the `discovery` attestor as role `ramdisk` (see
[attestation](/docs/guides/nodes/attestation#discovery)), logs JSON, then runs the agent until the machine kexecs
into its installed OS or powers off. Its image needs `bcd` plus the tools its kinds call: `nvme-cli`, `hdparm`,
util-linux (`blkdiscard`, `blockdev`, `lsblk`, `mount`), e2fsprogs' `debugfs` and `mtools` (to write into and read
from a deployed image's ext and FAT filesystems without mounting them; 1ae1789, d9a900f), `udevadm`, `kexec-tools` and, for `diag.dcgm`, `dcgmi`.

## Upgrades [#upgrades]

### Before v0.4: rerun the script [#before-v04-rerun-the-script]

Run the same script, or one generated for a newer `--bcd-version`, without a token on a host that already has
`/etc/bcd/config.toml`. It downloads and verifies the new binary, replaces `/usr/local/bin/bcd` with an atomic
rename, rewrites the key, the unit and the top-level keys of the configuration, and restarts the service:

```bash
sudo sh install-bcd.sh
# bcd upgraded to 0.1.1 and restarted.
```

The configuration's top-level keys (gateway, site and the three pins) come from the new script, and every section
of the existing file (`[plugins]`, `[boot]`, `[imds]`, `[journal]`, `[log]`, ...) is kept as it was (4225772;
earlier scripts rewrote the file whole and dropped them). Top-level keys you added by hand, such as `data_dir` or
`gateway_server_name`, are not kept, so put them back after an upgrade or regenerate the script. The plugin drop-in
follows the kept configuration: a node with `[plugins]` keeps it, even when the new script was generated without
`--plugins`.

The node keeps its identity and journal across the restart. On reconnect its `Hello` carries the new
`daemon_version`, which the gateway records in `node_sessions.daemon_version`.

On a node that has already [self-updated](#self-update), the script's `mv -f` replaces the entrypoint symlink with a
plain file. The next self-update adopts that file into the current slot before it relinks the entrypoint (9006eee),
so it never falls back to the older binary the slot held; before that fix it did, rolling the script's upgrade back.

### Upgrading an unprivileged libvirt host [#upgrading-an-unprivileged-libvirt-host]

A VPS or sandbox host installed with `--unprivileged` by a script from before d418820 runs `bcd node` in the
`libvirt` group, which is root-equivalent, and its libvirt still relabels files. A new bcd there sends domain
operations through the helper anyway (ed3c7d2), which refuses to define a domain with a NIC until
`[helper] instance_bridges` is set, but the group membership and the relabelling stay until you run a
newer script, generated with `--unprivileged` (and `--instance-bridges` for the instances' bridges), on the host
without a token. It rewrites the `unprivileged.conf` drop-in without `SupplementaryGroups=libvirt`, sets
`dynamic_ownership = 0`, adds QEMU's user to the `bcd` group, restarts libvirt and then both bcd units:

```bash
better-compute daemon install-script --pool pool_vps --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.9.2 --sha256 amd64=<hex> --unprivileged --instance-bridges br0 > install-bcd.sh
sudo sh install-bcd.sh
```

Without `[helper] instance_bridges` the helper defines no domain with a NIC, so set the bridges before instances
with networks start there; the script adds them only when the configuration has no `instance_bridges` line yet.
`bcd doctor`'s `libvirt` check fails until the host is set up this way.

### From v0.4: rollouts and self-update [#from-v04-rollouts-and-self-update]

The v0.4 design moves upgrades into desired state. A [rollout](/docs/guides/rollouts) sets
`NodeSpec.daemon = { version, url, sha256, signature }` on nodes wave by wave (1%, 10%, 50%, 100% by default) within
each pool's disruption budget, and a node succeeds when it rejoins reporting that version. A daemon too old to speak
the server's minimum protocol version gets `Welcome.upgrade_required`, and its session accepts only the
`daemon.upgrade` command kind, which carries the same `DaemonSection`; the rollout controller sends it to such nodes.

### Self-update [#self-update]

bcd updates itself in place (de9f380, `daemon/internal/agent/update.go`, `daemon/internal/selfupdate`), for both the
`daemon` section and `daemon.upgrade` (node and site roles). Since 2d67b25 the privileged half (`selfupdate.Manager`:
verify, swap, confirm, roll back) runs in process on a root node and inside `bcd helper` on an
[unprivileged one](#running-bcd-node-unprivileged); see
[self-update through the helper](/docs/daemon#self-update-through-the-helper). The steps are the same:

1. **Only a signed release.** `DaemonSection.signature` is the release's signed manifest (`manifest.json` from
   `better-compute release sign`). At least `[update] release_threshold` distinct keys pinned in `[update]
   release_pins` must have signed it (purpose `release-manifest`), and it must be for product `bcd` and the section's
   version, have a sequence not below the release this node ran or was installed at (`[update] release_sequence`; no
   rollback), be unexpired on both the host clock and server time, and list the bare binary
   `bcd_<version>_<os>_<arch>` with the section's SHA-256. Without release pins bcd takes no binary at all: the
   control plane alone can never hand a node one. The URL must be `https://`.
2. **Two slots.** The binaries live in `bcd-a` and `bcd-b` of the slots directory (`[update] slots_dir`: default
   `<data_dir>/bin`, or `/usr/local/lib/bcd` under the helper), with `current` and `previous` symlinks and
   `slots.json`. The first update adopts the binary the install script put at the entrypoint (`[update]
   entrypoint`, default `/usr/local/bin/bcd`) into slot `a` and makes the entrypoint a symlink to `current`; an
   entrypoint that is a symlink somewhere else is refused, and a plain file found there later (a rerun of the install
   script) is copied into the current slot first (9006eee). Running as root, adoption takes only a root-owned regular
   file that group and others cannot write (9e71b1f). The node downloads the release into
   `<data_dir>/staging/bcd-<version>`, at most 512 MiB, checked against the manifest's size and digest as it streams,
   with an idle watchdog; the swap then copies it into the other slot, checking the size and digest again as it
   writes. It opens the staged file beneath the staging directory's parent (`os.Root`), refuses a symlinked staging
   directory, opens with `O_NOFOLLOW|O_NONBLOCK` (a FIFO cannot hang it), and takes only a regular file with one link;
   under the helper the file must also be owned by the node's uid (c139b9c).
3. **Swap and confirm.** bcd writes `upgrade.json` with a deadline of now plus 120 seconds (fsynced), points
   `current` at the new slot and execs the entrypoint once everything is closed. The new binary confirms itself on
   its first `Welcome`: the release floor moves up (`update.json`) and `upgrade.json` goes. Without a `Welcome` by
   the deadline it swaps back and restarts; the old binary then records the version as failed ("bcd X did not
   start"), reports the `daemon` section `FAILED` with reason `UpdateFailed`, and never tries that version again
   until `bcd update --clear-failed`. Under the helper, the node asks the helper to confirm, and the helper's own
   watchdog also swaps back an update still unconfirmed 30 seconds after its deadline and restarts `bcd.service`.
4. **Crash loops.** Under systemd the first update writes `bcd-rollback.service` and a `bcd.service` drop-in
   (`50-bcd-rollback.conf`: `StartLimitIntervalSec=600`, `StartLimitBurst=5`, `OnFailure=bcd-rollback.service`),
   which restores `previous`, only while `upgrade.json` exists. fakehw builds never write them.
5. **No going back below the floor.** A confirmed update is never undone (c139b9c). `slots.json` records each
   slot's version, manifest sequence and the SHA-256 of the bytes written into it. The binary already in the other
   slot is taken without a manifest only when its recorded sequence is known and not below the release floor, so a
   section that names the older version the node ran before is refused: without a manifest there is nothing to
   verify, and that release's manifest is below the floor (the `daemon` section reports `FAILED`). An update that never confirmed is still swapped back by the deadline,
   the crash-loop unit or the helper's watchdog. To undo a release that confirmed, roll forward: sign a newer release
   with a higher sequence and roll it out ([rollouts](/docs/guides/rollouts#pause-resume-abort)).

`daemon.upgrade` is self-terminating, like `node.reboot`: it reports `RUNNING` before the swap and `SUCCEEDED` from the
new binary once it is welcomed, or `FAILED` with `update-failed` after a swap back; a restart without a reboot is
decided by `upgrade.json`, not the boot id. Running workloads are not touched. `bcd sim` and `bcd ramdisk` run
without self-update: the `daemon` section stays `SectionUnsupported` and the kind `UNSUPPORTED`. `bcd update` shows
the slots, the release floor, a pending update and failed versions from `[update] slots_dir` (run it as root on a
helper node, whose records are root's), and `bcd doctor` warns without release pins, fails when the slots are in use
but the entrypoint does not link to `current`, and warns about a pending update or a failed version (2845f41). Both
find the slots and records the same way (8eca2b1): `[update] slots_dir`, else `/usr/local/lib/bcd` under the
helper, else `<data_dir>/bin` with the records in `data_dir`. `tests/e2e/bcd.test.ts` updates a real bcd to a second signed build through
`daemon.upgrade`, a real exec and its `Welcome` (fc0ca6a), and delivers a signed build to an unprivileged node
through `bcd helper` (2d67b25); no release has been cut, so no real host has self-updated yet. See
[delivery](/docs/daemon/delivery).

The fleet-wide order for any upgrade is: expand migrations, then workers and api, then gateways one at a time
with `Drain`, then daemons in waves, and contract migrations in the next release. See
[version skew](/docs/daemon/protocol#version-skew).

## Removing bcd [#removing-bcd]

Neither the CLI nor bcd has an uninstall step, and PLAN.md designs none. Retire the node in the control plane first
so nothing is scheduled on it (see [operating nodes](/docs/guides/nodes/operating-nodes)). Then stop and disable
`bcd.service` and remove the files listed above.

## Release artifacts [#release-artifacts]

bcd is configured to build with goreleaser from `daemon/` (`.goreleaser.yaml`) for Linux on amd64 and arm64, with
cgo and `zig cc`, set up for a go-nvml backend that would load `libnvidia-ml`; the GPU code in the tree today reads
`nvidia-smi` and needs no cgo (cb987e5). A release produces
`bcd_<version>_linux_{amd64,arm64}.tar.gz`, each with a `.sha256`, and the bare binaries
`bcd_<version>_linux_{amd64,arm64}`. The bare binary is what the install script downloads, what a signed release
manifest lists and what self-update verifies. The same release builds `bcx-ipmi`, the site agent's IPMI BMC
plugin, without cgo and ships it only as `bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz` (348ff56), so a release
still has exactly one bare binary per platform, bcd's, which rollouts and self-update take. goreleaser also writes an SPDX SBOM per archive (syft) and a keyless
cosign signature and certificate for every artifact. `.github/workflows/release.yml` runs it on a `v*` tag (or a
dispatch naming one), appends everything to that tag's GitHub release, and adds SLSA level 3 provenance
(`bcd.intoto.jsonl`) from the generic SLSA generator. No tag has been released yet; CI
(`.github/workflows/daemon.yml`) vets, tests and runs `goreleaser check`, and a local snapshot build is
`goreleaser release --snapshot --clean --skip=sign,sbom`.

Until then a binary comes from a plain `go build` in `daemon/`. The development and test builds are cross-compiled
without cgo (`GOOS=linux GOARCH=amd64 CGO_ENABLED=0`); `node scripts/daemon-test.mjs --build-bcd` writes the fake
hardware build to `daemon/bin/bcd-linux-amd64`. go-nvml is not a dependency, so a cgo build gains nothing today: the default build reads GPUs through `nvidia-smi`
without cgo (cb987e5).


# On Kubernetes (/docs/daemon/kubernetes)

> Run bcd as a privileged DaemonSet on the GPU nodes of a cluster you already operate, so the control plane adopts them without owning their host configuration.



Most machines run bcd the way [the install script](/docs/daemon/install) puts it there: a binary and a systemd unit
on a host the control plane provisions and configures. A cluster you already operate is different. Its GPU nodes
have a driver, a container toolkit and a MIG layout that something else manages, usually the NVIDIA GPU operator,
and you want the control plane to adopt them without taking that over. For that, bcd runs inside the cluster as a
DaemonSet: one privileged pod per GPU node, which joins the node once and then runs `bcd node`.

This is the opposite direction from a [Kubernetes pool](/docs/guides/capacity/kubernetes). There, bcd is installed on
machines the control plane owns, and the `k8s-join` config unit joins them to a cluster as workers. Here the cluster
and its nodes exist first, and bcd runs inside the cluster to report their GPUs and hand the bound ones to pods.

> **Built and dry-run in CI, never run in a cluster.** 
  `tests/provider-kubernetes.test.ts`, `tests/provider-kubernetes-cli.test.ts` and bcd's
  `TestKubernetesDaemonSetContract` check the generated manifests and the `kubernetes manifest` command, and
  `TestHostedManager` runs the hosted device plugins against a fake kubelet. Since ac1ca4b the `container` job of
  `.github/workflows/daemon.yml` builds `daemon/Containerfile`, runs `bcd --version` and `kubectl version --client`
  from the image, renders the manifests (`scripts/k8s-render-manifests.mjs`, with placeholder pins and image digest)
  and has a [kind](https://kind.sigs.k8s.io) cluster's API server accept them: it creates the Namespace, then
  `kubectl apply --dry-run=server` the rest, which checks the schema and Pod Security admission. That is a dry run: no
  pod is scheduled, no image is pulled, and bcd has not run as a DaemonSet against a control plane anywhere yet.

## What you get [#what-you-get]

* **One pod per selected node.** The DaemonSet runs on every node, or on those `--node-selector` names (for example
  `nvidia.com/gpu.present=true`, the label NVIDIA's GPU Feature Discovery sets), tolerates every taint by default,
  runs at `system-node-critical` priority and rolls one node at a time.
* **The host's view, and only what bcd uses of it.** The `bcd` container is privileged, with host networking, and
  mounts exactly the host paths listed in [host mounts](#host-mounts): `/dev`, `/sys`, `/var/lib/bcd`, the CDI
  directories, the kubelet sockets bcd talks to, the kubelet's own credentials, and the container runtime's and
  NVIDIA toolkit's configuration, read-only. Never `/var/lib/kubelet` itself and never the host's `/etc/kubernetes`.
  The pod does not share the host's PID namespace (`hostPID: false`). It runs under the RuntimeClass `nvidia` (the GPU
  operator's; `nvidiaRuntimeClass`), and the `bcd` container asks for `NVIDIA_VISIBLE_DEVICES=all` with
  `NVIDIA_DRIVER_CAPABILITIES=utility` only, so bcd gets `nvidia-smi` and the driver's libraries to read and reset
  GPUs (b6d5548); workload pods still get CDI devices only.
* **No Kubernetes identity of its own.** No service account token: bcd talks to its gateway, and what it does in the
  cluster (its Node's labels, deleting pods, DRA claims and slices) it does by running the image's `kubectl` with the
  kubelet's own credentials, which NodeRestriction confines to its Node (see [API access](#api-access)).
* **One join per node.** The identity lives in the host's `/var/lib/bcd`, so it survives pod restarts and image
  rollouts; every later start finds it and needs no token.
* **No host configuration.** `config.toml` sets `[host] units = false` by default. bcd applies no `NodeSpec.host`
  unit and reports the `host` section `UNSUPPORTED` with reason `HostManagedExternally` (see
  [host configuration](/docs/daemon/host-config)), which the gateway records as an informational node condition. The
  inventory still reports what the node has, so capacity books only the GPUs, MIG instances and huge pages that the
  cluster's own tooling set up.
* **GPUs for pods.** bcd hands the GPUs of `binding: 'kubernetes'` reservations to the kubelet, with no `k8s-join`
  record: see [GPUs for pods](#gpus-for-pods). It gives pods only those GPUs only when bcd is the only source of GPUs
  on the node: see [GPU isolation](#gpu-isolation).
* **No self-update and no reboots.** The configuration has no `[update] release_pins`, so bcd refuses every
  self-update and you upgrade by rolling out a new image; bcd never reboots a node.

## Host mounts [#host-mounts]

The `bcd` container's host paths follow what `daemon/internal/kube` uses and nothing more (27f2c1b, a5eee84,
d11810c). Mounting all of `/var/lib/kubelet` would give a compromised bcd pod every pod's volumes, secrets and service
account tokens and the kubelet's checkpoints; the host's `/etc/kubernetes` holds `admin.conf` and the cluster CA's
keys on a kubeadm control-plane node. `tests/provider-kubernetes.test.ts` asserts the exact set, with and without DRA.

| Host path                                                                                                                                                                                         | Mode       | hostPath type         | Why                                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------- | ---------------------------------------------------------------------------------------------------- |
| `/dev`, `/sys`                                                                                                                                                                                    | read-write | `Directory`           | GPUs, cgroups, device nodes                                                                          |
| `/var/lib/bcd`                                                                                                                                                                                    | read-write | `DirectoryOrCreate`   | The node key, identity and journal, kept across pod restarts                                         |
| `/etc/cdi`                                                                                                                                                                                        | read-write | `DirectoryOrCreate`   | bcd writes each hold's CDI spec                                                                      |
| `/var/run/cdi`                                                                                                                                                                                    | read-only  | `DirectoryOrCreate`   | bcd only reads NVIDIA's generated spec there                                                         |
| `/var/lib/kubelet/device-plugins`                                                                                                                                                                 | read-write | `Directory`           | bcd's plugin sockets, and the kubelet's registration socket it dials                                 |
| `/var/lib/kubelet/pod-resources`                                                                                                                                                                  | read-only  | `Directory`           | bcd only connects to the socket; a read-only bind mount still allows `connect()`                     |
| `/var/lib/kubelet/plugins_registry`, `/var/lib/kubelet/plugins/gpu.bcd.dev`                                                                                                                       | read-write | `DirectoryOrCreate`   | Only with `gpuApi: 'dra'`: the DRA registration and driver sockets, and the driver's `prepared.json` |
| The kubelet's kubeconfig: `/etc/kubernetes/kubelet.conf` (kubeadm, the default), `/var/lib/rancher/k3s/agent/kubelet.kubeconfig` (k3s) or `/var/lib/rancher/rke2/agent/kubelet.kubeconfig` (rke2) | read-only  | `File`                | Mounted where bcd reads it, `/etc/kubernetes/kubelet.conf` (see [API access](#api-access))           |
| The kubelet's certificate directories: `/var/lib/kubelet/pki` (kubeadm), `/var/lib/rancher/k3s/agent` (k3s) or `/var/lib/rancher/rke2/agent` (rke2)                                               | read-only  | `Directory`           | Each at its own path, so the paths inside the kubeconfig resolve                                     |
| The container runtime's configuration, default `/etc/containerd/config.toml`                                                                                                                      | read-only  | `File` or `Directory` | bcd reads which NVIDIA runtime the container runtime runs (see [GPU isolation](#gpu-isolation))      |
| The NVIDIA toolkit's configuration, default `/usr/local/nvidia/toolkit/.config`                                                                                                                   | read-only  | `Directory`           | bcd reads whether the toolkit accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods                |

Besides these, the pod has the ConfigMap's `config.toml`, a memory-backed `emptyDir` at `/run/bcd` and an `emptyDir`
at `/tmp`. The container runtime's and toolkit's configuration are never created: a missing path stays missing, so the
pod does not start on a node that lacks it rather than seeing an empty directory, and hosted bcd treats a runtime
configuration it cannot read as `ForeignGpuExposure`. The kubelet credentials refuse a certificate directory that holds
more than the kubelet's own: `/etc`, `/etc/kubernetes`, `/etc/kubernetes/pki`, `/var/lib/kubelet` or anything under it
but `pki`, `/var/lib`, `/var` and `/`. Which paths these are is set with the options in
[generate the manifests from TypeScript](#generate-the-manifests-from-typescript), or with `--kubelet-credentials`,
`--container-runtime` and `--nvidia-toolkit`.

## Deploy it [#deploy-it]

Since 3341e50 there is one path: the manifests are rendered for your deployment by the CLI, with
`@better-compute/provider-kubernetes`'s `bcdDaemonSetManifests`, so they cannot drift from what bcd and the control
plane expect. The hand-kept manifests that `deploy/kubernetes` used to carry are gone; its README walks the same steps.

      ### Build the image [#build-the-image]

    `daemon/Containerfile` builds the cgo-free host build (`-tags nonvml`; GPUs through `nvidia-smi`) on
    `debian:bookworm-slim` with `ca-certificates`, `nftables`, `kmod`, `pciutils` and `util-linux`. Build and push it,
    and note its digest:

    ```bash
    docker build -f daemon/Containerfile -t registry.example.com/better-compute/bcd:0.1.0 \
      --build-arg VERSION=0.1.0 daemon
    docker push registry.example.com/better-compute/bcd:0.1.0
    ```

    `RELEASE_SEQUENCE` is a second build argument (default 0). No published image exists; build your own. Since 9bdd43a
    the image also carries `kubectl` v1.34.1 at `/usr/local/bin/kubectl`, which bcd runs for its Node's labels, for
    deleting pods of released devices and for DRA. The build downloads it from `dl.k8s.io` for `TARGETARCH` (default
    `amd64`; `arm64` too) and checks it against the pinned SHA-256 of that release; a mismatch, or another architecture,
    fails the build. bcd runs it as the kubelet (see [API access](#api-access)). CI builds this image for `linux/amd64` on
    every change to `daemon/**`, `proto/**` or `packages/provider-kubernetes/**` (the `container` job, ac1ca4b), but
    publishes nothing.
  
      ### Add the command [#add-the-command]

    The CLI never imports providers, so the application adds `kubernetes manifest` (a2eb28f) to its configuration:

    ```ts title="better-compute.config.ts"
    import { kubernetesManifestCommand } from '@better-compute/provider-kubernetes';

    export const commands = [kubernetesManifestCommand()];
    ```

    An application may pass `kubernetesManifestCommand({ pins })` with the trust anchors from its own PKI bundle
    (`readPkiPins` of `@better-compute/pki`); otherwise the pins are flags.
  
      ### Render the manifests and the join token [#render-the-manifests-and-the-join-token]

    ```bash
    better-compute kubernetes manifest --tenant root --pool pool_01k… \
      --image registry.example.com/better-compute/bcd@sha256:<64 hex digits> \
      --serving-ca-pin sha256/… --join-signing-pin sha256/… --join-encryption-pin sha256/… \
      --max-uses 8 --ttl 1h \
      --secret-out bcd-join-secret.json --out bcd.json
    ```

    The three pins are trust anchors, and they come only from you (d2cd813): the three flags together (all or none),
    from `better-compute pki show` on the control-plane host, or the application's `pins`. A run without them is refused
    before any call. The command never pins what the network says: from the control plane's `/.well-known/better-compute`
    (at `--control-plane`, which must be https, else the configured deployment in process) it takes only data that
    carries no trust, the gateway's URL and name and the join key, and it refuses a join key that does not match
    `--join-encryption-pin` before a token is minted.

    It renders the manifests first, so a bad image or pin fails before any token exists: a Namespace (`bcd-system`,
    labelled `pod-security.kubernetes.io/enforce: privileged`), a ConfigMap with `config.toml`, and the DaemonSet.
    Then it mints a multi-use join token with `hosted: 'kubernetes'` and writes it only as the join Secret's manifest,
    to `--secret-out`, a new file created 0600, never the output. The call is decided as `compute:join-tokens:create`.
    After writing the Secret, the command reminds you (ab9339b) to revoke the token (`join-tokens revoke <id>`) and
    delete the Secret once every node has joined, with a fresh token for a later batch of nodes, and prints the
    [GPU isolation](#gpu-isolation) requirements.

    | Flag                                                              | Default                                                                 |                                                                                                                                                                                                                                                                                                                                                                                                     |
    | ----------------------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `--tenant`                                                        | `BETTER_COMPUTE_TENANT`                                                 | The tenant that owns the pool (required)                                                                                                                                                                                                                                                                                                                                                            |
    | `--pool`                                                          | required                                                                | The pool the cluster's nodes join                                                                                                                                                                                                                                                                                                                                                                   |
    | `--image`                                                         | required                                                                | The bcd image, pinned by digest (`registry/repo[:tag]@sha256:<64 hex>`)                                                                                                                                                                                                                                                                                                                             |
    | `--serving-ca-pin`, `--join-signing-pin`, `--join-encryption-pin` | `BCD_SERVING_CA_PIN`, `BCD_JOIN_SIGNING_PIN`, `BCD_JOIN_ENCRYPTION_PIN` | The gateway serving CA's and the JoinService keys' `sha256/…` pins, from the PKI bundle; all three, unless the application passes `pins`                                                                                                                                                                                                                                                            |
    | `--secret-out`                                                    | required                                                                | A new file (0600) for the join token Secret manifest                                                                                                                                                                                                                                                                                                                                                |
    | `--out`                                                           | the output                                                              | A new file for the manifests                                                                                                                                                                                                                                                                                                                                                                        |
    | `--control-plane`                                                 | the configuration                                                       | The API's public https URL, where the well-known document is served: the origin for `better-compute serve`, or the origin and the API's base path when compute is mounted inside an application                                                                                                                                                                                                     |
    | `--namespace`, `--name`                                           | `bcd-system`, `bcd`                                                     |                                                                                                                                                                                                                                                                                                                                                                                                     |
    | `--node-selector`                                                 | every node                                                              | `key=value` pairs, such as `nvidia.com/gpu.present=true`                                                                                                                                                                                                                                                                                                                                            |
    | `--host-units`                                                    | off                                                                     | Let bcd apply `NodeSpec.host` units                                                                                                                                                                                                                                                                                                                                                                 |
    | `--gpu-api`                                                       | `device-plugin`                                                         | `dra` for the DRA driver (see [GPUs for pods](#gpus-for-pods))                                                                                                                                                                                                                                                                                                                                      |
    | `--container-runtime`                                             | `containerd`                                                            | `containerd` (`/etc/containerd/config.toml`), `k3s` (`/var/lib/rancher/k3s/agent/etc/containerd/config.toml`), `rke2` (`/var/lib/rancher/rke2/agent/etc/containerd/config.toml`) or `cri-o` (`/etc/crio/crio.conf` and `/etc/crio/crio.conf.d`): the nodes' container runtime, whose configuration is mounted read-only for the [GPU isolation](#gpu-isolation) check (`containerRuntime`; 3775f46) |
    | `--kubelet-credentials`                                           | `kubeadm`                                                               | `kubeadm` (`/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki`), `k3s` (`/var/lib/rancher/k3s/agent/kubelet.kubeconfig` and `/var/lib/rancher/k3s/agent`) or `rke2` (the same under `/var/lib/rancher/rke2/agent`): the kubelet's kubeconfig and client certificates, mounted read-only, which bcd acts with (`kubeletCredentials`; 3d6f79a, a96b238)                                         |
    | `--nvidia-toolkit`                                                | `gpu-operator`                                                          | `gpu-operator` (`/usr/local/nvidia/toolkit/.config`) or `packages` (`/etc/nvidia-container-runtime`): where the NVIDIA container toolkit's configuration lives, mounted read-only for the same check (`nvidiaToolkit`; 3775f46)                                                                                                                                                                     |
    | `--nvidia-runtime-class`                                          | `nvidia`                                                                | The RuntimeClass bcd's own container runs under, which gives it `nvidia-smi` and the driver's libraries (`nvidiaRuntimeClass`; ab9339b); `none` for no RuntimeClass, when the image carries them                                                                                                                                                                                                    |
    | `--ttl`                                                           | `24h`                                                                   | How long the token admits nodes, 1 minute to 24 hours                                                                                                                                                                                                                                                                                                                                               |
    | `--max-uses`                                                      | `100`                                                                   | How many nodes it admits                                                                                                                                                                                                                                                                                                                                                                            |
    | `--key`                                                           | a new UUID                                                              | The token's idempotency key, to retry the same request                                                                                                                                                                                                                                                                                                                                              |

    **Distribution, runtime and toolkit.** Pick `--container-runtime` and `--kubelet-credentials` for the distribution the
    nodes run (`k3s` or `rke2` for those; `cri-o` for CRI-O, else `containerd`, with `kubeadm` credentials), and
    `--nvidia-toolkit packages` when the toolkit comes from NVIDIA's distribution packages rather than the GPU operator. Every path must exist on every selected node: the mounts are never
    created, so a wrong choice leaves the pod unable to start, and hosted bcd withdraws every GPU when it cannot read the
    runtime's configuration. Other paths (containerd's `conf.d`, another `toolkit.installDir`) need
    `bcdDaemonSetManifests` with `{ files, dirs }`, and so do kubelet credentials outside the three presets
    (`kubeletCredentials: { kubeconfig, certificateDirs }`).

    **Token hygiene** (d6f8549). A DaemonSet cannot use one single-use token per node, so the token is multi-use, and
    anyone who reads the Secret while it is live can join a machine to the pool. Give it `--max-uses` equal to the node
    count and a short `--ttl` (an hour covers a rollout).
  
      ### Apply [#apply]

    Configure the GPU stack as [GPU isolation](#gpu-isolation) requires, then apply the Secret first and the manifests
    after:

    ```bash
    kubectl apply -f bcd-join-secret.json
    kubectl apply -f bcd.json
    ```

    On each node the `join` init container runs `bcd join` with the token Secret mounted read-only (mode 0400), as root
    with no capabilities, no privilege escalation and a read-only root filesystem; on a node that joined before it
    succeeds without spending a use. `bcd join --token-file` refuses a token file that group or others can read, one
    that is not a regular file (a Secret volume's symlink into `..data` is followed) and one over 4 KiB (c29bb16). The
    `bcd` container then runs `bcd node`, and never sees the token. The nodes appear in
    [`nodes.list`](/docs/reference/api/nodes#list) as `adopting`, as any joined machine does (see
    [joining nodes](/docs/guides/nodes/joining)).
  
      ### Revoke the token and delete the Secret [#revoke-the-token-and-delete-the-secret]

    Once every node has joined, revoke the token ([`joinTokens.revoke`](/docs/reference/api/join-tokens#revoke)) and delete
    the Secret:

    ```bash
    kubectl -n bcd-system delete secret <name>
    ```

    The pods keep running and restarting without it, since a joined node keeps its identity in `/var/lib/bcd`. Since
    27f2c1b the join-token Secret volume is `optional: true`, so a pod of a joined node that restarts or is rescheduled
    after the Secret is gone starts instead of waiting in `ContainerCreating`: its `bcd join` finds the identity, exits 0
    without a token file and spends nothing. For nodes added later, mint a new token and Secret, and remove both again.
  
## Trust [#trust]

The cluster's admins have root on its machines: anyone who may run a privileged pod there can read what bcd holds.
So a node that joins with a hosted token lands `tenant-controlled`, with a `node-tenant` certificate that never
receives platform secrets, unless the pool's `policy.kubernetes.platformOperated` is `true`, which says the platform
itself operates the cluster (a27156a). Since 259d82c that flag counts only on a pool the platform owns: the pool's
owner must be the owner of its site, the site must not be `custom`, and, when the deployment names a root tenant, the
owner must be it. A tenant's pool on a platform site cannot vouch for its own cluster by editing its policy. The token
decides this, never what the node claims, and it only ever lowers trust: a node on a tenant's own `custom` site stays
`tenant-owned`, and a reclaim through a hosted token lowers an existing `platform` node the same way. Replaying
`joinTokens.create` with the same key and another `hosted` value is `CONFLICT`.

A `tenant-controlled` node adds no capacity: placement excludes it, as for a
[dedicated-root lease](/docs/guides/bare-metal/dedicated-root). For the platform's own clusters, give the pool
`policy.kubernetes: { platformOperated: true }`, with none of `apiServer`, `caCertSha256` and `joinTokenRef`: those
three together mean the machines should join a cluster through the `k8s-join` unit, and a partial set is
`INVALID_INPUT` (7db110a).

### Host paths and Pod Security [#host-paths-and-pod-security]

bcd keeps its node key and journal in the hostPath `/var/lib/bcd` (files 0600, owned by root). Any pod allowed to
mount host paths as root can read them, and with the key it can act as the node. So restrict host paths for
everything else (d6f8549): label workload namespaces with Pod Security Admission `baseline` or `restricted`, which
both forbid hostPath and privileged pods. Only `bcd-system`, which the manifests label
`pod-security.kubernetes.io/enforce: privileged`, and your platform's own system namespaces should run privileged.
bcd's pod runs without `hostPID` and with no service account token.

### API access [#api-access]

bcd reaches the Kubernetes API only by running the image's pinned `kubectl` as the kubelet: every call passes
`--kubeconfig /etc/kubernetes/kubelet.conf`. Since 27f2c1b the DaemonSet mounts only that file there (read-only,
hostPath type `File`, from the kubelet's own kubeconfig, which `--kubelet-credentials` or `kubeletCredentials` names),
and the kubelet's client-certificate directories read-only at their own paths (default `/var/lib/kubelet/pki`), never the host's
whole `/etc/kubernetes` (see [host mounts](#host-mounts)). The pod has `automountServiceAccountToken: false`
and the manifests create no RBAC. That is deliberate (e7d0765, which reverted 9bdd43a's service-account credentials):
RBAC cannot be scoped to one node, so a service account would have given every privileged bcd pod cluster-wide
`patch` on Nodes and `delete` on pods, while the NodeRestriction admission plugin confines the kubelet's identity, and
so bcd, to its own Node and that Node's pods. What bcd does with it:

| Work                                                    | `kubectl`                                                                                                          |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Its Node's `bcd.dev/*` labels                           | `get node`, `label node --overwrite`                                                                               |
| The reaper                                              | `delete pod --wait=false --grace-period=10`, forced a minute later                                                 |
| DRA (`gpu_api = "dra"`)                                 | `api-versions`, `get resourceclaim`, `apply --server-side --field-manager=bcd` and `delete` of its `ResourceSlice` |
| Other GPU DRA drivers ([GPU isolation](#gpu-isolation)) | `get resourceslices --field-selector spec.nodeName=<node>`, at most once a minute (f88bcdb)                        |

## GPU isolation [#gpu-isolation]

bcd hands a pod exactly the GPUs its reservation holds, and it can enforce that only when nothing else hands out GPUs
on the node (d6f8549). With the NVIDIA GPU operator its defaults are unsafe here: its own device plugin advertises
every GPU, and its container runtime gives an unprivileged pod any GPU it names in `NVIDIA_VISIBLE_DEVICES`, so any
pod could take every GPU. Required:

| Setting                                                                                                                                                                                                        | Why                                                                                                                                                                                  |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `devicePlugin.enabled=false` in the GPU operator                                                                                                                                                               | bcd's device plugins (or its DRA driver) are the only source of GPUs for the kubelet                                                                                                 |
| `accept-nvidia-visible-devices-envvar-when-unprivileged = false` in the NVIDIA container toolkit's `config.toml` (the operator's `toolkit` env `ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false`) | Devices arrive by CDI only                                                                                                                                                           |
| CDI specs generated with `--device-name-strategy=uuid`                                                                                                                                                         | bcd's device plugin then answers `nvidia.com/gpu=<uuid>`; without that spec it answers `bcd.dev/gpu=<uuid>` plus `NVIDIA_VISIBLE_DEVICES`, which is safe only with the setting above |

The generated DaemonSet carries these requirements, and the Pod Security and token rules above, in its
`bcd.dev/requires` annotation (`gpuIsolationRequirements`), since JSON manifests have no comments.

bcd enforces the first two itself (9f860ad, widened in f88bcdb). It offers no GPUs while any of these holds: its
device plugins stop, its DRA slice is deleted, and it reports the condition `ForeignGpuExposure` naming what to close.

* **Another device plugin.** The kubelet can also allocate an `nvidia.com/*` resource (pod-resources' allocatable
  resources).
* **Another GPU DRA driver.** The node's ResourceSlices, read with the kubelet's credentials at most once a minute,
  show a driver other than `gpu.bcd.dev` whose name marks it as a GPU driver (for example `gpu.nvidia.com`) offering
  devices. An API server without `resource.k8s.io`, or one that refuses the list, is no evidence either way.
* **The toolkit the container runtime runs.** bcd reads the container runtime's configuration (containerd's
  `/etc/containerd/config.toml`, k3s's and RKE2's under `/var/lib/rancher/<k3s|rke2>/agent/etc/containerd/`, CRI-O's
  `/etc/crio/crio.conf`, and the `.toml` and `.conf` files of `/etc/containerd/conf.d`, `/etc/containerd/config.d` and
  `/etc/crio/crio.conf.d`) for a `BinaryName` or `runtime_path` that runs `nvidia-container-runtime` (the `.cdi`
  variant, which takes CDI names only, excepted). For each, it derives the configuration that binary reads: the GPU
  operator's `<installDir>/toolkit/.config/nvidia-container-runtime/config.toml` when the binary sits in a `toolkit`
  directory, else `/etc/nvidia-container-runtime/config.toml`. That file must exist and set
  `accept-nvidia-visible-devices-envvar-when-unprivileged = false`: missing or unreadable, the toolkit takes its
  default and accepts `NVIDIA_VISIBLE_DEVICES` from any pod, so that is exposure too.
* **Any known toolkit configuration.** `/etc/nvidia-container-runtime/config.toml` or the operator's
  `/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml`, when present, accepts the variable (the key
  `true`, or absent, the toolkit's default).
* **Hosted, no readable runtime configuration.** On a DaemonSet node bcd cannot rule the toolkit out when it can read
  none of the container runtime's configuration files, so it counts that as exposure. That is why the manifests mount
  them read-only as `File` or `Directory` and never `DirectoryOrCreate`, which would make a missing directory look
  empty (d11810c; see [host mounts](#host-mounts)).

The gateway records the condition as a node condition (b91dcff). Since ebd9806 it is NoSchedule while true (advisory
only on a tenant-controlled node), and the node's active reservations turn `Degraded` with reason `ForeignGpuExposure`
(4570dfe), since their GPUs are no longer isolated. So a cluster with the GPU operator's defaults gets no GPUs
through bcd, and its nodes take no new work, until you apply the settings above.

`bcd doctor`'s `gpu-isolation` check (35534e5) runs the same checks. Since f88bcdb it also runs on a DaemonSet node
(`[host] units = false`): it fails when no container runtime configuration is readable and when a toolkit
configuration lets unprivileged pods pick GPUs. On a node bcd joined with `k8s-join` it also fails on another GPU DRA
driver in the node's ResourceSlices while the kubelet is up.

## GPUs for pods [#gpus-for-pods]

With `[host] units = false` the node is a worker because the cluster runs bcd there, not because `k8s-join` joined
it, so bcd's Kubernetes side runs in hosted mode (1e5f96b). It needs no `k8s-join` record: the joined-mode files in
`/var/lib/bcd-hostconfig` are not read here.

* **Device plugins.** bcd runs its kubelet device plugins whenever
  `/var/lib/kubelet/device-plugins/kubelet.sock` exists, advertising the devices of the node's
  `binding: 'kubernetes'` holds as `bcd.dev/gpu`, `bcd.dev/gpu.mig` and `bcd.dev/gpu.slot`, as on a joined worker
  (see [the device plugin](/docs/guides/capacity/kubernetes#the-device-plugin)).
* **Labels.** bcd keeps its Node's `bcd.dev/*` labels itself (`bcd.dev/node-id`, `bcd.dev/site`, `bcd.dev/gpu.count`,
  `bcd.dev/gpu.product`, `bcd.dev/gpu.driver`), naming the Node from `NODE_NAME` (the DaemonSet's `spec.nodeName`
  field reference) and using the kubelet's kubeconfig, `/etc/kubernetes/kubelet.conf` (see
  [API access](#api-access)). It updates them at most once
  a minute, touches only `bcd.dev/*`, sets no taints, and removes `bcd.dev/*` labels it no longer sets.
* **The reaper.** Pods that keep a released device are deleted after 30 seconds with a 10-second grace period, and
  forced a minute later if they still hold it, as on a joined worker, DaemonSet pods included (8bcbe92), through the
  kubelet's pod-resources socket in the read-only `/var/lib/kubelet/pod-resources` mount; without that socket bcd
  reports `PodResourcesUnavailable`, which is NoSchedule for the whole node while true (ebd9806; advisory on a
  tenant-controlled node). A released allocation is reported to the control plane only once nothing holds its devices
  (cd5fc82, 8bcbe92), and one still held after 5 minutes raises `KubernetesReleaseStuck`, also NoSchedule while true
  (see [release](/docs/guides/capacity/kubernetes#release)). A hosted node always counts as a worker there, so a
  kubelet bcd cannot ask means the devices stay in use. The process backstop a joined worker has (a pod process with a
  released GPU's device node open) is off here: without `hostPID` bcd's `/proc` is its own pod's, where its own
  `nvidia-smi` would look like a holder (f88bcdb). A hosted node relies on pod-resources and, with DRA, the driver's
  `prepared.json` record in the `/var/lib/kubelet/plugins/gpu.bcd.dev` mount. The same holds for clearing an
  unreadable `releasing.json`, which keeps every GPU claimed and raises `KubernetesReleasesUnreadable` (NoSchedule while true,
  4884a65) until no pod may hold a GPU of an allocation the node does not know (a18f423, b01130f; see
  [an unreadable releases file](/docs/guides/capacity/kubernetes#an-unreadable-releases-file)).
* **Device plugin or DRA.** `[kubernetes] gpu_api` in `config.toml` chooses how the GPUs reach pods:
  `"device-plugin"` (the default) or `"dra"`, bcd's DRA driver `gpu.bcd.dev` (7d24836), never both. Any other value
  stops bcd from starting. `--gpu-api` and `bcdDaemonSetManifests({ gpuApi })` write it (e4b0c01). See
  [device plugin or DRA](/docs/guides/capacity/kubernetes#device-plugin-or-dra) for what the driver publishes.

All of these run the `kubectl` the image carries, as the kubelet (see [API access](#api-access)).

## Generate the manifests from TypeScript [#generate-the-manifests-from-typescript]

`kubernetes manifest` calls `bcdDaemonSetManifests` (2c9f953). An application can call it itself; it is not a VPS
provider and creates no machines, it produces manifests you apply, and it mints no token.

```ts title="bcd-k8s.ts"
import { writeFileSync } from 'node:fs';
import { bcdDaemonSetManifests, renderKubernetesManifests } from '@better-compute/provider-kubernetes';

const manifests = bcdDaemonSetManifests({
  image: 'registry.example.com/better-compute/bcd@sha256:<64 hex digits>',
  gatewayUrl: 'https://gw.eu-central.example.com',
  siteId: 'site_01k…',
  pins: { servingCaPin, joinSigningPin, joinEncryptionPin }, // from your PKI bundle
  controlPlaneUrl: 'https://compute.example.com', // where /.well-known/better-compute is served, or joinEncryptionJwk
  joinTokenSecret: { name: 'bcd-join' },
  nodeSelector: { 'nvidia.com/gpu.present': 'true' },
});
writeFileSync('bcd.json', renderKubernetesManifests(manifests)); // kubectl apply -f bcd.json
```

`bcdDaemonSetManifests` returns a Namespace, a ConfigMap (`config.toml`, and the pinned JoinService key when you pass
`joinEncryptionJwk`) and a DaemonSet; `renderKubernetesManifests` turns them into one `v1` `List` for
`kubectl apply -f`. It refuses (`INVALID_CONFIG`) an image not pinned by digest, a gateway or control-plane URL that
is not https, malformed pins, names or resource quantities, a `gpuApi` other than `device-plugin` or `dra`, a JWK
with a private part, and host paths that are not absolute or that [host mounts](#host-mounts) rules out.

| Option                                   | Default                                                                |                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ---------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image`                                  | required                                                               | `registry/repo[:tag]@sha256:<64 hex>`                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `gatewayUrl`, `siteId`, `pins`           | required                                                               | As the pool's site and the PKI bundle give them                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `joinEncryptionJwk` or `controlPlaneUrl` | one required                                                           | The JoinService encryption key, or where `bcd join --server` fetches it; pinned either way                                                                                                                                                                                                                                                                                                                                                                                    |
| `joinTokenSecret`                        | required                                                               | `{ name, key = 'token' }`, a Secret with a `hosted: 'kubernetes'` token                                                                                                                                                                                                                                                                                                                                                                                                       |
| `namespace`, `name`, `createNamespace`   | `bcd-system`, `bcd`, `true`                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `nodeSelector`, `tolerations`            | every node, every taint                                                |                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `priorityClassName`                      | `system-node-critical`                                                 | `null` for none                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `imagePullSecrets`, `resources`          | none; requests 100m and 128Mi, limit 512Mi                             |                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `hostUnits`                              | `false`                                                                | `true` writes no `[host] units = false`, so the pool's host units apply                                                                                                                                                                                                                                                                                                                                                                                                       |
| `gpuApi`                                 | none (bcd's default, `device-plugin`)                                  | `'dra'` writes `[kubernetes] gpu_api = "dra"` (e4b0c01)                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `nvidiaRuntimeClass`                     | `nvidia`                                                               | The RuntimeClass that gives bcd's container the driver's user space; `null` when the image carries it (b6d5548)                                                                                                                                                                                                                                                                                                                                                               |
| `kubeletCredentials`                     | `'kubeadm'`: `/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki` | A preset (3d6f79a; `kubeletCredentialPresets`): `'k3s'` (`/var/lib/rancher/k3s/agent/kubelet.kubeconfig`, `/var/lib/rancher/k3s/agent`) or `'rke2'` (the same under `/var/lib/rancher/rke2/agent`), or `{ kubeconfig, certificateDirs }` with 1 to 4 directories. The kubeconfig is mounted read-only at `/etc/kubernetes/kubelet.conf`, each directory at its own path; a directory holding more than the kubelet's certificates, or an unknown preset, is refused (27f2c1b) |
| `containerRuntime`                       | `'containerd'`                                                         | `'k3s'`, `'rke2'`, `'cri-o'`, or `{ files, dirs }` (1 to 8 absolute paths in all): the runtime configuration mounted read-only (`containerRuntimeConfigs`; d11810c)                                                                                                                                                                                                                                                                                                           |
| `nvidiaToolkit`                          | `'gpu-operator'`                                                       | `'packages'`, or `{ files, dirs }` for another `toolkit.installDir`: the toolkit configuration mounted read-only (`nvidiaToolkitConfigs`; d11810c). A path both options name is refused                                                                                                                                                                                                                                                                                       |
| `gatewayServerName`, `logLevel`          | none, `info`                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

The pod's `bcd` container gets `NODE_NAME` and has `bcd status` as its liveness probe, and the pod rolls when
`config.toml` changes (annotation `bcd.dev/config-sha256`). Both sides check the contract:
`scripts/daemon-k8s-contract.mjs` writes the generated pod to `daemon/cmd/bcd/testdata/k8s-daemonset.json`, and bcd's
`TestKubernetesDaemonSetContract` parses its `config.toml` (the `gpu_api` key included) and command lines with bcd's
own configuration parser and flag sets.

## Upgrades [#upgrades]

Render the manifests again with the new image digest and apply them. The update strategy replaces one pod at a time.
The identity and the journal are on the host, so a new pod resumes where the old one stopped. Rollouts and
`daemon.upgrade` do not apply to these nodes, since bcd takes no self-update without release pins.

## Limits [#limits]

* **No VPS instances.** The image carries neither `virsh` nor `cloud-hypervisor`, so these nodes run no instance
  runtime.
* **No reboots.** `node.reboot` runs `systemctl reboot`, which the image does not have.
* **The pod sees its own filesystem.** Only the mounted host paths are the host's; that is why the host converger is
  off by default.
* **Host resources are the kubelet's.** A `binding: 'kubernetes'` hold gets no cgroup slice: the kubelet enforces its
  pods' requests and limits (e01d7ff). See [Kubernetes workers](/docs/guides/capacity/kubernetes#host-resources-and-pods).

  - [Kubernetes workers](/docs/guides/capacity/kubernetes): Reservations with binding kubernetes, the device plugin and DRA.

  - [Joining nodes](/docs/guides/nodes/joining): Join tokens, hosted tokens and trust.

  - [Installing bcd](/docs/daemon/install): config.toml and the files on a host.


# Protocol (/docs/daemon/protocol)

> Transport, the proto packages, the AgentGateway service and its frames, signed envelopes, message size bounds and the version-skew rules.



Every bcd mode (node, site agent, ramdisk) speaks one protocol to the control plane: `agent/v1`, a gRPC service
defined in `proto/bettercompute/agent/v1`. The daemon always dials out, the connection carries one long-lived
bidirectional stream of frames, and everything security-relevant inside it is either authenticated by TLS or
signed end to end, so the gateway in the middle can relay but not forge.

This page explains how the protocol behaves. For every service, message, field number and enum value, see the
generated [protocol reference](/docs/reference/protocol), built from the `.proto` files with each declaration's own
comment; the message names below link to their entries there.

## Transport [#transport]

<Mermaid
  chart="flowchart LR
  D[&#x22;bcd&#x22;] -->|&#x22;TLS to gw.region:443&#x22;| LB[&#x22;L4 load balancer, TLS passthrough&#x22;]
  LB --> G[&#x22;gateway: TLS terminates here&#x22;]"
/>

* **Endpoint.** The daemon dials `gw.<region>:443` (the `gateway_url` of its config). An L4 load balancer passes
  TLS through, so TLS terminates in the gateway process and the gateway sees the client certificate.
* **One HTTP/2 connection** carries the `Session` stream and, from v0.3, on-demand `Tunnel` streams. The gateway
  sends an HTTP/2 PING every 20 seconds.
* **Certificates.** The gateway's serving certificate comes from a separate gateway serving CA
  (`better-compute pki issue-gateway --san gw.<region>`), and bcd pins that CA's SPKI hash. Nodes present client
  certificates from the per-region node CA, which signs client certificates only. `Join` is server-authenticated
  TLS (the node has no certificate yet); `Renew`, `Session` and `Tunnel` are mutual TLS.
* **gRPC framing.** `serveGateway` implements gRPC on `node:http2` without a framework: `POST` with an
  `application/grpc` or `application/grpc+proto` content type, 5-byte length-prefixed messages, `grpc-status`
  and a percent-encoded `grpc-message` in trailers (trailers-only for immediate errors), no compression.
  `grpc-timeout` is tolerated and ignored, because sessions are long-lived by design. Other methods get HTTP 405,
  other content types 415.
* **Daemon side.** bcd (`daemon/internal/transport`) uses connect-go over the Go code generated into `daemon/gen/`
  (protoc-gen-go and protoc-gen-connect-go, through `daemon/buf.gen.yaml` and `node scripts/daemon-proto-gen.mjs`;
  `--check` verifies it is current). It speaks gRPC over HTTP/2 with TLS 1.3 pinned to the serving CA's SPKI hash,
  never the system roots, presents its node certificate, and sends HTTP/2 PINGs every 20 seconds. When
  `gateway_url` names an IP address, `gateway_server_name` gives the name the gateway certificate must carry.
  `tests/e2e/bcd.test.ts` runs it against `serveGateway`'s mTLS listener. The FakeDaemon in
  `@better-compute/testing` speaks the same protocol over real HTTP/2 or in memory.

### WebSocket fallback (v0.4) [#websocket-fallback-v04]

Some networks run a TLS-inspecting proxy that strips client certificates. For them the gateway also serves the same
frames over a WebSocket:

| Item          | Value                                                                                                                                                                                                                                                                                                 |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL           | `wss://<gateway>/v1/session`, `GET` with `Upgrade: websocket`, version 13                                                                                                                                                                                                                             |
| Subprotocol   | `bc.agent.v1`                                                                                                                                                                                                                                                                                         |
| Framing       | The same length-prefixed `AgentFrame` and `ServerFrame` messages as the gRPC `Session`                                                                                                                                                                                                                |
| Daemon proof  | `x-bc-nonce` (22 to 128 base64url characters) and `x-bc-proof`: an ES256 JWT of type `bc-agent-proof+jwt`, signed by the node key, with the node chain in `x5c`, `sub` the node id, `aud` the gateway's public host, a lifetime of at most 120 s, `iat` within 60 s, a single-use `jti` and the nonce |
| Gateway proof | The `101` response carries `x-bc-gateway-proof`: a JWS of type `bc-gateway-proof+jwt` over the nonce, signed by the serving key with the serving chain in `x5c`; the daemon checks it against the serving CA it pinned                                                                                |

A WebSocket session is marked signed-only: the gateway sends it only commands and desired states that carry a
signature, and unsigned ones stay pending. The proxy can still see frame metadata (kinds, sizes, timing) and can
delay or drop frames, but it cannot forge intent. PLAN.md allows this mode only once all commands and specs are
signed: with `pki.commandSigning` both are (the gateway relays each generation's stored signed spec), so a
WebSocket daemon receives its desired state. There is no long-poll or unary path before v1. The gateway side is
`attachGatewayWs` and `serveGatewayWs`, and `connectGatewayWs` is the FakeDaemon's dialer.

**Renewal without a client certificate** (06cb334, PLAN.md §7.1). Behind such a proxy the mTLS `Renew` cannot work,
and a node would lose its identity after 24 hours, so the WebSocket listener also takes `POST /v1/renew`: the
`RenewRequest` bytes as `application/x-protobuf`, answered with the `RenewResponse` bytes, errors mapped as for gRPC
(status, `grpc-status`, `x-bc-reason`, `retry-after`). No TLS identity is used: the request carries the current chain
and a proof of possession by the current key, so the peer is the node the chain's leaf names, and JoinService checks
chain, revocation, generation and the proof (with its replay cache). A node gets 6 renewals an hour, and a body is at
most 64 KiB. `gatewayRenewHandler` serves it for hosts that attach the WebSocket listener to their own server.

**In bcd** (6cd48be). A `[websocket] url` in `config.toml` moves `bcd node` and `bcd site` to this transport: a
dependency-free RFC 6455 client dials `wss://.../v1/session` (through the environment's HTTP CONNECT proxy when one
is set), with masked binary messages of at most 16 MiB, answered pings and the close handshake. TLS is verified
against the pinned serving CA, or against `[websocket] proxy_ca_file` for the transport only; either way the 101's
`x-bc-gateway-proof` must chain to the pinned serving CA for server authentication, be signed by that leaf, and name
this node and nonce with an `iat` within 60 s, or bcd never talks to the gateway. The identity is re-read at every
dial, so a renewal applies at the next one, and renewals go to `POST /v1/renew` on the same origin, honouring the
gateway's pushback. Refusals' `x-bc-reason` and pushback (`grpc-retry-pushback-ms`, `retry-after`) steer the redial.
In WebSocket mode bcd refuses tunnels (they need the gRPC transport) and refuses to dial without command-signing
roots, since the session receives signed intent only. Go tests cover the session, unproven gateways and the renewal
against an in-process gateway, and `tests/e2e/bcd.test.ts` runs it end to end (4b2d5aa): beside a
`createDevStack({ signedIntent: true })` stack, the TypeScript `serveGatewayWs` with a serving certificate from the
CA the node pins; the node restarts with `[websocket] url`, runs a signed `inventory.refresh` over the WebSocket
session and renews over `POST /v1/renew`. `daemon install-script --websocket-url` writes the section (see
[installing bcd](/docs/daemon/install#generate-the-script)).

## Proto packages [#proto-packages]

| Package                   | Files                                                                                                                                                                              | Stability             | Holds                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `bettercompute.agent.v1`  | [`session.proto`](/docs/reference/protocol#sessionproto), [`commands.proto`](/docs/reference/protocol#commandsproto), [`inventory.proto`](/docs/reference/protocol#inventoryproto) | frozen, additive only | The [`AgentGateway`](/docs/reference/protocol#agentgateway) service, the frames, [`NodeSpec`](/docs/reference/protocol#nodespec), the v0.1 command payloads, [`NodeInventory`](/docs/reference/protocol#nodeinventory) v1                                                                                                                                                                                          |
| `bettercompute.common.v1` | [`common.proto`](/docs/reference/protocol#commonproto)                                                                                                                             | frozen, additive only | [`Signature`](/docs/reference/protocol#signature), [`TrustBundle`](/docs/reference/protocol#trustbundle), [`DriverManifest`](/docs/reference/protocol#drivermanifest), and the [`AgentRole`](/docs/reference/protocol#agentrole), [`Lane`](/docs/reference/protocol#lane), [`CommandState`](/docs/reference/protocol#commandstate) and [`CommandRejectReason`](/docs/reference/protocol#commandrejectreason) enums |
| `bettercompute.site.v1`   | [`site.proto`](/docs/reference/protocol#siteproto)                                                                                                                                 | draft until v0.2      | Site agent payloads: boot lease state, proxy-DHCP entries, iPXE scripts, BMC operations, discovery assertions, and NodeSpec's `site` section ([`SiteSpec`](/docs/reference/protocol#sitespec))                                                                                                                                                                                                                     |
| `bettercompute.driver.v1` | [`driver.proto`](/docs/reference/protocol#driverproto)                                                                                                                             | draft until v0.5      | bcx, the out-of-process driver protocol ([`DriverPlugin`](/docs/reference/protocol#driverplugin): `Describe`, `Validate`, `Invoke`, `Stream`, `Health`)                                                                                                                                                                                                                                                            |

`@better-compute/protocol` carries the TypeScript code generated by protoc-gen-es (`node scripts/proto-gen.mjs`),
plus helpers: bounded encoders and decoders, the command kind table, the signing-input layout and version
negotiation. It depends only on `@bufbuild/protobuf` and is browser-safe. The draft packages are exported as the
`site` and `driver` namespaces. `protocolRegistry` holds every message, for `Any` payloads and JSON.

## The AgentGateway service [#the-agentgateway-service]

The service as `session.proto` declares it (see [`AgentGateway`](/docs/reference/protocol#agentgateway) in the
reference for each request and response message):

```proto
service AgentGateway {
  rpc Join(JoinRequest) returns (JoinResponse);
  rpc Renew(RenewRequest) returns (RenewResponse);
  rpc Session(stream AgentFrame) returns (stream ServerFrame);
  rpc Tunnel(stream TunnelFrame) returns (stream TunnelFrame);  // v0.3
}
```

| RPC       | Auth                          | What the gateway does                                                                                                                                                                                                                                                                                                                                                                                                  |
| --------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Join`    | server TLS, serving CA pinned | Counts it against the handshake bucket and relays it to JoinService. The token is sealed (compact JWE to the JoinService encryption key) and the response is signed with the join-signing key, so the gateway can neither read the one nor forge the other                                                                                                                                                             |
| `Renew`   | mTLS                          | Relays it with the peer's node id. JoinService verifies the proof of possession: a signature by the current node key under purpose `renew-pop`, within 60 s of server time, not replayed within 120 s; for a certificate naming a TPM attestation key, also `key_attestation`, that AK's certification of the new key (d94f1f3; see [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key)) |
| `Session` | mTLS                          | Runs the session: see [sessions](/docs/daemon/sessions)                                                                                                                                                                                                                                                                                                                                                                |
| `Tunnel`  | mTLS                          | Relays one interactive stream, opened by the daemon after an `OpenTunnel` frame                                                                                                                                                                                                                                                                                                                                        |

The node id always comes from the certificate's one SPIFFE URI SAN, `spiffe://<trust domain>/site/<site>/node/<node>`,
never from a frame (any other URI SAN is a `urn:bc:` statement about the key, never an identity). `Join` also carries the hardware claims, the attestor name (`join-token` in v0.1;
`x509pop`, `provisioned` and `discovery` in v0.2; `cloud-iid` in v0.3; `tpm` in v0.5), attestor evidence, the
daemon version and the role (see [`JoinRequest`](/docs/reference/protocol#joinrequest) and
[`HardwareClaims`](/docs/reference/protocol#hardwareclaims)). A
[`JoinResponseBody`](/docs/reference/protocol#joinresponsebody) may carry a
[`JoinDeferral`](/docs/reference/protocol#joindeferral) (v0.2) instead of a certificate: the
daemon then sends `Join` again with the same CSR, after `retry_after_seconds` or with its answer to a challenge.
The join flow is described in [joining nodes](/docs/guides/nodes/joining). bcd sends `join-token` (for `bcd join`,
as role `node` or `site`), `provisioned` (for `bcd join --node-id` on an OS a deploy installed; `--attestor` names
any other) and `discovery` (for `bcd ramdisk`, as role `ramdisk`, with the site agent's serialized
`SignedDiscoveryAssertion` as the evidence); it answers no attestor challenge yet.

PLAN.md places JoinService and the secret broker in the api, behind internal mTLS. The gateway consumes them as a
port, `GatewayServices` (`join`, `renew`, `secret`, `trustBundle`), so the gateway package never imports the
server; `better-compute serve` passes the compute instance's `gatewayServices`.

## Frames [#frames]

The first [`AgentFrame`](/docs/reference/protocol#agentframe) of a session must be
[`Hello`](/docs/reference/protocol#hello) and the first [`ServerFrame`](/docs/reference/protocol#serverframe) is
[`Welcome`](/docs/reference/protocol#welcome). After that both sides send frames as they need to.

| AgentFrame (daemon to gateway) | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hello`                        | Identity, versions, journal cursors, inventory hashes, and `trust_version`: the trust bundle version the daemon holds (0 for none)                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `heartbeat`                    | A [`Heartbeat`](/docs/reference/protocol#heartbeat): free lane credits, coarse load, applied generation; inventory hashes on every 10th, and on the first after changed inventory sections (427787a)                                                                                                                                                                                                                                                                                                                                                                |
| `update`                       | A [`CommandUpdate`](/docs/reference/protocol#commandupdate): acked, running, or a terminal outcome                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `events`                       | An [`EventBatch`](/docs/reference/protocol#eventbatch) of journaled events for one `journal_id`                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `status`                       | A [`StatusReport`](/docs/reference/protocol#statusreport): observed and applied generation, section states, conditions; from v0.3 the hosted instances and `pressure` (`HostPressure`: CPU, memory and I/O PSI "some" avg10 and CPU steal); `discovered` from a site agent (unknown PXE machines) and `volume_storage` from a runtime host (the volume store's space, b8e7271); `trust_version` once the daemon applied a newer trust bundle mid-session (0 means unchanged; bcd sets it only in the first report after the change, the FakeDaemon in every report) |
| `inventory`                    | One [`InventorySection`](/docs/reference/protocol#inventorysection): the canonical encoding of a section and its SHA-256                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `spec_request`                 | A [`SpecRequest`](/docs/reference/protocol#specrequest) for a full `DesiredState`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `secret_request`               | Asks for one secret, sealed to a one-time key (v0.2)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `workload_token_request`       | A hosted instance's workload identity token, sealed to a one-time key (v0.5)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

| ServerFrame (gateway to daemon) | Purpose                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `welcome`                       | Epoch, liveness settings, lane credits, event ack, requested inventory sections, server time, trust bundle                                                                                                                                                                                                                                  |
| `command`                       | A [`Command`](/docs/reference/protocol#command): the stored body ([`CommandBody`](/docs/reference/protocol#commandbody)), an optional signature and the delivery attempt, plus a [`BatchApproval`](/docs/reference/protocol#batchapproval) and [`BatchProof`](/docs/reference/protocol#batchproof) for a batched destructive command (v1.0) |
| `cancel`                        | A [`Cancel`](/docs/reference/protocol#cancel): stop a command between actions                                                                                                                                                                                                                                                               |
| `desired`                       | A [`DesiredState`](/docs/reference/protocol#desiredstate): a full NodeSpec or a section patch at a generation                                                                                                                                                                                                                               |
| `ack`                           | An [`Ack`](/docs/reference/protocol#ack): cumulative event ack for a journal, and command ids whose results are stored                                                                                                                                                                                                                      |
| `open_tunnel`                   | Open a `Tunnel` stream for an exec ticket (v0.3)                                                                                                                                                                                                                                                                                            |
| `drain`                         | A [`Drain`](/docs/reference/protocol#drain): reconnect elsewhere after `reconnect_after_ms`                                                                                                                                                                                                                                                 |
| `revoked`                       | A [`RevocationNotice`](/docs/reference/protocol#revocationnotice): this certificate is revoked; the session closes                                                                                                                                                                                                                          |
| `secret`                        | The answer to a `secret_request`: a JWE, or an error                                                                                                                                                                                                                                                                                        |
| `workload_token`                | The answer to a `workload_token_request`: a JWE and its expiry, or an error (v0.5)                                                                                                                                                                                                                                                          |

A gateway ignores an `AgentFrame` case it does not know, so a newer daemon can send frames an older gateway does
not understand. Sessions are covered in [sessions](/docs/daemon/sessions); commands, desired state and events in
[delivery](/docs/daemon/delivery).

### Secrets [#secrets]

Commands carry secret references (`secret://`, `vault://`, `env://`), never values. For each reference the daemon
sends [`SecretRequest`](/docs/reference/protocol#secretrequest)`{ref, workflow_id, step_id, allocation_id, epk,
node_sig, request_id}`, where `epk` is a fresh
P-256 public key used once and `node_sig` covers every other field. The broker checks the signature and that the
reference is bound to the agent's current step or allocation, or, for a credential the node's own spec names, to
workflow `host:<unit or section>` and step `<field>` (`host:k8s-join` / `tokenRef`, `host:telemetry` / `headersRef`;
see [sealed secrets](/docs/operations/security#sealed-secrets)), and answers with a
[`SecretResponse`](/docs/reference/protocol#secretresponse) holding a compact JWE
(`ECDH-ES+A256KW`, `A256GCM`) to `epk`, or with an `error` of `not-found`, `not-bound`, `access-denied` or
`unavailable`. Gateways see ciphertext only. A session whose certificate is tenant-controlled always gets
`access-denied`.

bcd builds the request and opens the answer in `daemon/internal/pki/secrets.go` (`NewSecretRequest`,
`OpenSecretResponse`): the JWE has `typ` `bcsecret+jwe` and must be bound to the same reference and request id and
unexpired, and a refusal surfaces its error code. The agent matches answers by request id on the session that sent
the request, waits at most 10 seconds and ignores an answer to nothing. The Go tests check both directions against
fixtures produced by `@better-compute/pki`. These bcd features send one: a site agent's BMC operations (the
credential, and for `bmc.rotate` the new one) and `site.httpProxy` secret headers, bound to the command's workflow
step; its boot server's discovery tokens, bound to the boot entry's workflow; the ramdisk's `deploy.image` user data,
bound to the step; a VPS host's instance user data, bound to the instance's allocation; the `k8s-join` unit's
bootstrap token (7579184) and the OTLP exporter's headers (27c7ee2), bound to the node's spec.

### Workload tokens (v0.5) [#workload-tokens-v05]

A VPS host's metadata service gets a hosted instance's workload identity token the same way:
[`WorkloadTokenRequest`](/docs/reference/protocol#workloadtokenrequest)`{request_id, instance_id, audience, epk,
node_sig}`, where `audience` is empty for the issuer's
default, `epk` is a fresh P-256 key used once, and `node_sig` (purpose `workload-token`) covers the other fields. The
gateway relays it to `GatewayServices.workloadToken` outside the frame queue, and the answer is
[`WorkloadTokenResponse`](/docs/reference/protocol#workloadtokenresponse)`{request_id, jwe, expires_at, error}`: a
compact JWE of the token to `epk`, or an `error` of
`not-bound`, `disabled`, `audience`, `rate-limited`, `access-denied` or `unavailable`. A gateway without the service
answers `unavailable`, and a tenant-controlled session always gets `access-denied`. bcd implements the host's half
the same way as secrets (`NewWorkloadTokenRequest`, `OpenWorkloadTokenResponse`, a `bcworkload+jwe` answer bound to
the instance and the request id), and its metadata service (`[imds] listen`) calls it for the running instance that
holds the request's source address. See [workload identity](/docs/guides/vps/instances#workload-identity).

### Tunnels (v0.3) [#tunnels-v03]

[`OpenTunnel`](/docs/reference/protocol#opentunnel)`{tunnel_id, ticket}` asks the daemon to open a `Tunnel` stream
of [`TunnelFrame`](/docs/reference/protocol#tunnelframe)s. The ticket is an ES256 JWT, valid 60
seconds, with a single-use `jti`, audience `node:<id>`, a `sid` and an `op`. The gateway verifies it before asking,
and the daemon verifies it again against the JWKS in its NodeSpec `access` section and its own `jti` set. The
daemon's first `TunnelFrame` is `opened`; then `data` on channel 0 (stdin and stdout, or the byte stream) or 1
(stderr), `resize` with rows and columns, and `closed` with an exit code. bcd refuses a ticket by opening nothing,
so the gateway's open times out; it runs at most 16 tunnels at once. What it runs for each op, limits and
re-authorization are in [tunnels](/docs/guides/access/tunnels#the-node-half).

## Message size bounds [#message-size-bounds]

Everything is bounded (principle P12). The bounds live in `@better-compute/protocol`, and both sides decode with
them before parsing:

| Constant                 | Bound | Applies to                                                                   |
| ------------------------ | ----- | ---------------------------------------------------------------------------- |
| `MAX_COMMAND_BODY_BYTES` | 1 MiB | A serialized `CommandBody`, a command payload, a command result              |
| `MAX_SPEC_BODY_BYTES`    | 4 MiB | A serialized [`DesiredStateBody`](/docs/reference/protocol#desiredstatebody) |
| `MAX_FRAME_BYTES`        | 5 MiB | One `Session` or `Tunnel` frame, and every gRPC message the gateway reads    |

`encodeMessage` and `decodeMessage` throw `ProtocolError` with `FRAME_TOO_LARGE` or `INVALID_FRAME`. Decoding keeps
unknown fields, so a relay that re-encodes a newer peer's message loses nothing. Bulk data never rides the session:
log collection uploads to a presigned URL and images are fetched by URL.

## Signed envelopes [#signed-envelopes]

Anything a relay must not alter travels as `bytes body` plus a `Signature`, and the signature covers the exact body
bytes, so nobody ever re-encodes to verify. A [`Signature`](/docs/reference/protocol#signature) is ES256 in the raw 64-byte `r || s` form, a `kid` and an
optional certificate chain. The signed bytes are domain-separated by purpose:

```text
input = "bettercompute.v1/" || purpose || 0x00 || part_1 || ... || part_n
part  = uint32_be(len(bytes)) || bytes
```

| Purpose               | Signer                                                                                                                                                                | Parts                                                                                                                                                   |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `command`             | worker key                                                                                                                                                            | `[Command.body]`                                                                                                                                        |
| `desired-state`       | worker key                                                                                                                                                            | `[DesiredState.body]`                                                                                                                                   |
| `join-response`       | JoinService signing key                                                                                                                                               | `[JoinResponse.body]`                                                                                                                                   |
| `trust-bundle`        | JoinService signing key                                                                                                                                               | `[SignedTrustBundle.body]`                                                                                                                              |
| `attested-report`     | node key                                                                                                                                                              | `[AttestedReport.body]`                                                                                                                                 |
| `command-update`      | node key                                                                                                                                                              | `[command_id, uint32_be(state), sha256(result)]`                                                                                                        |
| `secret-request`      | node key                                                                                                                                                              | `[request_id, ref, workflow_id, step_id, allocation_id, epk]`                                                                                           |
| `workload-token`      | node key                                                                                                                                                              | `[request_id, instance_id, audience, epk]` (v0.5)                                                                                                       |
| `renew-pop`           | current node key                                                                                                                                                      | `[sha256(csr), node_id, client_time]`                                                                                                                   |
| `x509pop`             | node key of a possibly expired certificate                                                                                                                            | `[sha256(csr), node_id, client_time]`                                                                                                                   |
| `discovery-assertion` | site agent node key (`bcd site`, one per iPXE script it serves)                                                                                                       | `[SignedDiscoveryAssertion.body]`                                                                                                                       |
| `batch-approval`      | operator approver key (v1.0 [quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)), in [`BatchApproval.signatures`](/docs/reference/protocol#batchapproval) | `[batch_id, scope_root, uint32_be(scope_size), timestamp(not_after), trust_domain]`; each verifier supplies the trust domain from its own configuration |
| `approver-set`        | a current operator approver key                                                                                                                                       | `[uint64_be(approver_set_epoch), sha256(approverSetInput), trust_domain]` (4835f4f, c6214dc): a change of the trust bundle's quorum fields              |

Worker keys chain to the command-signing roots of the [`TrustBundle`](/docs/reference/protocol#trustbundle) the
daemon received in `JoinResponse`, and a newer bundle arrives in `Welcome.trust`, a
[`SignedTrustBundle`](/docs/reference/protocol#signedtrustbundle) signed by the pinned JoinService key. The bundle also carries the v1.0
quorum's `approver_keys`, `approver_threshold`, `quorum_tiers` and `unbatched_destructive_per_hour`, empty unless
`pki.approvers` is set, `approver_set_epoch` (field 8, c6214dc), the set's own epoch, separate from `version` and
raised with every change of the set, and `approver_set_signatures`, by which the current approvers sign a change of
those fields for that epoch (purpose `approver-set`, parts `[uint64_be(approver_set_epoch),
sha256(approverSetInput), trust_domain]`), published on every version while the set is current. A daemon holding set
`H` at epoch `e` takes `(N, e')` only when `e' > e` and its current threshold of `H`'s keys signed it, `N == H`
included; otherwise it keeps its quorum and applies the rest of the bundle. Both daemons report the set they enforce,
`approver_set_epoch` and `approver_set_sha256`, in `Hello` (fields 18 and 19) and beside `trust_version` in
`StatusReport` (10 and 11), since a bundle's version can be taken while its quorum part is refused (see
[changing the approver set](/docs/operations/pki#changing-the-approver-set)). The daemon accepts a bundle only
when its version is greater than the one it holds, and the control plane raises the version whenever the bundle's
content changes, so a change always reaches the fleet. The envelopes exist from v0.1 with optional signatures, so
turning them on is additive. See [delivery](/docs/daemon/delivery#signed-commands-and-attested-results) for which
kinds require them.

## Errors outside frames [#errors-outside-frames]

Refusals that end a stream use gRPC status codes:

| Code                 | When                                                                                                                                                                                                           |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UNAUTHENTICATED`    | No valid node certificate; a revoked or superseded certificate; the certificate expired during the session                                                                                                     |
| `PERMISSION_DENIED`  | `Hello.node_id` is not the certificate's node; the certificate is not valid for `Hello.role`; the node has not joined                                                                                          |
| `INVALID_ARGUMENT`   | The first frame is not `Hello`; `Hello` lacks `role`, `boot_id` or `journal_id`; a second `Hello`; an undecodable message                                                                                      |
| `RESOURCE_EXHAUSTED` | The handshake token bucket, the stream limit or the pending-Hello limit; retry after the `grpc-retry-pushback-ms` trailer the refusal carries (see [opening](/docs/daemon/sessions#opening)), else with jitter |
| `ABORTED`            | A newer session of the same node superseded this one                                                                                                                                                           |
| `UNAVAILABLE`        | The gateway is starting, draining, stopping or fenced, or a frame failed on a temporary error; reconnect                                                                                                       |
| `UNIMPLEMENTED`      | An unknown method                                                                                                                                                                                              |

## Version skew [#version-skew]

**Negotiation.** `Hello` carries `proto_min`, `proto_max`, `daemon_version` and `capabilities`. The gateway picks
the highest version inside both ranges (`negotiateProtocolVersion`) and returns it in `Welcome.protocol_version`,
along with its own capabilities. It also stores the daemon's capabilities as `node_sessions.capabilities`, so the API
can refuse what an old daemon cannot do safely: `tunnels.open` opens an instance `ssh` or `tcp` tunnel only when the
node advertises `tunnel.instance-dial` (see [sessions](/docs/daemon/sessions#hello)). The gateway also writes each
capability as the system node label `bc.dev/cap.<name>=1`, and placement gates on those labels so an older bcd never
gets a spec field it would drop as unknown: vps places instances that need `instance.network-policy`,
`instance.ssh-host-only`, `instance.placement` or `instance.volumes` only where they are advertised, and capacity
places reservations that book host resources only on `devices.limits` nodes and `binding: 'kubernetes'` ones only on
`binding.kubernetes` nodes (409f783). A plain GPU hold without host resources needs no capability and still places on
an N-2 daemon. Today both sides speak exactly version 1 (`PROTOCOL_VERSION = { min: 1, max: 1 }`
in TypeScript, `ProtoMin` and `ProtoMax` in Go). When the ranges do not overlap, the daemon is below the server's
minimum: `Welcome.upgrade_required` is set, `Welcome.protocol_version` is the gateway's minimum, no desired state or
command other than `daemon.upgrade` is sent, and the FakeDaemon and bcd reject anything else `UNSUPPORTED`. bcd
runs `daemon.upgrade` (de9f380) when its `[update]` section pins release keys; without pins it refuses the update,
and the binary must be replaced by hand.

**The gateway's range.** `createGateway({ protocol: { min, max } })` sets the versions a gateway speaks; the default
is `PROTOCOL_VERSION`. Negotiation and the `Welcome` use it, so a gateway at `{ min: 1, max: 3 }` welcomes a daemon
at 1..1 at version 1 and one at 1..2 at version 2, while a gateway at `{ min: 2, max: 3 }` restricts a 1..1 daemon to
`upgrade_required`. A range with `min` below 1, `max` below `min` or a non-integer bound is `INVALID_CONFIG` at
construction. It exists for the v1.0 N-2 skew tests and staged protocol bumps (`tests/gateway-protocol.test.ts`);
no version beyond 1 is defined yet.

**The skew matrix.** `tests/skew-matrix.test.ts` runs each supported daemon line against real dev stacks
(`createDevStack({ gateway: { protocol }, daemon: { profile } })`). A FakeDaemon `profile` from `daemonSkewProfiles`
in `@better-compute/testing` makes it behave like an older bcd: the version and protocol range it announces, its
capabilities, the kinds it rejects `UNSUPPORTED`, the spec sections it reports `SectionUnsupported`, the inventory
sections and `Hello` fields it never sends. The rows today are N (0.5.0), N-1 (0.4.0, no `instance.snapshot`) and
N-2 (0.3.0, also no `daemon` section and no `trust_version`); each is welcomed at the highest shared version,
adopted, runs `inventory.refresh` and gets its missing kinds rejected. `belowMinimumProfile`, and the current line
against a gateway at `{ min: 2, max: 3 }`, get `upgrade_required`: no spec, and ordinary commands stay pending. A
release adds a row on top and drops the one that leaves the window.

**What is recorded.** The gateway stores `Hello.daemon_version` and `Hello.proto_max` in `node_sessions`
(migration `core/0008_session_versions`), which is how the rollout controller sees a node finish an upgrade and
finds nodes that need `daemon.upgrade`.

**Policy.** Daemons may lag N-1 minor versions before 1.0 and N-2 from 1.0. The range widens when a release adds a
version and its minimum rises when the skew window drops one. The upgrade order is expand migrations, workers and
api, gateways, daemons, then contract migrations a release later.

**Additive only.** `agent/v1` and `common/v1` were frozen at the end of P0. CI runs `buf breaking` with the `FILE`
rule set against `proto/baseline.binpb`; the draft `site` and `driver` packages are excluded until their phases
freeze them. Within the frozen packages, change is absorbed at every level:

| Something new        | An older peer                                                                                                |
| -------------------- | ------------------------------------------------------------------------------------------------------------ |
| A field              | Ignores it; relays keep it as an unknown field                                                               |
| An `AgentFrame` case | The gateway ignores the frame                                                                                |
| A command kind       | The daemon rejects it `UNSUPPORTED`                                                                          |
| A NodeSpec section   | The daemon reports condition `SectionUnsupported` with the field number as the reason and converges the rest |
| A protocol version   | Negotiated down, or `upgrade_required` below the minimum                                                     |

## Built-in command kinds [#built-in-command-kinds]

`commands.proto` holds the payloads and results of the v0.1 kinds, and `commandKinds` in `@better-compute/protocol`
holds their flags; the daemon's dispatcher must carry the same table. Later kinds add their own messages: the
ramdisk's `diag.dcgm`, `disk.wipe`, `deploy.image` and `boot.kexec` in `bettercompute.agent.v1`, the site agent's
`bmc.*`, `site.httpProxy`, `boot.prepareRamdisk` and `boot.cleanUp` in `bettercompute.site.v1` (their flags are
`metalCommandKinds` in `@better-compute/metal`, and bcd implements them), then `driver.invoke`, which bcd runs in the
bcx plugins it hosts (see [drivers](/docs/extending/drivers#driver-plugins-in-bcd)), and `daemon.upgrade`, which bcd
runs as its [self-update](/docs/daemon/install#self-update) (de9f380).

| Kind                | Lane    | Retry-safe | Self-terminating | Attested | Destructive | Roles               |
| ------------------- | ------- | ---------- | ---------------- | -------- | ----------- | ------------------- |
| `diag.nvml`         | control | yes        | no               | yes      | no          | node, ramdisk       |
| `inventory.refresh` | control | yes        | no               | no       | no          | node, site, ramdisk |
| `logs.collect`      | bulk    | yes        | no               | no       | no          | node, site, ramdisk |
| `node.reboot`       | control | yes        | yes              | no       | yes         | node, site          |
| `gpu.reset`         | control | no         | no               | yes      | yes         | node                |

Metal's kinds, by lane, are in [delivery](/docs/daemon/delivery#kinds-by-lane); every module kind and its timeout
is in [command kinds](/docs/reference/command-kinds).


# Running bcd on real hardware (/docs/daemon/real-hardware)

> A bring-up checklist for bcd on a real machine, what bcd hardware and bcd doctor check before a join, what changes between root, bcd helper and an unprivileged node, WSL 2 specifics, and exactly what has and has not run on real hardware.



Most of bcd is tested against fakes: a fake HGX node (`fakehw`), a fake host for the config units, fake `virsh`,
`qemu-img` and `nvidia-smi`. Round 6 (2026-10-04, `bbfa7f4` onwards) ran it on a real machine for the first time and
fixed what that found. This page is for the operator bringing bcd up on a new server or workstation: the order to do
things in, the two local commands that tell you what bcd sees (`bcd hardware`, `bcd doctor`), how the privilege mode
changes what bcd can do, what is special about WSL 2, and an exact list of what has and has not run on real hardware.
Installing the binary and the systemd units is on [installing bcd](/docs/daemon/install); the GPU side in depth is on
[GPUs on a node](/docs/daemon/gpus).

> **One real machine so far.** 
  Everything below that ran on real hardware ran on one machine: a workstation running WSL 2 (Ubuntu) on an AMD
  Threadripper PRO 9995WX (192 threads) with one GeForce RTX 5090 (driver 616.56, CUDA 13.4), as an ordinary user,
  without root, an IOMMU, DMI identity or a TPM. No bare-metal server and no data-center GPU has run bcd yet.

## Which build [#which-build]

Real hardware needs a host build: one without the `fakehw` tag, which reads `/proc`, `/sys` and `nvidia-smi`.

| Build          | Tags                                                          | Used by                                                                        |
| -------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Release        | none (`daemon/.goreleaser.yaml`: real NVML and real hardware) | The install script's pinned binaries                                           |
| Host, cgo-free | `nonvml`                                                      | `daemon/Containerfile` (bcd as a DaemonSet), `tests/e2e/bcd-host-real.test.ts` |
| Simulated      | `nonvml,fakehw`                                               | Unit tests, the other e2e suites, `bcd sim`                                    |

Both host builds read GPUs through `nvidia-smi`; the `nonvml` build only makes doctor's `nvml` check warn (see
[the NVML backend](/docs/daemon/gpus#the-nvml-backend)). `bcd version` prints the build tags. `bcd hardware` says
which backend it runs (`host` or `fakehw`) in its first line, and `--capture` refuses a fakehw build.

```bash
cd daemon
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags nonvml -o bin/bcd ./cmd/bcd
```

## Bring-up checklist [#bring-up-checklist]

1. **Firmware.** Turn on VT-x or AMD-V for VMs, and VT-d or AMD-Vi for GPU passthrough. Under Secure Boot the NVIDIA
   kernel module must be signed by an enrolled key.
2. **Operating system.** systemd as init, cgroup v2 (unified) mounted at `/sys/fs/cgroup`, an NTP client, and
   `nouveau` blacklisted on a GPU host. Install the NVIDIA driver yourself, or leave it to the
   [`nvidia-driver` unit](/docs/daemon/host-config#nvidia-driver).
3. **Look before joining.** Copy a host build of bcd onto the machine and run `sudo bcd hardware`. Read the findings:
   a missing product UUID, placeholder serials or no NICs change how the machine can join (below).
4. **Preflight.** Write `/etc/bcd/config.toml` (the install script does) and run `sudo bcd doctor --config
   /etc/bcd/config.toml`. Fix every `fail` line. The install script runs doctor too, but only prints its warning.
5. **Pick the privilege mode.** Root (the install script's default) or `--unprivileged` with `bcd helper`. Running
   `bcd node` as an ordinary user without the helper works for trying bcd out, but takes no GPU holds
   ([below](#root-bcd-helper-or-unprivileged)).
6. **Join.** A machine whose DMI gives no product UUID or serials (WSL, many VMs, boards with unprogrammed SMBIOS)
   has no trusted hardware key, and JoinService refuses it unless the join token is
   bound to an identifier it does have: create the token with `expect: { mac }` (`--expect-mac` on the CLI), the MAC
   of its default-route NIC (see
   [binding a token to one machine](/docs/guides/nodes/joining#binding-a-token-to-one-machine)).
7. **Check the node.** `nodes.get` lists its conditions. `GpuModelUnknown` keeps a GPU node from scheduling until the
   control plane's model table knows the card (see [GPU naming](/docs/guides/nodes/inventory-and-health#gpu-naming));
   `GpuCountMismatch` means the pool expected another count or model.
8. **Watch it run.** `bcd status` shows the session, the generations, each section and one line per GPU (see
   [bcd status](/docs/daemon#bcd-status)).

## bcd hardware [#bcd-hardware]

`bcd hardware` (e82d3a8) prints what this binary sees of the machine, without joining or talking to anything: the
inventory bcd would report and a list of bring-up findings. Run it as root: the product UUID, serials, DIMMs and NVMe
health are root-only, and an ordinary user reads them empty.

| Flag            | Does                                                                                                                                                                                                                                                                                                         |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| (none)          | A summary per section, then the findings                                                                                                                                                                                                                                                                     |
| `--json`        | The whole inventory as JSON, and no findings                                                                                                                                                                                                                                                                 |
| `--capture DIR` | Copies exactly the procfs and sysfs files the collector read into `DIR`, so `inventory.Collector{Root: DIR}` replays the same machine (identical section hashes). A path ending in `.tar.gz` writes a reproducible archive instead, since sysfs symlinks do not survive a Windows checkout. Host builds only |
| `--redact`      | With `--capture`: replaces serials, UUIDs, MACs, WWNs, GPU UUIDs and the hostname with stable stand-ins of the same shape (UUID path components too), so a fixture tells nothing about the machine. Without `--capture` it is a usage error (exit 2)                                                         |

The summary has one line per system, CPU (with its NUMA nodes and isolated CPUs), memory (with each DIMM), disk, NIC,
GPU, BMC and OS:

```text
bcd <version> (host backend)

System    <vendor> <product> (board <vendor> <board>), BIOS <vendor> <version>, UEFI (Secure Boot), TPM 2.0
CPU       <model> (<vendor> x86_64): 2 sockets, 96 cores, 192 threads
  node0   cpus 0-47,96-143, 503.8 GiB
Memory    1007.6 GiB
  DIMM    <slot>       64.0 GiB DDR5 4800 MT/s ECC <manufacturer> <part>
Disk      nvme0n1    3.5 TiB   nvme <model> (serial <serial>, wwn <wwn>)
NIC       eth0       <mac> mlx5_core 0000:18:00.0 100000 Mb/s fw <version>
GPU 0     NVIDIA H100 80GB HBM3 00000000:1B:00.0 GPU-<uuid>
BMC       <vendor> <mac> <ip> (IPMI, Redfish) fw <version>
OS      ubuntu 24.04, kernel <release>, cgroup v2, runtimes containerd,libvirt

Findings
  ok    dmi       <vendor> <product>, UUID <uuid>
  ...
```

The GPUs come from the same collection `bcd node` makes (procfs, then `nvidia-smi`; see
[where GPU facts come from](/docs/daemon/gpus#where-gpu-facts-come-from)), except that `bcd hardware` keeps no memory
of GPUs, so a GPU passed through to a VM at that moment is not listed. A capture copies procfs and sysfs only;
`nvidia-smi`'s answers are not part of it (GPU fixtures are captured separately, under
`daemon/internal/nvml/testdata`).

### Findings [#findings]

`inventory.HostFindings` reviews the collection. Every finding has a status (`ok`, `warn`, `fail`), and `bcd
hardware` exits 1 when one fails:

| Check         | Says                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dmi`         | `warn`: no DMI at all (a container, or a VM without SMBIOS; the node is identified by MACs only); the product UUID and serials read empty because bcd is not root; no product UUID; a placeholder product UUID (`00000000-…`, `ffffffff-…`, `03000200-0400-0500-0006-000700080009` and similar), so the hardware key falls back to serials and MACs and a reclaim needs `expect` or approval; a placeholder system serial |
| `cpu`, `numa` | `fail` when `/proc/cpuinfo` is unreadable; `warn` without `/sys/devices/system/node` (NUMA-pinned placements cannot be honoured)                                                                                                                                                                                                                                                                                          |
| `memory`      | `fail` when `/proc/meminfo` is unreadable; `warn` when EDAC reports uncorrectable errors; otherwise the total and the huge page pools                                                                                                                                                                                                                                                                                     |
| `dimms`       | `warn` when only some DIMMs are ECC, or (as root) when the SMBIOS table lists no memory devices                                                                                                                                                                                                                                                                                                                           |
| `disks`       | `warn` with no physical disk under `/sys/block`, or a disk reporting a critical warning, at least 90% wear or media errors                                                                                                                                                                                                                                                                                                |
| `bmc`         | `warn` when the host has an IPMI interface but the BMC did not answer through `/dev/ipmi0` (load `ipmi_devintf`)                                                                                                                                                                                                                                                                                                          |
| `nics`        | `warn` with no physical NIC: no MAC identity, no PXE MAC for metal                                                                                                                                                                                                                                                                                                                                                        |
| `gpus`        | `ok`: the GPU count and driver version                                                                                                                                                                                                                                                                                                                                                                                    |
| `cgroup`      | `fail` without cgroup v2: hold slices and instance limits need it (`systemd.unified_cgroup_hierarchy=1`)                                                                                                                                                                                                                                                                                                                  |
| `kvm`         | `warn` without `/dev/kvm`                                                                                                                                                                                                                                                                                                                                                                                                 |
| `iommu`       | `warn` for GPUs without an IOMMU: no passthrough                                                                                                                                                                                                                                                                                                                                                                          |

Round 6 fixed what the first real machine showed (e82d3a8): disks behind Hyper-V's storvsc, iSCSI or a RAID
controller's virtual drives are no longer called SAS; SCSI and SATA serials come from the kernel's `vpd_pg80`; a NIC
on VMBus has no PCI address; SR-IOV virtual functions and a BMC's Redfish USB host interface are not the machine's
NICs; and arm64 topology comes from sysfs, with core names from the MIDR part (Grace reads `Neoverse-V2`). Later
commits added DIMMs from SMBIOS (82fef0e), NIC firmware (9b83ece), NVMe health (362435e), SATA and SAS health through
`smartctl` (97089e9) and the BMC (9ddb5ea).

## bcd doctor as preflight [#bcd-doctor-as-preflight]

`bcd doctor --config FILE` checks the configuration, `nvidia-smi`, the inventory, the host and the node's identity,
one line per check, and exits 1 when any check fails. The checks it always had are listed on
[the overview](/docs/daemon#modes), the GPU checks on [GPUs on a node](/docs/daemon/gpus#bcd-doctors-gpu-checks).
Since e2a0cf0 a host build ends with a real-host preflight (`doctor_host.go`):

| Check          | `ok`                                                                                                         | Otherwise                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `systemd`      | systemd is init (`/run/systemd/system`)                                                                      | `fail`: hold slices, the helper's units and instance units are systemd units                                                                                                                                                                                                                                                                                                                     |
| `delegation`   | The cgroup root delegates `cpu`, `cpuset`, `memory`, `io`, `pids` and `hugetlb` (`cgroup.subtree_control`)   | `warn` naming the controllers the kernel offers but the root does not delegate; systemd enables one once a unit asks for it, so this usually settles once bcd's slices exist                                                                                                                                                                                                                     |
| `kvm`          | (only reported when missing)                                                                                 | `warn` with the cause: the CPU has `vmx` or `svm` but the module is not loaded or is blacklisted; a VM without nested virtualization; or no VT-x/AMD-V flags (enable them in the firmware)                                                                                                                                                                                                       |
| `vfio`         | On a GPU host with an IOMMU: `vfio-pci` loaded                                                               | `warn`: passthrough loads it on first use                                                                                                                                                                                                                                                                                                                                                        |
| `iommu-groups` | Each GPU in an IOMMU group of its own (with its other functions and PCI bridges)                             | `warn` (6cf5122) naming each GPU that shares its group with another device, such as a NIC or an NVMe drive behind a PCIe switch or a root port without ACS: `bcd helper` refuses to pass such a GPU through, so move it to a CPU-attached slot or enable ACS. Also `warn` for GPUs with no IOMMU group, and for an inventory with GPUs but no NVIDIA function on the PCI bus (a paravirtual GPU) |
| `clock`        | The kernel says an NTP client keeps the clock synchronized (`adjtimex`, read-only), with its estimated error | `fail` when unsynchronized: certificates, tickets and signed intents compare times; `warn` when it cannot be read                                                                                                                                                                                                                                                                                |
| The findings   |                                                                                                              | Every finding of [`bcd hardware`](#findings) that is not `ok`, except `kvm`, `iommu` and `cgroup`, which other checks report                                                                                                                                                                                                                                                                     |

The GPU checks of f84f1f9 (`gpu-bus`, `nouveau`, `persistence`, `fabric-manager`, `gpu-host`) and da60bbd
(`gpu-binding`) run in the same pass. On the round-6 workstation doctor reported `nvml` ok (616.56), the `gpu` line
ok, `gpu-host` warning about WSL, `systemd` ok, the clock synchronized, `hugetlb` not yet delegated and no DMI.

## Root, bcd helper or unprivileged [#root-bcd-helper-or-unprivileged]

bcd runs in one of three ways, and they differ in what reaches the control plane:

|                                                      | `bcd node` as root                                                                              | `bcd node` as `bcd`, beside a root `bcd helper`                                                                              | `bcd node` as an ordinary user, no `[helper]`                                                                      |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Set up by                                            | The install script's default                                                                    | `daemon install-script --unprivileged` ([running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)) | Running the binary by hand, as the real-host e2e does                                                              |
| GPU holds                                            | Writes `/etc/cdi` and hold slices itself                                                        | Through the helper's `file.write` and `slice.apply`                                                                          | Refused at once with `node-unprivileged` ([unprivileged nodes](/docs/daemon/devices-and-holds#unprivileged-nodes)) |
| `Hello` capabilities on a GPU host                   | `cdi`, `devices.limits`, `binding.kubernetes`                                                   | The same                                                                                                                     | None of the three, so reservations with host resources or a Kubernetes binding are never placed there              |
| Host config units                                    | Applied directly                                                                                | Applied by the helper's `host.unit`, from signed specs ([under bcd helper](/docs/daemon/host-config#under-bcd-helper))       |                                                                                                                    |
| Product UUID and serials                             | Read from sysfs                                                                                 | Asked from the helper's `dmi.read` (a fixed list of `/sys/class/dmi/id` files, 128 bytes each, no arguments)                 | Read empty: the hardware key leans on MACs                                                                         |
| DIMMs (SMBIOS type 17), NVMe health, the BMC in-band | Read directly                                                                                   | Asked from the helper's `hw.read` (parsed values only, at most 64 entries each, 30 seconds overall)                          | DIMMs from EDAC only; no NVMe health; the BMC's interfaces from sysfs only                                         |
| SATA and SAS health (`smartctl`)                     | Read with smartmontools installed                                                               | Not part of `hw.read`: `smartctl` needs root                                                                                 | Not read: `smartctl` needs root                                                                                    |
| GPU passthrough persistence                          | Set by bcd around its own `virsh` and VFIO calls                                                | Set by the helper around `libvirt.start`, `libvirt.restore` and `libvirt.destroy`                                            |                                                                                                                    |
| `doctor`                                             | Also checks the helper's directories, its binding and signed-intent pins when `[helper]` is set | Run it as root to get those                                                                                                  | Warns `gpu-binding` on a GPU host                                                                                  |

`dmi.read` and `hw.read` (29ed48d, 32beca7, 45bc17f) are read-only and take no arguments: the helper enumerates the
entries and controllers itself and never lets the node name a path. What they return is inventory only, never an
attestation input.

## WSL 2 [#wsl-2]

WSL 2 is a convenient place to try bcd on a GPU workstation, and it is where round 6 ran. It is a Hyper-V VM with a
Linux kernel, and differs from a server in ways bcd notices:

| Area       | On WSL 2                                                                                                               | Effect on bcd                                                                                                                                                                                                                |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GPU        | Paravirtual, through `/dev/dxg`; no `/dev/nvidia*`, no NVIDIA kernel module, no PCI function, no `/proc/driver/nvidia` | Inventory and samples from `nvidia-smi` only; no passthrough, `gpu.reset`, MIG, DCGM or XIDs; the `nvidia-driver` unit never installs a Linux driver. See [WSL 2 paravirtual GPUs](/docs/daemon/gpus#wsl-2-paravirtual-gpus) |
| DMI        | Usually no product UUID or serials                                                                                     | No trusted hardware key: bind the join token to the MAC (`expect: { mac }`)                                                                                                                                                  |
| TPM        | None                                                                                                                   | The node key is a PKCS#8 file; `--attestor tpm` cannot be used                                                                                                                                                               |
| IOMMU      | None                                                                                                                   | No passthrough; doctor's `iommu` warns                                                                                                                                                                                       |
| KVM        | `/dev/kvm` when nested virtualization is on                                                                            | VMs possible; doctor's `kvm` says why when it is missing                                                                                                                                                                     |
| systemd    | Only when enabled for the distribution                                                                                 | Doctor's `systemd` check fails without it, and hold slices cannot work                                                                                                                                                       |
| cgroups    | v2, but `hugetlb` not delegated at the root at first                                                                   | Doctor's `delegation` warns                                                                                                                                                                                                  |
| Devices    | Disks on Hyper-V's storvsc, NICs on VMBus                                                                              | Disks are not SAS; NICs have no PCI address (fixed in e82d3a8)                                                                                                                                                               |
| Networking | The Windows host is the default route                                                                                  | A control plane on Windows is reached at the default-route address, with `gateway_server_name` naming the certificate's host                                                                                                 |

`testdata/hosts/wsl2-threadripper-9995wx.tar.gz` in `daemon/internal/inventory` is that machine, captured with
`bcd hardware --capture … --redact`: the first real-host fixture, replayed by the inventory tests.

## What has run on real hardware [#what-has-run-on-real-hardware]

Every item here ran on the round-6 workstation, as an ordinary user unless it says otherwise.

| What                                  | Where                                                                                     | Result                                                                                                                                                                                                                                                                                      |
| ------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Join, adoption and the real inventory | `tests/e2e/bcd-host-real.test.ts` with `BCD_E2E_REAL=1` (92e3385)                         | A dev stack on Windows with external TLS; a `nonvml` host build in WSL joins over TLS 1.3 and mTLS with a token bound to the default-route MAC, is adopted, and its inventory matches the kernel: `uname -r`, the CPU count and memory within 5%. One GPU in inventory through `nvidia-smi` |
| An unprivileged GPU node              | The same suite (da60bbd)                                                                  | No `bc.dev/cap.devices.limits` label, and a 1-GPU reservation never reaches `binding` or `active` there. Before the GPU model table's version 2 (223022b) the RTX 5090 was `GpuModelUnknown` and the reservation step skipped (8e9602c)                                                     |
| OTLP GPU metrics                      | The same suite (898a23c)                                                                  | `NodeSpec.telemetry` pointed at a collector in the test; the export carries every GPU `nvidia-smi` lists, `bcd.gpu.memory.total` equal to `nvidia-smi`'s framebuffer, and a power limit. 4 of 4 tests passed                                                                                |
| `bcd status` and `bcd doctor`         | The same suite                                                                            | `status --json` answers with the node, connected, role `node`; doctor runs and reports                                                                                                                                                                                                      |
| `nvidia-smi` parsing                  | `TestSnapshotLiveHost` in `daemon/internal/nvml` with `BCD_REAL_GPU=1` (0915ab7, 9a6fee9) | Every GPU yields UUID, name, architecture, VRAM, compute capability and bus id; light samples agree with full ones                                                                                                                                                                          |
| GPU host units, planned only          | `TestRealGPUUnitsPlan` in `daemon/internal/hostconfig` with `BCD_REAL_GPU=1` (2e71a60)    | `nvidia-driver` with the host's branch and `gpu-settings` with the current power limit plan nothing; branch 470 is refused terminally; `container-toolkit` plans the toolkit install, runtime configuration and `/etc/cdi/nvidia.json`. Nothing is applied                                  |
| Kernel settings, as root              | `TestRealKernel` in `daemon/internal/hostconfig`                                          | A sysctl, the transparent huge page policy and 2 MiB pages set and put back (see [host configuration](/docs/daemon/host-config))                                                                                                                                                            |
| Doctor's real-host and GPU checks     | Run by hand (e2a0cf0, f84f1f9)                                                            | As listed [above](#bcd-doctor-as-preflight)                                                                                                                                                                                                                                                 |

```bash
# Linux, or Windows with WSL (BCD_WSL_DISTRO, default Ubuntu); builds bcd with -tags nonvml first
BCD_E2E_REAL=1 pnpm vitest run tests/e2e/bcd-host-real.test.ts
# The real-GPU Go tests (scripts/daemon-test.mjs forwards BCD_* into WSL)
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestSnapshotLiveHost ./internal/nvml/
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestRealGPUUnitsPlan ./internal/hostconfig/
```

The suite asserts only what holds on any Linux host and logs the rest (GPU count, KVM), so it runs on a laptop, in
WSL or on a GPU server; its GPU tests skip without `nvidia-smi`.

## What has not [#what-has-not]

* **A GPU hold on real hardware.** bcd ran unprivileged without `[helper]`, so no CDI spec was written to a real
  `/etc/cdi`, no hold slice was created on a real systemd, and the reservation test took its unprivileged branch.
* **A container from a CDI name.** The workstation has no `nvidia-ctk`, so no vendor spec was generated and no
  container was started with `bcd.dev/gpu=<id>`.
* **Passthrough.** No IOMMU, so no GPU was bound to `vfio-pci`, and the persistence-mode sequence around it
  (f25abbe, 5bb9f22) and the passed-through GPU memory (c5c52c4) are tested against fakes only. Neither
  `libvirt-kvm` nor `cloud-hypervisor` has run a real VM.
* **MIG, NVLink, NVSwitch, Fabric Manager, MPS.** The only GPU is a GeForce card without them.
* **Data-center GPUs, ECC and row remapping, XIDs, `gpu.reset`, DCGM.** None on a real board.
* **Applying host units.** Driver installs, MIG layouts, boot arguments, 1 GiB pages, power limits and Kubernetes
  joins are planned against the real driver at most, never applied.
* **The TPM.** No TPM on the workstation; every TPM test uses the software TPM.
* **Root-only readers.** The NVMe and IPMI ioctl paths, DIMMs from SMBIOS and `smartctl` have not run as root on a
  real server.
* **A released binary through the install script.** No release has been tagged (see
  [installing bcd](/docs/daemon/install)).


# Instance runtimes (/docs/daemon/runtimes)

> The runtimes bcd runs VPS and sandbox instances on, what each one does for disks, NICs, GPUs, volumes, snapshots, consoles and seeds, how placement and admission apply, and how they run under bcd helper.



A VPS host's `bcd node` runs the instances its NodeSpec's `instances` section lists through a **runtime**: a driver
for one hypervisor or container engine behind a small Go interface. The `instances` converger decides what should
exist and in which state; the runtime makes it so. This page covers the runtimes themselves. What the converger does
around them (the desired set, user data, admission, network rules, snapshot transfers) is on
[instances](/docs/daemon/delivery#instances), and the control plane's side is on
[VPS instances](/docs/guides/vps/instances#runtimes).

> **No VMM has booted under bcd on real KVM.** 
  `libvirt-kvm` (362317c) and `cloud-hypervisor` (2598f0a) are written, reviewed and tested against fake tools only: a
  fake `virsh` and `qemu-img`, a fake cloud-hypervisor REST API, fake `systemd-run`, `ip` and seed tools. As of round 6
  (`.claude/coordination.md`, 2026-10-04) no virtual machine has booted under bcd on a real KVM host: the development
  machine has `/dev/kvm` in WSL but no hypervisor installed, and installing one needs the user's approval. The end-to-end
  tests (`tests/e2e/bcd-vps.test.ts`, `bcd-placement.test.ts`) run a fakehw build's file-backed fake runtime through the
  real control plane. Treat everything below about real VMs as the code's intent, checked by unit tests, not as
  production behaviour.

## Which runtimes a host runs [#which-runtimes-a-host-runs]

There is no configuration key. A host build of `bcd node` runs each built-in runtime whose tool is on its `PATH` when
it starts (`newRuntimes` in `cmd/bcd/hardware_host.go`), then adds bcx plugin runtimes:

| Runtime              | Turned on by                                                                                       | Files under                    | Id (`InstanceSpec.runtime`) |
| -------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------ | --------------------------- |
| `libvirt-kvm`        | `virsh` on the `PATH`                                                                              | `data_dir/libvirt`             | `libvirt-kvm`               |
| `cloud-hypervisor`   | `cloud-hypervisor` on the `PATH`                                                                   | `data_dir/cloud-hypervisor`    | `cloud-hypervisor`          |
| A bcx plugin runtime | A hosted driver plugin whose manifest lists the kind `runtime`, with an id no built-in runtime has | The plugin's own               | The driver id               |
| The fake runtime     | A fakehw build (`go build -tags fakehw`), always                                                   | `data_dir/fakehw-runtime.json` | `libvirt-kvm`               |

Only node agents run runtimes; `bcd site` and `bcd ramdisk` run none. Every runtime the
node runs is advertised in `Hello.drivers` with kind `runtime`, which the gateway turns into the node label
`bc.dev/driver.<runtime>`. vps places a hosted instance only on hosts that carry its runtime's label (0fca413). If a
spec names a runtime this host lacks anyway, that instance fails with `runtime <id> is not installed on this host`.
A host with no runtime at all fails a non-empty `instances` section terminally with `this host runs no instance
runtime`, and `bcd doctor` warns ("neither virsh nor cloud-hypervisor: this host runs no VPS instances").

All volumes share one store, `data_dir/volumes`, whatever the runtime (see [volumes](#volumes)).

## The Runtime interface [#the-runtime-interface]

Every runtime implements `runtime.Runtime` (`daemon/internal/runtime/runtime.go`):

| Method                                          | Does                                                                                                                                                                                                                              |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Name()`                                        | The driver id                                                                                                                                                                                                                     |
| `List(ctx)`                                     | Every domain the runtime holds, managed or not, with its state, resources, addresses, MACs, GPUs and open volumes                                                                                                                 |
| `Ensure(ctx, spec, userData, restore)`          | Defines the domain for a spec, or updates an existing one (resources, NICs, GPUs). It never starts it. `userData` is the NoCloud user data for a new domain; `restore` is a downloaded, verified snapshot a new domain boots from |
| `SetState(ctx, id, state)`                      | Starts, stops, pauses or resumes a managed domain (`running`, `stopped`, `paused`)                                                                                                                                                |
| `Snapshot(ctx, id, includeMemory, cpuModel, w)` | Writes a snapshot tar of a managed domain to `w`, and answers the CPU model a memory state depends on                                                                                                                             |
| `Destroy(ctx, id)`                              | Removes a managed domain with its disks and taps                                                                                                                                                                                  |

Optional interfaces add what only some runtimes do: `Console` (a serial console), `Rebind` (the [identity
rebind](/docs/daemon/delivery#identity-rebind) disk), `VolumeGrower` (tell a running guest a volume grew), the
balloon's size and usage readers, `SelfAdmitting` (a runtime that checks capacity itself, so bcd's admission leaves
it alone) and `NormalPagesRestore` (memory restores run on normal pages).

Two rules hold for every runtime:

* **Ownership.** A domain is `Managed` only when bcd created it. Every other domain is listed so the control plane can
  report the leak, and is never defined over, started, stopped or destroyed.
* **Tap names.** NIC `i` of instance `id` lives on the host interface `runtime.TapName(id, i)`: `bc`, the first 10 hex
  digits of the SHA-256 of the instance id, `n`, then the index (it fits `IFNAMSIZ`). bcd's nftables rules match taps
  by name, so they exist before the tap does.

## libvirt-kvm [#libvirt-kvm]

`runtime.Libvirt` drives `virsh -c qemu:///system` and `qemu-img`. Each domain it defines carries bcd's metadata
(namespace `https://better-compute.dev/libvirt/1`: allocation, image digest and format, addresses, the snapshot it
was restored from), which is how bcd tells its own domains apart.

| Concern | What `libvirt-kvm` does                                                                                                                                                                                                                                                                  |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Disk    | A qcow2 overlay `instances/<id>/disk.qcow2` on a base image downloaded once over https, verified against its SHA-256, checked as a [tenant image](/docs/daemon/delivery#tenant-images) and kept as `images/<sha256>.<format>`. An `iso` image boots from a cdrom onto a blank qcow2 disk |
| Grow    | `qemu-img resize` while stopped, `virsh blockresize` while running; a smaller size fails the instance                                                                                                                                                                                    |
| NICs    | virtio NICs on the spec's bridges, on taps named `TapName`                                                                                                                                                                                                                               |
| CPU     | `host-passthrough`, so a memory snapshot restores only on the same CPU model                                                                                                                                                                                                             |
| States  | `start` (`--paused` for a paused start), `suspend`, `resume`. A stop is `virsh shutdown`, then `virsh destroy` after 60 seconds (`ShutdownTimeout`). A start that fails with "Cannot allocate memory" is a capacity failure                                                              |
| Delete  | `destroy` and `undefine`, then the instance directory is removed                                                                                                                                                                                                                         |
| Console | The pty libvirt gives the domain's serial port                                                                                                                                                                                                                                           |

Running as root, bcd adds `o+x` (search, never read) to each directory from `data_dir` down to a domain's disks,
seed, volumes and base image, so QEMU's own user (`libvirt-qemu` or `qemu`) reaches them while listing stays closed
(f7b8805). Under the helper the model is different (below).

fakehw builds replace the runtime with a file-backed fake under the same id. It writes the domain XML bcd would define
on a real host to `data_dir/fakehw-plans/<id>.xml` (or `<id>.error` when the spec would be refused) through
`runtime.RenderLibvirt`, so tests can assert pinning, NUMA binding, huge pages, the balloon, volumes and GPU addresses
that the fake cannot run.

## cloud-hypervisor [#cloud-hypervisor]

`runtime.CloudHypervisor` runs one cloud-hypervisor VMM per instance in a systemd transient unit
`bcd-ch-<id>.service` in `bcd-instances.slice` (started with `systemd-run --collect`, so a VM outlives bcd), and drives
it over its REST API on `instances/<id>/api.sock`. cloud-hypervisor keeps no registry, so `instances/<id>/vm.json`
is bcd's record: the allocation, image and base, the VMM's uid, addresses, networks, MACs, GPUs, placement, volumes,
the VM configuration, and what was handed to the running VMM (VFIO groups, bound PCI addresses, volumes). A directory
without that record, marked `better-compute/cloud-hypervisor/1`, is not bcd's.

| Concern  | What `cloud-hypervisor` does                                                                                                                                                                                                                                     |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Firmware | Boots `/usr/share/cloud-hypervisor/CLOUDHV.fd`; `bcd doctor` warns when it is missing                                                                                                                                                                            |
| Disk     | The same verified and checked qcow2 overlays as `libvirt-kvm` (it reuses that image code over its own directory), `image_type` stated for each disk, `backing_files` only for bcd's own overlay. An `iso` image is refused                                       |
| NICs     | Persistent taps named `TapName`, created and bridged by bcd before the VMM starts (933c51b). Since b12c217 bcd opens each tap and hands its file descriptor to the VMM (`vm.add-net`, or `net_fds` on a restore), so the VMM never names a tap                   |
| Identity | Each VMM runs as its instance's own uid from `0x7FFD0000` up, recorded in `vm.json`, with no capabilities and no network (see [the cloud-hypervisor unit](/docs/daemon/delivery#the-cloud-hypervisor-unit))                                                      |
| Grow     | `qemu-img resize` while stopped, `vm.resize-disk` while running                                                                                                                                                                                                  |
| States   | `vm.create` and `vm.boot` (then `vm.pause` for a paused start), `vm.pause`, `vm.resume`. A stop is the ACPI power button for up to 60 seconds, then `vm.shutdown`, `vmm.shutdown` and `systemctl stop`, and fails unless the unit is then `inactive` or `failed` |
| Delete   | Stops the VM first, so a running VMM's directory is never removed, then removes its taps and directory                                                                                                                                                           |
| Console  | The serial port's unix socket, `instances/<id>/serial.sock`                                                                                                                                                                                                      |

It needs systemd 247 or later, which `bcd doctor` checks, and a cloud-hypervisor whose API takes NIC file descriptors.

## bcx plugin runtimes [#bcx-plugin-runtimes]

A bcx driver plugin whose manifest lists the kind `runtime` becomes a runtime on `bcd node` (`agent.PluginRuntime`,
[walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx)). It speaks the TypeScript
`RuntimeDriver` contract over `Invoke`, with positional JSON arguments:

| Operation | Arguments                                     | Answer                                 |
| --------- | --------------------------------------------- | -------------------------------------- |
| `list`    | none                                          | `[{ id, state, addresses, message? }]` |
| `apply`   | core's `InstanceSpec` with its `desiredState` | `{ id, state, addresses, message? }`   |
| `remove`  | the instance id                               | `null`                                 |

`Ensure` only records the spec; `SetState` sends `apply` with the desired state, so the plugin both defines and starts.
A new instance's user data and a restore (as a local `file://` URL of the verified snapshot) go with its first
`apply` only. The idempotency key is `instance:<id>:apply:<state>` or `instance:<id>:remove`. Everything a runtime
plugin lists counts as bcd's.

What bcd passes a plugin, and what it does not:

* **Passed** (b8a1801): each NIC's tap name, gateway and nameservers, the network policy, `sshFromHostOnly`,
  `identityRebindRef`, the placement, and each volume's path and serial in attach order.
* **Enforced by bcd.** When the tap rules guard an instance (a `PRIVATE` or `NONE` policy, or SSH from the host
  only) and a running plugin instance has no interface under one of its tap names, bcd stops and fails it (`runtime <id> did not create taps …: its network rules cannot apply`).
* **Not supported.** Snapshots (`Snapshot` answers `unsupported`, so no plugin snapshot exists to restore from,
  although a restore's verified file would be passed on); the identity rebind and `identity_rebound_ref`; consoles.

Plugins need root to start, so an unprivileged node under the helper hosts none (see [bcd helper](/docs/daemon/helper)).

## The fake runtime [#the-fake-runtime]

`runtime.Fake` keeps its domains in a JSON file, so they outlive the daemon process. A new boot id (a host reboot)
stops every domain, as it would stop real VMs. It honors optional vCPU and memory limits itself (`AdmitsItself`, so
bcd's host admission skips it), refuses a disk that shrinks, refuses a snapshot from another runtime and a memory
snapshot from another CPU model, refuses a memory snapshot beside a read-write volume, echoes on its "console", and
acts as a compliant guest for the identity rebind. The TypeScript `FakeHypervisor` follows the same rules, which is
what `better-compute dev --sim` and the simulator run.

## Passthrough GPUs [#passthrough-gpus]

An instance's `gpu_device_ids` are GPU UUIDs. Both built-in runtimes find each GPU's PCI address from the inventory's
bus id (a GPU already on `vfio-pci` is found by its remembered address), and an instance whose GPU has none fails.

| Step             | `libvirt-kvm`                                                                | `cloud-hypervisor`                                                                                                                                                                                                                                                                       |
| ---------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Binding          | Managed PCI `<hostdev>`: libvirt rebinds the device itself                   | bcd binds every endpoint of the GPU's IOMMU group to `vfio-pci` before the start (`driver_override`, unbind, `drivers_probe`), the GPU's audio function included; bridges stay on `pcieport` (be4de2a). It checks the devices moved, so a host without `vfio-pci` loaded fails the start |
| Unbind bound     |                                                                              | An unbind waits at most 30 seconds (`UnbindTimeout`); past that the start fails rather than block every section                                                                                                                                                                          |
| Persistence mode | Off for the domain's GPUs before a start, on again after a destroy (5bb9f22) | Off before a GPU leaves `nvidia`, on again once it is back (f25abbe)                                                                                                                                                                                                                     |
| Giving back      | libvirt reattaches on destroy                                                | A stop gives the groups back to root; a destroy hands the devices back to their host driver, even when the spec dropped a GPU meanwhile                                                                                                                                                  |

Persistence mode matters because `nvidia-persistenced` (which the `nvidia-driver` [host
unit](/docs/daemon/host-config#nvidia-driver) keeps running) holds every GPU open, and the driver's unbind waits until
the GPU is closed. Both runtimes refuse a memory snapshot of an instance with passthrough GPUs, since the GPU's state
lives in the device (`an instance with passthrough GPUs cannot be snapshotted with its memory; snapshot its disk
only`). The host's IOMMU and `vfio-pci` come from the [iommu](/docs/daemon/host-config#iommu) and
[kernel-modules](/docs/daemon/host-config#kernel-modules) units, and `bcd doctor` warns when `vfio-pci` is missing on
a host with an active IOMMU.

## NoCloud seeds [#nocloud-seeds]

User data never travels in the spec. The converger fetches it with a `SecretRequest` bound to the allocation, and the
runtime writes it into a NoCloud seed, built with `genisoimage` (`-volid cidata -joliet -rock`), else `xorriso -as
mkisofs` with the same options:

| File             | Content                                                                                                                                                                                                                                                                                                                                                                                           |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user-data`      | The fetched user data, or `#cloud-config` with an empty object when there is none                                                                                                                                                                                                                                                                                                                 |
| `meta-data`      | `instance-id: <id>` and `local-hostname: <id>`                                                                                                                                                                                                                                                                                                                                                    |
| `network-config` | Only when a NIC has leased addresses, a gateway or nameservers (0791fc9): cloud-init network config version 2, each NIC matched by MAC, its CIDR addresses static with DHCP off, a default route through its gateway, and its nameservers. A NIC without addresses keeps `dhcp4: true`. A NIC without a valid MAC beside static NICs fails the seed, since cloud-init would leave it unconfigured |

`libvirt-kvm` attaches `seed.iso` as a read-only cdrom; `cloud-hypervisor` attaches the same file as a read-only raw
disk. A restored instance keeps the seed in its snapshot, attached as raw and never probed. `bcd doctor` fails a VPS
host without either `genisoimage` or `xorriso`.

## Placement [#placement]

`InstanceSpec.placement` (from a flavor's `placement`, see [flavors](/docs/guides/vps/flavors#placement)) is checked
before anything is created (`runtime.PlacementOf`), and a placement the host cannot apply fails the instance:

| Field                 | Rule                                                                                                                                                                                                 |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpus`                | A kernel cpulist with at least as many CPUs as vCPUs. With exactly one CPU per vCPU, vCPU `i` is pinned to the `i`-th CPU; otherwise the vCPUs float within the set. A range spans at most 4096 CPUs |
| `mems`                | A cpulist of NUMA nodes the memory is bound to                                                                                                                                                       |
| `hugepage_size_bytes` | 0, 2 MiB or 1 GiB, and the memory must be a whole number of pages                                                                                                                                    |
| `balloon`             | A virtio balloon, dropped when huge pages back the memory                                                                                                                                            |

| Placement  | `libvirt-kvm`                                          | `cloud-hypervisor`                                                     |
| ---------- | ------------------------------------------------------ | ---------------------------------------------------------------------- |
| CPUs       | `vcpu cpuset` (floating) or `cputune vcpupin` (pinned) | `cpus.affinity`, and `AllowedCPUs` on the VMM's unit                   |
| NUMA       | `numatune` in strict mode                              | `AllowedMemoryNodes` on the VMM's unit                                 |
| Huge pages | Locked `memoryBacking` huge pages                      | `memory.hugepages`; the unit's `MemoryMax` drops to the VMM's overhead |
| Balloon    | An explicit `memballoon` (`none` unless asked)         | A balloon with free page reporting                                     |

A restore keeps bcd's pinning, never the snapshot's. A `cloud-hypervisor` memory restore runs on normal pages whatever
the placement says, because its saved configuration keeps no huge pages; admission charges it that way
(`RestoresOnNormalPages`). The balloon controller that resizes balloons under host memory pressure is on [memory
ballooning](/docs/daemon/delivery#memory-ballooning). bcd advertises `instance.placement` in `Hello` on hosts with a
runtime, and vps sends placements only there.

## Admission [#admission]

Before a runtime starts a domain that is not running (a resume from `PAUSED` is not a start), bcd checks that the host
can give it its memory (0dd122a), counting what is committed rather than used: running and paused managed domains on
normal pages with their VMM overhead (1/32 of the memory plus 256 MiB), bound holds' memory limits, and starts
already admitted in the same pass, against `MemTotal` less the huge page pool and 1 GiB of headroom. An instance
with huge pages is checked against free pages of that size on its NUMA nodes and host-wide (aeaa982). A start that
does not fit stays stopped with `ResourcesExhausted: …` in its status, is retried on the next pass, and is not an
`ERROR`. The rules and what vps does with the refusal are on [instances](/docs/daemon/delivery#instances).

A runtime that reports `ErrNoCapacity`, or a `libvirt-kvm` start that fails for lack of memory, is a capacity failure
of that instance. A runtime that admits itself (the fake) is left alone, and a host whose `/proc/meminfo` cannot be
read starts without the check, with a warning.

## Volumes [#volumes]

Persistent volumes are sparse raw files in `data_dir/volumes/<id>.raw`, shared by every runtime on the host and
attached by serial (`/dev/disk/by-id/virtio-<serial>` in the guest), `vdb` onward after the root disk:

| Runtime            | Attached as                                                                                                                                                                   |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `libvirt-kvm`      | Raw virtio disks, read-only when asked                                                                                                                                        |
| `cloud-hypervisor` | `Raw` disks with their serial, bound into the VMM's unit; a read-write one handed to the VMM's user while it runs, a read-only one readable by that user through an ACL entry |
| A plugin           | Path and serial in its spec                                                                                                                                                   |

Growth is applied live through `VolumeGrower` (`virsh blockresize`, `vm.resize-disk`; b5415a2). bcd tracks which
domains have which volumes open and never lets a second writer in, applies stops and deletes before starts, and
never destroys a volume a running VM has open. Both built-in runtimes refuse a memory snapshot beside a read-write
volume (`ErrMemorySnapshotWithVolumes`). The store's rules, space accounting and reports are on [persistent
volumes](/docs/daemon/delivery#persistent-volumes).

## Snapshots and restores [#snapshots-and-restores]

`instance.snapshot` asks the runtime for a tar written to a staging file under `data_dir/snapshots`, which bcd hashes
and uploads; `restore_from` downloads and verifies a snapshot before a new instance boots from it. The transfers are
on [instances](/docs/daemon/delivery#instances); the runtimes' formats are:

| Runtime            | Tar contents                                                                                                                                                                                                                                                                                                            | Restore                                                                                                                                                                                                                                             |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `libvirt-kvm`      | `meta.json` (runtime, CPU model, `includeMemory`), the overlay flattened onto its base (`qemu-img convert`), the seed, and with memory `virsh save`'s state. The domain then carries on with `virsh restore --xml` of bcd's own `domain.xml` (81f9c21). A disk-only snapshot of a running domain pauses it for the copy | Unpacks into the new instance's directory, checks the disk and seed, and starts with `virsh restore` and the new domain's XML                                                                                                                       |
| `cloud-hypervisor` | `meta.json`, the flattened overlay, the seed, and with memory cloud-hypervisor's own snapshot of the paused VM under `ch/` (`config.json`, `state.json`, `memory-ranges`); a running VM is paused and resumed                                                                                                           | The saved `config.json` must have exactly the instance's vCPUs and memory; bcd rebuilds it keeping only CPU, memory and platform keys and supplies payload, disks, NICs, serial, console, rng and devices itself, then `vm.restore` and `vm.resume` |

A snapshot names its runtime in `meta.json`, and the other runtime refuses it. A memory snapshot restores only on the
same CPU model. A memory restore's guest gets its own SSH identity through the [identity
rebind](/docs/daemon/delivery#identity-rebind) (ca08d5c), which both built-in runtimes implement with a hot-added
`vdz` disk.

## Serial consoles [#serial-consoles]

Instance [console tunnels](/docs/guides/access/tunnels#the-node-half) (f8273ef) reach a running, managed instance's
serial port, never the host's shell. `libvirt-kvm` opens the pty libvirt gave the domain's serial port;
`cloud-hypervisor` dials `instances/<id>/serial.sock`; the fake echoes what it is sent, prefixed with `console: `. Under
the helper, root opens the libvirt pty and passes the node its descriptor (`libvirt.console`). Plugin runtimes have
no console.

## Under bcd helper [#under-bcd-helper]

On an unprivileged `bcd node` (`[helper]` set), `forHelper` (`cmd/bcd/agent.go`) wires each built-in runtime to
[bcd helper](/docs/daemon/helper):

| Runtime            | Goes through the helper                                                                                                                                                                                                                             | Stays with the node                                                                                                                                             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `libvirt-kvm`      | Every privileged `virsh` verb, as `libvirt.*` (ed3c7d2): define, the state changes, blockresize, setmem, the rebind disk's attach and detach, save, restore and console; root validates each definition default-deny and acts only on bcd's domains | Reads over libvirt's read-only socket (`virsh -r` for `list`, `domstate`, `dumpxml`, `domstats`; any other verb is refused), its files under `data_dir/libvirt` |
| `cloud-hypervisor` | Taps, VFIO groups and the VMM unit, as `ch.*` (d86fcbe, 7fe74a1); root builds the unit from the instance record it checks                                                                                                                           | Every file, shared with the VMM's uid through POSIX ACL entries (the unit's `UMask` becomes `0007`), and the VMM's API socket                                   |

libvirt then runs with `dynamic_ownership = 0`, and QEMU's user is in the node's group, so instance disks are 0660
(read-only ones 0640) and the directories on the way get group search instead of others' search. Without `[helper]
instance_bridges`, root defines no domain with a NIC and creates no tap. What root checks for each operation is on
[bcd helper](/docs/daemon/helper#libvirt) and in the overview's [libvirt through the
helper](/docs/daemon#libvirt-through-the-helper) and [cloud-hypervisor through the
helper](/docs/daemon#cloud-hypervisor-through-the-helper).

## What the tests cover [#what-the-tests-cover]

| Tests                                                                                                             | Against                                                                  | Covers                                                                                                                                                                                                                                            |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/runtime` `libvirt_test.go`                                                                              | A fake `virsh` and `qemu-img` (writing real qcow2 headers since b12c217) | Domain XML and metadata, states, grow and shrink, snapshot and clone with memory, a CPU mismatch, destroy, a mismatched image digest, base image and overlay checks                                                                               |
| `internal/runtime` `cloudhypervisor_test.go`                                                                      | A fake VMM API and fake `systemd-run`, `qemu-img`, `ip` and seed tools   | Define, states, memory snapshot and restore into a clone, the rebuilt restore configuration, taps before the VMM, the unit's uid and confinement, NIC fds, VFIO groups handed back, a rewritten overlay, a VMM that survives its stop, the rebind |
| `internal/runtime` `placement_test.go`, `netconfig_test.go`, `vfio_test.go`, `volumes_test.go`, `balloon_test.go` | Fakes and temporary files                                                | Placement parsing, network-config rendering, VFIO binding against a fake sysfs, the volume store, the balloon                                                                                                                                     |
| `internal/runtime` `hypervisor_test.go`, `ch_privileged_linux_test.go` and the helper's tests                     | Fake helper sockets and tools                                            | Both runtimes under the helper                                                                                                                                                                                                                    |
| `tests/e2e/bcd-vps.test.ts`, `bcd-placement.test.ts`                                                              | A fakehw bcd's fake runtime and the real control plane                   | Create, start, delete, placement plans and the balloon under simulated pressure                                                                                                                                                                   |

None of these runs a real VMM. Round 6 (2026-10-04) brought bcd onto a real host with an RTX 5090 in WSL, but that
host has no hypervisor installed, so no instance has booted on real KVM.


# Self-update (/docs/daemon/self-update)

> The node side of daemon upgrades, from the signed release manifest and its checks to the A/B slots, the release floor, confirmation, rollback, the crash-loop unit, the helper's half and the bcd update command.



bcd replaces its own binary when the control plane asks for another version (PLAN.md §6 "Self-update", v0.4;
de9f380). It takes a binary only from a release manifest signed by keys pinned on the node, so a compromised control
plane cannot hand it one. The new binary goes into the other of two slots, runs, and must reach a gateway within 120
seconds or it is swapped back. Once a release is confirmed, the node never runs an older one.

The control plane's side, signing releases and rolling a version out in waves within each pool's disruption budget,
is on [rollouts](/docs/guides/rollouts). [Installing bcd](/docs/daemon/install#self-update) summarizes the steps for
operators. This page is the node's side in full.

> **Tested, not yet used on a real fleet.** 
  `tests/e2e/bcd.test.ts` updates a real bcd to a second build signed under the pinned key through `daemon.upgrade`,
  with a real exec and its `Welcome` (fc0ca6a), and delivers a signed build to an unprivileged node through a real `bcd
  helper` (2d67b25). The helper's root end-to-end test swaps, rolls back and records a failure as root (035fc41).
  `internal/selfupdate` unit tests cover the manifest checks, the slots, the floor and the staged-file rules. No bcd
  release has been cut, so no real host has updated itself yet.

## How an update reaches a node [#how-an-update-reaches-a-node]

A node is told its target version in two ways, which carry the same `DaemonSection`:

| Path                     | When                                                                                                                                                                                                                                                     | Kind                                                            |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `NodeSpec.daemon`        | The rollout controller sets it as nodes start, wave by wave                                                                                                                                                                                              | Desired state, converged by the `daemon` section                |
| `daemon.upgrade` command | A node whose daemon is below the server's minimum protocol version gets `Welcome.upgrade_required`; its session accepts only `daemon.upgrade`, so the controller also enqueues one with the same target (idempotency key `rollout:<rolloutId>:<nodeId>`) | `control` lane, retry-safe, self-terminating, 15-minute timeout |

```json title="DaemonSection"
{
  "version": "1.4.0",
  "url": "https://downloads.example.com/bcd/1.4.0/linux-amd64",
  "sha256": "9f2c...e41a",
  "signature": "<manifest.json bytes: { manifest, signatures }>"
}
```

`signature` is the release's signed manifest as `better-compute release sign` wrote it. A section naming the version
the node already runs plans nothing, and a `daemon.upgrade` for it succeeds at once. `bcd sim` and `bcd ramdisk` run
without self-update: the `daemon` section is `SectionUnsupported` there and `daemon.upgrade` is `UNSUPPORTED`.

<Mermaid
  chart="sequenceDiagram
  participant N as bcd node
  participant M as Manager (in process, or bcd helper)
  participant G as Gateway
  N->>M: Prepare(target)
  M-->>N: artifact to download (or none: the other slot holds it)
  N->>N: download to data_dir/staging, size and SHA-256 checked
  N->>M: Swap(target)
  M->>M: verify again, copy into the other slot, upgrade.json, current -> new slot
  N->>N: exec the entrypoint (the new binary)
  N->>M: Settle: await the Welcome
  N->>G: Hello
  G-->>N: Welcome
  N->>M: Confirm: the floor moves up, upgrade.json goes"
/>

## The release manifest [#the-release-manifest]

A release manifest (`type: bettercompute.release`, `spec: 1`) lists one version's files. Its published form is
`{ manifest, signatures }`: the canonical body in base64 and one `{ kid, sig }` per signer (see [sign a
release](/docs/guides/rollouts#sign-a-release)). bcd verifies it in `pki.VerifyReleaseManifest`
(`daemon/internal/pki/release.go`), in the same order as `@better-compute/pki` `release.ts`:

1. **Shape.** At most 1 MiB. The body parses, has exactly the manifest's fields, spelled exactly (so `Version`
   beside `version` cannot read differently in Go and JavaScript), and is well-formed: product
   `^[a-z][a-z0-9-]{0,63}$`, a semver version, a positive sequence, `expiresAt` after `issuedAt`, 1 to 16 distinct
   `ES256` keys, at least one artifact, each with a plain distinct name, an `os-arch` platform, a hex SHA-256 and a size.
2. **Signatures.** At least `threshold` distinct pinned keys signed it under purpose `release-manifest`. A signature
   counts only when its `kid` is a key the manifest lists, that key's SPKI matches a pin, and the P1363 signature
   verifies. Each pin counts once, however often it is configured; at most 16 signatures are read.
3. **Canonical form.** The body is byte for byte its canonical JSON (keys sorted by UTF-16 code units, no whitespace,
   plain integers).

Then `selfupdate.Manager` checks the release against the node:

| Check                                                                                         | Refusal                                                                                               |
| --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| The target is a release version                                                               | `"<v>" is not a release version`                                                                      |
| It is not a version that failed here before                                                   | `bcd <v> did not start; running <running> (<why>)`                                                    |
| The section's SHA-256 is 32 bytes                                                             | `the release sha256 must be 32 bytes`                                                                 |
| Release pins are configured                                                                   | `no release keys are pinned ([update] release_pins): bcd takes no binary on the control plane's word` |
| A manifest is present and verifies                                                            | `no signed release manifest`, or the verifier's reason                                                |
| The product is `bcd`                                                                          | `the manifest releases <p>, not bcd`                                                                  |
| The manifest's version is the target                                                          | `the manifest releases bcd <m>, not <v>`                                                              |
| The sequence is not below the release floor                                                   | `release sequence <n> is older than <floor>, which this node ran or was installed at (rollback)`      |
| With no floor yet (a development build), the version is not older than the running one        | `bcd <v> is older than the running <r> (rollback)`                                                    |
| The manifest has not expired, on the host clock or the node's server time, whichever is later | `the release manifest expired at <time>`                                                              |
| It lists `bcd_<version>_<os>_<arch>` for this platform with the section's SHA-256             | `the manifest lists no bcd_<v>_linux_amd64 with the release sha256`                                   |
| That artifact is at most 512 MiB                                                              | `the binary is <n> bytes; at most 536870912 are taken`                                                |

Every one of these is a terminal refusal (`selfupdate.Refusal`): the `daemon` section goes `FAILED` with reason
`UpdateFailed` and `observed_generation` advances, and `daemon.upgrade` fails with `update-failed`. A download that
fails, or an update already in progress, is retryable instead.

## Where the files live [#where-the-files-live]

| Item                                                                                  | Root node (no helper)                                                                                      | Under bcd helper                                |
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| Slots `bcd-a`, `bcd-b`, links `current` and `previous`, `slots.json`                  | `[update] slots_dir`, else `<data_dir>/bin`                                                                | `[update] slots_dir`, else `/usr/local/lib/bcd` |
| `upgrade.json` (the pending update) and `update.json` (the floor and failed versions) | `data_dir`                                                                                                 | The slots directory                             |
| Staging download                                                                      | `<data_dir>/staging/bcd-<version>`                                                                         | The same; the node writes it                    |
| Entrypoint                                                                            | `[update] entrypoint`, default `/usr/local/bin/bcd`, a symlink to `<slots>/current` after the first update | The same                                        |
| Crash-loop units                                                                      | `/etc/systemd/system/bcd-rollback.service` and `bcd.service.d/50-bcd-rollback.conf`                        | The same, written by the helper                 |

`bcd update` and `bcd doctor` find the slots and records the same way (8eca2b1). Under the helper, `slots_dir` and the
entrypoint may not lie inside `data_dir`, which the node owns (9e71b1f), and every directory on the way to them must be
root's alone or the helper does not start.

`slots.json` records, per slot, the version, the manifest sequence (0 when unknown, such as the adopted install) and
the SHA-256 of the bytes written. A slot's record is cleared before its file is replaced, so a crash in between leaves
no record rather than a record of other bytes.

## Configuration [#configuration]

```toml title="/etc/bcd/config.toml"
[update]
release_pins = "sha256/<...>,sha256/<...>"
release_threshold = 2
release_sequence = 14          # optional
# slots_dir = "/usr/local/lib/bcd"
# entrypoint = "/usr/local/bin/bcd"
```

| Key                 | Default              | Meaning                                                                                                                                  |
| ------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `release_pins`      | none                 | The `sha256/...` SPKI pins of the release keys, comma-separated. Without any, bcd refuses every update                                   |
| `release_threshold` | 1                    | How many distinct pinned keys must have signed: 1 to the number of distinct pins, at most 16                                             |
| `release_sequence`  | 0                    | A floor this node starts from; the effective floor is the highest of this, the binary's embedded sequence and the last confirmed release |
| `slots_dir`         | see above            | Absolute path of the slots                                                                                                               |
| `entrypoint`        | `/usr/local/bin/bcd` | Absolute path systemd starts; the first update makes it a symlink                                                                        |

A release build embeds its manifest's sequence (`-X main.releaseSequence=<n>` from `BCD_RELEASE_SEQUENCE` in
`.goreleaser.yaml`), so a freshly installed node already refuses releases older than the one it was installed from.
`daemon install-script --release-pins` writes the section (see [generate the
script](/docs/daemon/install#generate-the-script)).

## The steps [#the-steps]

### Prepare and download [#prepare-and-download]

`Prepare` runs every check above before anything is downloaded. It answers the artifact to fetch, or nothing when the
other slot already holds the target with a known sequence not below the floor and the section's SHA-256: going back to
a binary that ran here before needs no manifest and no download (c139b9c).

The node then downloads the URL, which must be `https://` with a host and no credentials (`http://` only in fakehw
builds or with `dev_insecure_h2c`). Redirects are followed only to https. It writes `<staging>/bcd-<version>.part`,
hashing as it goes, reads at most the manifest's size plus one byte, ends a transfer that moves too little in its idle
window, and keeps the file only when the size and SHA-256 are exactly the manifest's.

### Swap [#swap]

`Swap` refuses when an update is already pending, then:

1. **Verifies again**, with the same checks as `Prepare`.
2. **Adopts the running binary** on the first update: copies the entrypoint's binary into slot `a` (creating each
   missing directory 0755 whatever the umask), points `current` at it and replaces the entrypoint with a symlink to
   `<slots>/current`. An entrypoint that is a symlink somewhere else is refused (`set [update] entrypoint`). On later
   updates, a plain file found at the entrypoint (an install script's `mv -f`) is copied into the current slot first,
   so the relink never falls back to an older binary (9006eee). Running as root, adoption takes only a root-owned
   regular file that group and others cannot write (9e71b1f).
3. **Writes the crash-loop units** when bcd runs under systemd (not in fakehw builds).
4. **Stages the binary** into the other slot. It opens `bcd-<version>` beneath the staging directory's parent with
   `os.Root`, refuses a symlinked staging directory, opens with `O_NOFOLLOW|O_NONBLOCK` (a FIFO cannot hang it), and
   takes only a regular file with one link; under the helper it must also be owned by the node's uid (c139b9c). It
   copies to `bcd-<slot>.tmp`, checking size and SHA-256 as it writes, so what is checked is what runs, then
   fsyncs, sets 0755 and renames.
5. **Records and swaps**: `upgrade.json` with `state: swapped`, the versions, the slots, the manifest sequence, the
   `daemon.upgrade` command id if any and a deadline of now plus 120 seconds, fsynced; then `previous` points at the
   old slot and `current` at the new one, each by an atomic symlink rename.

The agent then stops, closes everything, and `main` execs the entrypoint with the same arguments.

### Settle and confirm [#settle-and-confirm]

Every start of the agent calls `Settle`, which decides from where `current` points, never from the version the
process claims (review 11):

| `upgrade.json` | `current`                         | Action                                                                                                                                                   |
| -------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| None           | any                               | `none`                                                                                                                                                   |
| `swapped`      | the new slot, before the deadline | `await`: the new binary waits for its Welcome, with a timer to the deadline                                                                              |
| `swapped`      | the new slot, past the deadline   | `restart`: swapped back now, and the agent restarts into the old binary                                                                                  |
| any            | the old slot                      | `failed`: the version is recorded in `update.json` as failed with its reason ("bcd X did not start" unless a rollback said why), and `upgrade.json` goes |

On the first `Welcome`, the new binary calls `Confirm`: the release floor in `update.json` rises to the release's
sequence and `upgrade.json` is removed. A pending `daemon.upgrade` then completes `SUCCEEDED` from the new binary. The
helper confirms only while `current` is the pending slot.

### Rollback [#rollback]

An update that is not confirmed goes back three ways:

| Who                    | When                                                                                                                           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| The new binary itself  | No `Welcome` by the deadline: `RollBack` reverts `current`, removes `previous` and restarts                                    |
| `bcd-rollback.service` | The new binary crash-loops before it can: `StartLimitBurst=5` within `StartLimitIntervalSec=600` triggers `OnFailure=`         |
| `bcd helper`           | Under the helper, its own watchdog 30 seconds after the deadline, whatever the node does, then `systemctl restart bcd.service` |

The rollback unit runs only while `upgrade.json` exists with `"state":"swapped"` and `previous` is a link; it copies
`previous` over `current`, resets `bcd.service`'s failed state and starts it:

```ini title="/etc/systemd/system/bcd.service.d/50-bcd-rollback.conf"
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=5
OnFailure=bcd-rollback.service
```

The old binary then settles `failed`: the `daemon` section reports `FAILED` (`UpdateFailed`), `daemon.upgrade` fails
with `update-failed`, and that version is never tried again on this node until an operator runs `bcd update --clear-failed`. `update.json` keeps at most 32 failed versions.

## The release floor [#the-release-floor]

Confirmation is one way. `update.json`'s sequence is the floor, and `Prepare` refuses any manifest below it, so a
confirmed node runs no older release, the one still in its other slot included: that binary is reused without a
manifest only when its recorded sequence meets the floor (c139b9c). A control plane, compromised or mistaken, cannot
push a node back. To undo a bad release that nodes confirmed, sign a newer one with a higher sequence and roll it
forward (see [undo a bad release by rolling forward](/docs/guides/rollouts#undo-a-bad-release-by-rolling-forward)).

## Under bcd helper [#under-bcd-helper]

On an unprivileged node, the privileged half (`selfupdate.Manager`) runs inside [bcd helper](/docs/daemon/helper#self-update)
(2d67b25). The node's `UpdateClient` sends `update.prepare`, `update.swap`, `update.confirm`, `update.rollback` and
`update.settle`; it only downloads into `<data_dir>/staging` and restarts. Root uses its own `[update]` section from its
root-owned `config.toml`, its own clock (the node's server time can only make the expiry check stricter), and its own
records in the slots directory. The running version is the current slot's record in `slots.json` (before the first
update, the helper's own version), never the version the node reports. A terminal refusal comes back with
`refused: true`, so the node treats it as final.

A self-update restarts only the node. The helper keeps running the binary it started with until it restarts, which
works while both speak helper protocol version 1.

## bcd update [#bcd-update]

`bcd update [--config /etc/bcd/config.toml] [--clear-failed]` shows the local state. Run it as root on a helper node,
whose records are root's.

```text
version 1.4.0, release floor 14, 2 release pins (threshold 2)
slots: current b (1.4.0), other a (1.3.2)
failed: 1.4.1 (bcd 1.4.1 was not welcomed by a gateway before its deadline)
```

| Line                                                           | Shows                                                                  |
| -------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `version …, release floor …, … release pins (threshold …)`     | The running version, the effective floor and the pin count             |
| `slots: not in use yet (the first update adopts <entrypoint>)` | Before the first update                                                |
| `slots: current <slot> (<v>), other <slot> (<v>)`              | The slots and their versions                                           |
| `pending: <from> -> <to>, <state>`                             | An update between swap and confirmation, or a rollback not yet settled |
| `failed: <v> (<why>)`                                          | Each version this node will not try again                              |

`--clear-failed` forgets the failed versions (`failed versions cleared`), so a rollout may try them again. It is a
local decision, never the control plane's. `bcd doctor` also reports self-update: it warns without `release_pins`,
fails when the slots are in use but the entrypoint does not link to `current`, and warns about a pending update and
each failed version (2845f41).

## Limits [#limits]

* The node's running workloads are not touched by an update, but the agent restarts, so its session reconnects and the
  gateway replays what was in flight (see [failure modes](/docs/daemon/failure-modes#daemon-crash-or-node-reboot)).
* A node without systemd gets no crash-loop rollback (`bcd does not run under systemd; a crash-looping update is not
  rolled back` is logged); the deadline rollback still works when the new binary runs long enough to fire its timer.
* A downloaded binary is at most 512 MiB.
* Only `bcd node` and `bcd site` update themselves.


# Sessions (/docs/daemon/sessions)

> How a bcd session opens with Hello and Welcome, how epochs fence a stale gateway, the resume rules, heartbeats, liveness batching and reconnect back-off.



A session is one daemon's long-lived `Session` stream to one gateway. At most one session
per node is live at a time, and every session has an epoch: a number that rises each time the node says `Hello`.
The epoch is what keeps a gateway that lost track of the world from routing work to a node it no longer holds.

This page follows a session from open to close. The frames themselves are listed in
[protocol](/docs/daemon/protocol#frames), and every field of every message is in the generated
[protocol reference](/docs/reference/protocol).

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant D as bcd
  participant G as gateway
  participant P as Postgres
  D->>G: TLS handshake with the node certificate
  G->>G: revocation, stream limit, handshake bucket
  D->>G: Hello
  G->>P: one transaction: epoch + 1, route here, resume rules
  G-->>D: Welcome with the epoch
  G-->>D: Ack of replayed results
  G-->>D: DesiredState, if the node is behind
  G-->>D: replay of in-flight commands, in seq order
  D->>G: requested inventory sections, unacked events, status
  loop every 10 s
    D->>G: Heartbeat
  end"
/>

## Opening [#opening]

Before any frame, the gateway admits the stream. It refuses with `UNAVAILABLE` while it is not started, holds no
lease or is draining, and with `RESOURCE_EXHAUSTED` when it holds `maxStreams` sessions (default 5,000) or the
handshake token bucket (200 per second, shared with `Join`) is empty. A certificate whose serial is in the
revocation set is refused `UNAUTHENTICATED`. The daemon retries a refusal with jittered back-off, unless the refusal
says when to come back.

A busy refusal carries a retry pushback, the standard gRPC trailer `grpc-retry-pushback-ms` (in the trailers-only
status or the trailing one), jittered so refused daemons never return in step:

| Refusal                                       | Pushback                                                                                                                                                                |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Handshake bucket empty (`Session` or `Join`)  | Between 250 ms and the time the bucket needs to admit everyone refused this second (refusals / rate, at least 1 s), so a power-on herd spreads over exactly that window |
| More than `maxPendingHellos` Hellos in flight | 250 ms to 2 s: a Hello transaction takes milliseconds                                                                                                                   |
| At `maxStreams`                               | 5 to 30 s, or another gateway sooner                                                                                                                                    |

bcd waits exactly the pushback (`transport.Pushback`: capped at 15 minutes; a missing, malformed or negative value is
ignored, since bcd always retries) instead of its own doubling, and the doubling then starts over; a refused `Join`
uses it as its next delay. The FakeDaemon follows it the same way. Before this, a 10,000-node power-on herd's connect
tail was each daemon's own back-off after repeated bucket refusals, 65 s in the simulator.

The first frame must be `Hello`; anything else closes the stream `INVALID_ARGUMENT`. The gateway checks:

* `Hello.node_id` equals the node id of the certificate's SPIFFE SAN (`PERMISSION_DENIED` otherwise);
* `role` is set and, when the certificate carries a role, matches it (`PERMISSION_DENIED`);
* `boot_id` and `journal_id` are set (`INVALID_ARGUMENT`).

At most 64 Hello transactions run at once per gateway (`maxPendingHellos`); the next one is refused
`RESOURCE_EXHAUSTED`. The bound exists because in a reconnect storm the handshake bucket admits Hellos faster than a
slow database commits them, and an unbounded queue once starved the gateway's own lease renewal until it fenced
itself (see `bench/gateway-streams`).

### Hello [#hello]

[`Hello`](/docs/reference/protocol#hello) is kept small for reconnect storms: inventory travels as hashes, not
content.

| Field                                                                 | Meaning                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node_id`, `role`                                                     | Who is speaking; checked against the certificate                                                                                                                                                                                                                                         |
| `boot_id`                                                             | The kernel boot id, which changes on every boot                                                                                                                                                                                                                                          |
| `journal_id`                                                          | The journal's id, which changes when the journal is lost                                                                                                                                                                                                                                 |
| `daemon_version`, `proto_min`, `proto_max`, `capabilities`, `drivers` | Versions and what the daemon offers; each driver becomes the node label `bc.dev/driver.<id>=<version>`, and each kind it serves `bc.dev/driver-kind.<id>.<kind>=1` (14ceb1e)                                                                                                             |
| `last_epoch`                                                          | The last epoch the daemon saw; diagnostics only                                                                                                                                                                                                                                          |
| `applied_generation`, `applied_spec_hash`                             | The newest desired state it holds, and that spec's hash                                                                                                                                                                                                                                  |
| `unacked_results`                                                     | Terminal command results the control plane has not acknowledged                                                                                                                                                                                                                          |
| `running_command_ids`                                                 | Commands this daemon process still has accepted or running                                                                                                                                                                                                                               |
| `last_event_seq`                                                      | The highest event sequence written in this journal                                                                                                                                                                                                                                       |
| `inventory_hashes`                                                    | Inventory section name to the SHA-256 of its content                                                                                                                                                                                                                                     |
| `trust_version`                                                       | The version of the command-signing trust bundle the daemon holds and verifies with (0 for none)                                                                                                                                                                                          |
| `approver_set_epoch`, `approver_set_sha256`                           | The approver set the daemon enforces: its epoch and the SHA-256 of its canonical fields, empty without approver keys (fields 18 and 19; 856c00c). A daemon can take a bundle's version while refusing its quorum part, so `trust_version` alone does not say which approvers it enforces |

bcd builds it from the journal (the rules of `session.NewHello`): done results not yet acked, and entries of the
current daemon process that are not done. After a daemon restart the process is new, so `running_command_ids` is
empty and the crash rules below apply instead.

`capabilities` are free-form names the daemon claims for itself. The gateway writes them, as sent in each `Hello`,
to `node_sessions.capabilities` (migration `core/0024_session_capabilities`): only names matching
`[a-z0-9][a-z0-9._-]{0,63}`, de-duplicated, sorted, at most 64. A daemon that stops advertising a capability loses it
at its next session. The API reads the column to refuse what the connected daemon cannot do safely:

| Capability             | Checked by     | Means                                                                                                                                                                                                                             |
| ---------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tunnel.instance-dial` | `tunnels.open` | The daemon dials an instance's own address for instance `ssh` and `tcp` tunnels. Without it those tunnels are `UNSUPPORTED` with details `{ reason: 'daemon-capability', capability }`; `exec` and node consoles are not affected |

On a node with GPUs, bcd advertises the capabilities `cdi`, `devices.limits` and `binding.kubernetes` (409f783),
`nvml` when its inventory reports a CUDA driver version, and the driver `nvidia`, and a node agent
lists each built-in instance runtime it runs as a driver of kind `runtime` (`libvirt-kvm`: on a host build with
`virsh`, the libvirt runtime of 362317c, and on a fakehw build its file-backed runtime; `cloud-hypervisor`: on a host
build with the `cloud-hypervisor` binary, 2598f0a); a runtime a bcx plugin serves
is listed once, through its plugin's manifest. A node agent that runs an instance runtime also advertises
`tunnel.instance-dial`, because it then dials an instance ticket to the instance's own address, never the host's (see
[tunnels](/docs/guides/access/tunnels#the-node-half)); a host without a runtime does not. The FakeDaemon
advertises it when its tunnels are configured.

A node agent with a runtime also advertises what it enforces for instances. The gateway writes every capability as
the system label `bc.dev/cap.<name>=1` at each `Hello`, and drops the ones no longer advertised; vps then places an
instance that needs one only on nodes carrying its label, so an older bcd that would drop the field never gets it:

| Capability                | Label vps requires it for                                                                 |
| ------------------------- | ----------------------------------------------------------------------------------------- |
| `instance.network-policy` | An instance with a `private` or `none` network (cfdb918)                                  |
| `instance.ssh-host-only`  | Sandbox and warm instances, which set `ssh_from_host_only` (82f4f51)                      |
| `instance.placement`      | An instance whose flavor has a `placement` (82f4f51)                                      |
| `instance.volumes`        | An instance that attaches [sandbox volumes](/docs/guides/vps/sandboxes#volumes) (82f4f51) |

The FakeDaemon advertises all four when it plays a VPS host.

A GPU host's bcd advertises what it enforces for device assignments the same way (409f783), and capacity's placement
requires the label on create, re-place, repair and `capacity.preview`:

| Capability           | Label capacity requires it for                                                                                                                                                                                                                         |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `devices.limits`     | A reservation that books host resources (`vcpu`, `memoryGiB`, `hugepages1Gi`, or the pool's per-GPU share): bcd enforces `DeviceAssignment.limits` as the hold's cgroup slice. See [host resources](/docs/guides/capacity/reservations#host-resources) |
| `binding.kubernetes` | A reservation with `binding: 'kubernetes'`: bcd hands the devices to the kubelet and reports their release only once no pod holds them. See [older daemons](/docs/guides/capacity/kubernetes#older-daemons)                                            |

An older bcd drops both fields as unknown, so these keep it from getting a reservation it would run without limits or
never hand to the kubelet. A plain GPU hold without host resources, and a `node` binding, need neither, so they still
place on older daemons (N-2 skew). Capacity counts a `bc.dev/` label only when its source is `system`: an operator
label of the same name does not stand in for the capability. The FakeDaemon advertises both for GPU hosts unless a
skew profile or `capabilities` gives its list explicitly.

### The Hello transaction [#the-hello-transaction]

One transaction, fenced by the gateway's lease, does everything the reconnect proves:

1. **Certificate.** When the certificate serial is recorded in `node_certs`, a revoked certificate, or one from an
   older identity generation of the node, is refused `UNAUTHENTICATED`. Its `allowed_kinds` restrict what this
   session may receive.

2. **Previous session.** The node's `node_sessions` row is locked. A node without a row has not joined
   (`PERMISSION_DENIED`).

3. **Epoch bump and routing.**

   ```sql
   UPDATE node_sessions SET gateway_id = $gw, session_id = $s, session_epoch = session_epoch + 1,
     boot_id = $boot, journal_id = $journal, role = $role, daemon_version = $v, proto_max = $max,
     trust_version = $trust, capabilities = $caps,
     approver_set_epoch = $setEpoch, approver_set_sha256 = $setSha256,
     acked_event_seq = CASE WHEN journal_id IS DISTINCT FROM $journal THEN 0 ELSE acked_event_seq END,
     connected_at = compute_now(), last_seen = compute_now(), disconnected_at = NULL
   WHERE node_id = $node
   RETURNING session_epoch, acked_event_seq
   ```

4. **Resume rules**, below: self-terminating commands, the lost journal, and the replayed results.

5. **Replay set.** Every `dispatched`, `acked` or `running` command of the node moves to the new epoch. Those in
   `running_command_ids` are not re-sent (they still hold lane credits on the daemon); the rest are re-sent, in
   `seq` order, with their delivery `attempt` raised.

6. **Connectivity.** The node becomes `connected`, and `node.connected` is published with the epoch, the boot id
   and the flags `journalChanged`, `bootChanged` and `duplicateIdentity`.

7. **Inventory.** The hashes are recorded, and the sections whose hash changed or was never seen become
   `Welcome.request_sections`.

If an older session of the same node is live on this gateway, it is closed `ABORTED` ("superseded by a newer
session") without a disconnect write: its row already belongs to the new session.

### Welcome [#welcome]

The gateway's answer, [`Welcome`](/docs/reference/protocol#welcome):

| Field                      | Default                        | Meaning                                                                                                                                                                                            |
| -------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `epoch`                    |                                | `node_sessions.session_epoch` after the bump                                                                                                                                                       |
| `session_id`, `gateway_id` |                                | For logs and correlation                                                                                                                                                                           |
| `protocol_version`         |                                | The negotiated version                                                                                                                                                                             |
| `heartbeat_interval_ms`    | 10,000                         | The heartbeat period                                                                                                                                                                               |
| `lease_ttl_ms`             | 40,000                         | The node's liveness lease                                                                                                                                                                          |
| `credits`                  | control 8, provision 4, bulk 2 | Commands the gateway keeps in flight per lane ([`LaneCredits`](/docs/reference/protocol#lanecredits))                                                                                              |
| `acked_event_seq`          |                                | Cumulative ack for `Hello.journal_id` (0 after a journal change)                                                                                                                                   |
| `request_sections`         |                                | Inventory sections to send now                                                                                                                                                                     |
| `server_time`              |                                | `compute_now()`; the daemon keeps the offset to its own clock                                                                                                                                      |
| `trust`                    |                                | A newer command-signing trust bundle ([`SignedTrustBundle`](/docs/reference/protocol#signedtrustbundle) of a [`TrustBundle`](/docs/reference/protocol#trustbundle)), signed by the JoinService key |
| `upgrade_required`         | false                          | The daemon is below the minimum protocol; only `daemon.upgrade` follows                                                                                                                            |
| `capabilities`             |                                | What the gateway offers                                                                                                                                                                            |

An upgrade-only session (`upgrade_required`) is still dispatched to, but only `daemon.upgrade`: `Hello` replays an
in-flight upgrade, the dispatch loop sends a pending one, every other command stays `pending` until the daemon
rejoins at a supported version, and no spec is sent. This is how a rollout reaches a daemon too old for the current
protocol (see [rollouts](/docs/guides/rollouts#daemonupgrade-for-restricted-sessions)).

Right after `Welcome`, the gateway sends an `Ack` for the replayed results it stored. Then, unless the session is
upgrade-only, a full `DesiredState` when the node's generation is above `applied_generation` (or the node has a
spec and the daemon reported no spec hash), then the replay. Commands the certificate may not run are cancelled
`kind-not-allowed` instead of replayed. A WebSocket session skips unsigned commands and withholds unsigned specs; with
`pki.commandSigning` the NodeSpec controller signs every generation, and the gateway relays the stored signed bytes
(`nodes.spec_body`, `nodes.spec_signature`) verbatim, so WebSocket daemons receive their specs. bcd refuses to dial a
WebSocket session at all without command-signing roots, renews its certificate over `POST /v1/renew` on the same
listener instead of the mTLS `Renew`, and opens no tunnels in that mode (see
[the WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04)).

The daemon applies `Welcome` (`session.ApplyWelcome`): it records the epoch as `last_epoch`, forgets events up to
`acked_event_seq`, keeps the clock offset, sets its lane credits, and sends the requested inventory sections, its
unacked events and its latest status. It evaluates every `not_after` in server time through that offset.

`server_time` is not signed, so the daemon never lets it move far backwards. The journal keeps a server time mark,
advanced at every `Welcome` and heartbeat; a `Welcome` whose `server_time` lies more than 5 minutes before the mark
ends the session ("server time went backwards"). Otherwise a gateway could set the daemon's clock back into an
expired worker certificate's window and replay old signed commands. bcd and the FakeDaemon both enforce it, and bcd
also keeps the mark in `identity.json` so that a lost journal does not reset it. A `Welcome` inside the 5-minute
allowance is accepted, but it never moves the daemon's working clock back: both take the latest of the `Welcome`'s
time, the stored mark and the clock the process already runs on. Otherwise a gateway could reconnect "a few minutes
earlier" and replay an expired command, or a tunnel ticket whose `jti` was already pruned; bcd also keeps spent
ticket ids 5 minutes past their expiry. A `trust` bundle is accepted only
when it verifies under the pinned join-signing key and its version is newer than the one held.

The daemon acknowledges a newly applied bundle in its next [`StatusReport`](/docs/reference/protocol#statusreport)
(`trust_version`), not at the next
reconnect. The gateway stores the `Hello` value in `node_sessions.trust_version` as is, even when it is lower than
before (a restored disk), and a status report only ever raises it; `pki rotate-command-root --stage switch` waits
until every connected node holds the new version (see [PKI](/docs/operations/pki)). Both daemons send the version in
`Hello`. The FakeDaemon repeats it in every status report; bcd sets it only in the first report it sends after
applying a newer bundle and leaves it 0 (unchanged) otherwise, which spares the gateway a `node_sessions` update per
report. Whenever a report carries `trust_version`, it also carries the held approver set (`StatusReport` fields 10
and 11; e171f0d, 5fe7cbf). The gateway writes the set to `node_sessions.approver_set_epoch` and
`approver_set_sha256` (fa39201): from `Hello` as reported, and from such a report only when its `trust_version` is at
least the recorded one, so a late, older report never pairs an old set with a newer version. An empty digest (no
approver keys held) is stored as NULL in both. Doctor's `pki.quorum-lag` and `pki trust-version` read the columns
(see [nodes that hold another set](/docs/operations/pki#changing-the-approver-set)).

## Epochs and fencing [#epochs-and-fencing]

The epoch splits writes into two kinds.

**Routing and liveness writes are fenced.** Marking a command dispatched, flushing `last_seen` and writing a
disconnect all carry `AND session_epoch = $myEpoch`. A gateway whose node has since said `Hello` elsewhere matches
no row: its liveness flush tells it the session is a zombie, and it closes the stream `ABORTED`. Dispatch locks the
node's session row `FOR SHARE` while it marks commands, so a dispatch and a concurrent `Hello` epoch bump
serialize instead of racing. As a backstop, every `Hello` replays all in-flight rows anyway.

**Facts are not fenced.** Results, events and status reports are true no matter which session carried them, so
they are accepted from any epoch. Results are deduplicated by command id through the command state table, events
by `(node_id, journal_id, seq)`, and `observed_generation` only moves forward. The daemon records the epoch but
never fences on it.

On top of the per-node epoch, every routing transaction first re-checks the gateway's own `gateway:<id>` lease; see
[the gateway](/docs/daemon/gateway#the-lease-and-fencing).

## Resume rules [#resume-rules]

`Hello` settles what the reconnect proves before anything is replayed.

| What changed since the previous Hello      | Rule                                                                                                                            | Outcome                                                                                           |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `boot_id`, or the role                     | In-flight self-terminating commands (`node.reboot`, `boot.kexec`, `daemon.upgrade`) completed by the reboot or exec they caused | `succeeded`, reason `rebooted`; never replayed                                                    |
| `journal_id` (a previous one was recorded) | In-flight commands that are not retry-safe cannot be deduplicated any more                                                      | `failed`, reason `journal-lost`; never replayed                                                   |
| nothing, after a bcd self-update           | A `daemon.upgrade` restarts the process without a reboot, so the daemon itself settles it from `upgrade.json` (de9f380)         | `succeeded` from the new binary once welcomed, or `failed` with `update-failed` after a swap back |
| nothing                                    | `unacked_results` are applied, then the rest is replayed                                                                        | the journal deduplicates                                                                          |

Both rules wake the owning controllers, which decide what to do next; a workflow step whose command failed
`journal-lost` is re-decided, not blindly retried. A journal is lost on a ramdisk reboot, a reinstall, or
corruption. The node's identity stays with its key, so the node reconnects as itself with a new `journal_id`, and
the event ack restarts at 0 for the new journal.

The daemon has its own half of these rules, applied at start and on every duplicate delivery; see
[the journal](/docs/daemon/delivery#the-journal).

### Duplicate identity [#duplicate-identity]

A second live `Hello` for a node from another `boot_id` wins: the newer session takes the row, `node.connected`
carries `duplicateIdentity: true`, and `security.node.duplicate-session` is published with `nodeId`, `gatewayId`,
`previousBootId`, `bootId`, `alternations`, `windowSeconds` and `quarantined`. `DuplicateIdentity` is an
inventory-owned condition. A prior session that already went quiet (outside the live window) is superseded without
counting.

The Hello transaction counts these alternations in `node_sessions.duplicate_hellos` and `duplicate_since`
(`core/0016`). When the same certificate alternates between machines more than 3 times within 10 minutes, the gateway
quarantines it:

1. the presented certificate's serial is revoked with `revocation_reason = 'duplicate-identity'` (every live
   certificate of the node when no serial was recorded, as on header-authenticated development gateways), so every
   gateway's revocation feed drops the other clone too;
2. `security.node.quarantined` is published with `nodeId`, `reason`, `alternations` and the revoked `serials`;
3. the clone connected to this gateway gets a revocation notice, and the `Hello` is refused `UNAUTHENTICATED`
   ("certificate quarantined: duplicate identity") once the revocation has committed.

The counter resets with the quarantine, so a re-joined machine starts afresh; re-join the real machine with a new
join token. The limits are `createGateway({ duplicateHellos: { max, windowMs } })` (defaults 3 and 10 minutes).

## Liveness [#liveness]

**Heartbeats.** The daemon sends [`Heartbeat`](/docs/reference/protocol#heartbeat) every `heartbeat_interval_ms`
(10 seconds) with its free credits per lane, its load ([`NodeLoad`](/docs/reference/protocol#nodeload): load average,
available memory, running commands, pending events) and its applied generation.
Every 10th heartbeat also carries the inventory hashes, and since 427787a so does the first heartbeat after a
re-collection sent changed sections (a host unit that reserved huge pages, say): the gateway applies an inventory
only once the stored sections and the reported hashes agree, so the change no longer waits up to ten heartbeat
intervals before capacity sees it. The gateway sends an HTTP/2 PING every 20 seconds. Any frame
refreshes the session's last-seen time.

**Three missed heartbeats close the session.** The gateway's liveness loop runs every 5 seconds (half the
heartbeat interval, capped at the 15-second flush period) and closes any session silent for more than three
intervals, `UNAVAILABLE` with reason `heartbeat-timeout`. It also closes a session whose certificate has expired,
`UNAUTHENTICATED` with reason `certificate-expired`.

**No database write per heartbeat.** Last-seen times stay in memory. Every 15 seconds the gateway flushes them in one
statement, as ages rather than timestamps, so a gateway's clock skew cannot fake liveness:

```sql
UPDATE node_sessions s
   SET last_seen = compute_now() - make_interval(secs => v.age_ms / 1000.0)
  FROM unnest($ids, $ages, $epochs) AS v(id, age_ms, epoch)
 WHERE s.node_id = v.id AND s.session_epoch = v.epoch AND s.gateway_id = $gw
RETURNING s.node_id
```

A node missing from the result belongs to a newer session elsewhere, and its stream here is dropped.

**Transitions only.** The gateway writes `connected` at `Hello` and `disconnected` when a session closes. The
Liveness controller in the worker applies the transitions nobody reports:

| Transition                      | When                                                                               | Effect                                                       |
| ------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `connected` to `disconnected`   | The session's gateway lease expired (the gateway crashed or was partitioned)       | Event `node.disconnected`                                    |
| `disconnected` to `unreachable` | 40 s (the lease TTL) plus grace after the last sign of life                        | NoSchedule reason `unreachable`, condition `NodeUnreachable` |
| `unreachable` to `lost`         | After the pool's `lostAfter`, only for `provider` and `vps-host` pools that set it | Never automatic for metal                                    |

The grace is 120 seconds after a gateway loss, so sessions have time to move to another gateway, and 5 minutes after
the database or the Liveness controller itself was down for more than 20 seconds, so ages nobody could flush do not
read as dead nodes. None of these transitions releases anything: allocations, bookings and commands are untouched
(principle P6). See [disconnected is not dead](/docs/guides/concepts/consistency#disconnected-is-not-dead).

## Reconnecting [#reconnecting]

When a session closes for any reason, the daemon redials. Attempt `n` waits a uniformly random delay between 0 and
`min(60 s, 1 s x 2^n)` ("full jitter"), which spreads a reconnect storm across the window; a `Welcome` resets the
count. Go's `session.Backoff` and the FakeDaemon implement the same schedule.

| The session ends with                                                              | The daemon                                                                                                              |
| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| [`Drain`](/docs/reference/protocol#drain)`{reconnect_after_ms, reason}`            | Closes and redials after the given delay; rolling deploys send a random delay up to 60 s                                |
| [`RevocationNotice`](/docs/reference/protocol#revocationnotice)`{reason, serials}` | Stops using the certificate and does not redial with it                                                                 |
| `RESOURCE_EXHAUSTED`                                                               | Retries after the `grpc-retry-pushback-ms` the gateway sent and restarts its doubling; without one, retries with jitter |
| `ABORTED`, `UNAVAILABLE`, a reset, a timeout                                       | Redials with back-off; the load balancer may pick another gateway                                                       |

While disconnected the daemon keeps enforcing its last desired state, keeps journaling events, and never drops a
device assignment on its own. Everything it owes the control plane (results, events) is in the journal and goes
out after the next `Welcome`.

## BUSY [#busy]

The gateway keeps at most `credits` commands in flight per node and lane, and the daemon accepts at most as many.
The two counters can disagree for a moment (a reconnect, a lost frame), so every heartbeat carries the daemon's free
credits and the gateway resynchronizes its counters from them.

When the daemon has no free credit on a command's lane, or its journal is full, it answers
`REJECTED` with reason `BUSY`. `BUSY` is not terminal and not journaled: the gateway puts the command back to
`pending` with `not_before` set by a back-off per command and session, 1 second doubling to 30 seconds, clears its
epoch, and pokes dispatch. Nothing is dropped. This is the daemon refusing a command; a gateway refusing a session
is the [retry pushback](#opening) above.

## Closing [#closing]

A session that ends normally (the stream closes, a heartbeat timeout, an expired certificate, a revocation, a
protocol error, gateway shutdown) gets an epoch-fenced disconnect write:

```sql
UPDATE node_sessions SET gateway_id = NULL, disconnected_at = compute_now()
 WHERE node_id = $node AND session_epoch = $epoch AND gateway_id = $gw
```

The node becomes `disconnected`, `node.disconnected` is published with the reason, and the node's controller is
queued. In-flight commands stay as they are for the next session's replay. A superseded or fenced session skips the
write, because its row belongs to someone else.

## The node\_sessions table [#the-node_sessions-table]

One row per node, inserted by JoinService with epoch 0 and owned by the gateways from then on.

| Column                                      | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node_id`                                   | Primary key, references `nodes`                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `gateway_id`                                | The gateway routing to the node; `NULL` while disconnected. Dispatch joins on it                                                                                                                                                                                                                                                                                                                                                                                                   |
| `session_id`                                | The live session's id                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `session_epoch`                             | Raised by every `Hello`; fences routing and liveness writes                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `boot_id`, `journal_id`, `role`             | As of the last `Hello`; compared by the resume rules                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `acked_event_seq`                           | Cumulative event ack for `journal_id`                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `last_seen`                                 | Flushed from gateway ages every 15 s                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `connected_at`, `disconnected_at`           | Set by `Hello` and by the disconnect write                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `daemon_version`, `proto_max`               | From `Hello`; read by rollouts (migration `core/0008_session_versions`)                                                                                                                                                                                                                                                                                                                                                                                                            |
| `duplicate_hellos`, `duplicate_since`       | Duplicate-`Hello` count in the current window, for the quarantine (migration `core/0016_duplicate_identity`)                                                                                                                                                                                                                                                                                                                                                                       |
| `trust_version`                             | The trust bundle version the daemon reported in `Hello` or a later `StatusReport`; read by the command-root switch (migration `core/0023_session_trust_version`)                                                                                                                                                                                                                                                                                                                   |
| `approver_set_epoch`, `approver_set_sha256` | The approver set the daemon reported enforcing (migration `core/0026_session_approver_set`: `numeric(20, 0)` with a check of at least 0, and `bytea`), written by `Hello` and by a `StatusReport` that acknowledges a trust bundle at least as new as the recorded one (fa39201); both NULL while the daemon holds no approver keys. Read by doctor's `pki.quorum-lag` and `pki trust-version`, which count a NULL digest as lagging while the published bundle has keys (08eb62e) |

Routing is always this join at dispatch time, never a column copied onto commands, so a reconnect needs no rewrite
and a command enqueued during a reconnect cannot be stranded. See [delivery](/docs/daemon/delivery).


# Site agent and ramdisk (/docs/daemon/site-and-ramdisk)

> How bcd site runs the boot server, proxy DHCP, TFTP, the image cache and BMC drivers for a site, and how bcd ramdisk joins from its kernel command line, wipes disks, writes and verifies images and kexecs.



`bcd` has two modes for bare metal besides `bcd node&#x60;. &#x2A;*`bcd site`** is the site agent:
it drives BMCs, makes `site.httpProxy` requests for agentless machines, and, with a `[boot]&#x60; section, runs the site's
boot server (proxy DHCP, TFTP, iPXE scripts, the image cache). &#x2A;*`bcd ramdisk`** runs inside the PXE
ramdisk that boot server boots: it joins from its kernel command line and runs the
commissioning and deploy kinds (wipe, image, kexec, diagnostics). Both are the same binary and the same agent core as
`bcd node` (session, journal, signed commands); only the kinds, the configuration and the extra servers differ.

This page is the daemon-side view: flags, `config.toml` keys, start-up checks, what each server answers, the exact
scripts and error codes. What the control plane does with all of it is in the bare-metal guide:
[sites and site agents](/docs/guides/bare-metal/sites) (the boot lease, discovery, the discovery attestor),
[BMC drivers and credentials](/docs/guides/bare-metal/bmc) (the operations metal sends, IPMI, sanboot) and
[deploying machines](/docs/guides/bare-metal/deploying#what-the-ramdisk-does) (what each ramdisk kind does to disks).

> **Tested against fakes only.** 
  No real site has run `bcd site`, and no real machine has booted `bcd ramdisk`. The Go tests drive both against fakes:
  an `httptest` Redfish BMC, a scripted gateway session, real UDP and HTTP sockets on loopback, and fakehw disks that
  are sparse files. The only real-system pieces are two Linux tests that build real ext4 images with `mkfs.ext4` and
  write into and read out of them with `debugfs`. No test runs `bcd site` or `bcd ramdisk` against a control plane. The
  metal test suite and the dev stack use a simulated site agent instead. See [what the tests cover](#what-the-tests-cover).

## The two modes [#the-two-modes]

|               | `bcd site`                                                                             | `bcd ramdisk`                                                                                                              |
| ------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Runs on       | A machine next to the fleet, on the provisioning and BMC networks, one or two per site | The machine being commissioned or deployed, booted over PXE by the site agent                                              |
| Configuration | `config.toml` (`--config`, default `/etc/bcd/config.toml`)                             | None: the `bc.*` kernel parameters (`--cmdline`, default `/proc/cmdline`)                                                  |
| Identity      | Joined once with a site token (`bcd join --role site`), kept in `data_dir`             | Joined on every boot with a single-use `discovery` token and the site agent's signed assertion, kept on tmpfs (`/run/bcd`) |
| Certificate   | Role `site`, 24 hours, renewed                                                         | Role `ramdisk`, at most 2 hours, commissioning kinds only                                                                  |
| Kinds         | `bmc.*`, `site.httpProxy`, `boot.prepareRamdisk`, `boot.cleanUp`, plus shared kinds    | `disk.wipe`, `deploy.image`, `boot.kexec`, `diag.dcgm`, plus shared kinds                                                  |
| Extra servers | Boot server (HTTP), proxy DHCP (UDP 67, 4011), TFTP                                    | None                                                                                                                       |

Both reach the gateway outbound only, like every mode (see [outbound only](/docs/daemon#outbound-only)). They were
added in a302223 and 21a4647 (ramdisk) and 48cb975, 0c35daf and fc74ed9 (site).

## bcd site [#bcd-site]

### Flags and start-up [#flags-and-start-up]

`bcd site` takes one flag:

| Flag       | Default                | Meaning                                                                       |
| ---------- | ---------------------- | ----------------------------------------------------------------------------- |
| `--config` | `/etc/bcd/config.toml` | The configuration file. `join-encryption.jwk` is read from the same directory |

It takes no positional arguments (exit 2). At start it checks, in this order:

| Check                                                                                         | On failure                                                                                   | Exit |
| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---- |
| The configuration loads and validates (unknown keys and sections are errors)                  | `bcd site: <error>`                                                                          | 1    |
| `data_dir` holds a joined identity                                                            | `bcd site: <error>`                                                                          | 1    |
| The identity's role is `site`                                                                 | `this data directory holds a <role> identity; join with a site token (bcd join --role site)` | 1    |
| With `[boot] listen`: `join-encryption.jwk` beside the config                                 | `the boot server needs join-encryption.jwk: …`                                               | 1    |
| With `proxy_dhcp = true` or a `tftp` address: `server_ip` is an IPv4 address and `url` is set | `proxy_dhcp and tftp need [boot] server_ip (IPv4) and url`                                   | 2    |

Without `[boot] listen` it logs `no [boot] listen configured: this site agent does not serve PXE machines` and runs
the agent alone. With it, the boot server runs beside the agent. A boot server that fails later (no `kernel` or
`initrd`, a port it cannot bind) is logged as `boot server stopped` and does **not** stop `bcd site`: BMC work and
`site.httpProxy` go on. Watch for that log line.

Ports 67, 69 and 4011 are privileged. The unit the install script writes is for `bcd node` and has no
`CAP_NET_BIND_SERVICE`, so a site agent needs its own unit (see [installing a site agent](/docs/daemon/install#a-site-agent)).

### config.toml keys [#configtoml-keys]

`bcd site` reads the same file as `bcd node` (the format and the shared keys are on
[installing bcd](/docs/daemon/install#configtoml)). These are the keys that matter to a site agent:

| Key                                                                  | Default                                  | Used for                                                                                                                                                                                                                                                          |
| -------------------------------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gateway_url`, `gateway_server_name`, `site`, the three `*_pin` keys | (required, except `gateway_server_name`) | The session, and copied into every ramdisk's `bc.gateway`, `bc.server_name`, `bc.site` and `bc.pins`                                                                                                                                                              |
| `data_dir`                                                           | `/var/lib/bcd`                           | Identity, journal, boot entries, BMC memory and the image cache (see [files](#files-in-data_dir))                                                                                                                                                                 |
| `[site] bmc_networks`                                                | empty                                    | Comma-separated CIDRs that `site.httpProxy` never reaches, whatever the control plane says. Invalid CIDRs fail validation. When set, the agent's Hello carries the capability `site.bmc-networks` (fc0ca6a), and `bcd doctor` warns about a site agent without it |
| `[boot] listen`                                                      | empty (no boot server)                   | The boot server's HTTP address, such as `0.0.0.0:8080`                                                                                                                                                                                                            |
| `[boot] url`                                                         | empty                                    | How machines reach the boot server, such as `http://10.20.0.2:8080`. iPXE chains to it, ramdisks get `<url>/cache` as `bc.cache`, and BMCs fetch firmware from it. Without it, firmware updates are refused                                                       |
| `[boot] kernel`, `[boot] initrd`                                     | empty                                    | URLs of the site's ramdisk kernel and initramfs, which iPXE fetches. Both are required for the boot server to start                                                                                                                                               |
| `[boot] cmdline`                                                     | empty                                    | Appended to every ramdisk's kernel command line (`console=ttyS0,115200`)                                                                                                                                                                                          |
| `[boot] server_ip`                                                   | empty                                    | This agent's IPv4 address on the provisioning network: the proxy DHCP server id, the TFTP next-server                                                                                                                                                             |
| `[boot] proxy_dhcp`                                                  | `false`                                  | Proxy DHCP on UDP `:67` and `:4011`, all interfaces                                                                                                                                                                                                               |
| `[boot] tftp`, `[boot] tftp_root`                                    | empty                                    | The TFTP server's UDP address (`:69`) and the directory of iPXE builds it serves. `tftp_root` is also served over HTTP at `/boot/files/` for UEFI HTTP boot                                                                                                       |
| `[boot] cache_max_bytes`                                             | `0` (unbounded)                          | The image cache's byte budget                                                                                                                                                                                                                                     |
| `[plugins] …`                                                        | none                                     | Hosted bcx drivers, such as `bcx-ipmi` (see [IPMI through bcx-ipmi](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi))                                                                                                                                           |
| `[update] …`, `[websocket] …`                                        | none                                     | Self-update (`daemon.upgrade` runs on site agents too) and the WebSocket fallback, as on a node                                                                                                                                                                   |

A complete site agent configuration is on [installing a site agent](/docs/daemon/install#a-site-agent).

### How it connects to the control plane [#how-it-connects-to-the-control-plane]

A site agent joins with a [site token](/docs/guides/bare-metal/sites#enroll-site-agents-with-site-tokens) and then
holds a [session](/docs/daemon/sessions) like any node: Hello and Welcome, heartbeats, the journal, signed commands.
Three things are specific to it:

* **Its Hello** lists the bcx drivers it hosts (their manifests), which become the `bc.dev/driver-kind.*` labels metal
  checks BMC operations against, and `site.bmc-networks` when `[site] bmc_networks` is set.
* **Its desired state** is the `site` section of its NodeSpec (`SiteSpec`): the boot lease state (`active`, `epoch`,
  `ttl_ms`), `cache_images` and `proxy_targets`. The site converger records the lease, checks the sanboot entries,
  evicts and warms the cache (below).
* **Its fence.** Every server frame updates the time it last heard from the gateway. The agent answers PXE machines
  only while the `site` section says it holds the lease at a non-zero epoch **and** it heard from the gateway within
  the lease TTL (15 seconds when the section gives none). A partitioned agent stops answering by itself. The lease
  itself is the control plane's; see [one active boot server per site](/docs/guides/bare-metal/sites#one-active-boot-server-per-site).

The site section's status:

| Condition                                                                             | Status                                                                                    |
| ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| A live boot entry sanboots a digest the section does not name as an `iso` cache image | Terminal failure: `sanboot image <sha256> of <mac> is not an iso cache image of the site` |
| A cache image without a 32-byte `sha256`, or whose download failed or did not match   | Terminal failure naming the URL and the reason                                            |
| Cache images still downloading                                                        | Retried: `<n> cache images warming` (the section stays `PROGRESSING`)                     |
| Otherwise                                                                             | Converged                                                                                 |

### Kinds it runs [#kinds-it-runs]

The kinds registered for role `site` (`siteKinds` and the shared table in `kinds.go`):

| Kind                                                                                  | Lane    | Destructive   | What bcd does                                                   |
| ------------------------------------------------------------------------------------- | ------- | ------------- | --------------------------------------------------------------- |
| `bmc.query`                                                                           | control | no            | A read-only BMC operation                                       |
| `bmc.boot`                                                                            | control | yes           | Boot device, virtual media, BIOS settings                       |
| `bmc.power`                                                                           | control | yes           | `power.setPower`                                                |
| `bmc.rotate`                                                                          | control | yes           | `management.rotateCredential`                                   |
| `bmc.firmware`                                                                        | control | yes           | `firmware.update` through the verified cache                    |
| `bmc.storage`                                                                         | control | yes           | `raid.applyLayout`, `raid.clear`                                |
| `site.httpProxy`                                                                      | control | yes (0d42de8) | One HTTP request to an agentless machine of the site            |
| `boot.prepareRamdisk`                                                                 | control | no            | Stores one boot entry (`ProxyDhcpEntry`) by MAC                 |
| `boot.cleanUp`                                                                        | control | no            | Drops the listed MACs, or every entry of a workflow             |
| `inventory.refresh`, `logs.collect`, `driver.invoke`, `node.reboot`, `daemon.upgrade` | shared  | as on a node  | As on a node (`daemon.upgrade` only with `[update]` configured) |

A node refuses the site kinds, and a site agent refuses the ramdisk's. With command-signing roots in its trust
bundle, the agent refuses any command without a valid worker signature, and destructive kinds fall under the batch
quorum (see [signed commands](/docs/daemon/delivery#signed-commands-and-attested-results)). The table of kinds and
timeouts metal uses is on [how BMC operations run](/docs/guides/bare-metal/bmc#how-bmc-operations-run).

`boot.prepareRamdisk` refuses (`invalid-payload`) a MAC that does not parse and a `sanboot_sha256` that is neither
empty nor 32 bytes. Storing an entry also drops the MAC from the discovered list.

### The boot server [#the-boot-server]

`ServeBoot` starts proxy DHCP and TFTP when configured, then one HTTP server with a 10-second header timeout and
8 KiB of headers (0442561). Its routes:

| Route                           | Answers                                                                                                        | Gated by                                                                                                   |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `GET /boot/chain.ipxe`          | `chain <url>/boot/ipxe?mac=${net0/mac}&uuid=${uuid}&ip=${net0/ip}&buildarch=${buildarch}&platform=${platform}` | Nothing: it holds no secret                                                                                |
| `GET /boot/ipxe?mac=&uuid=&ip=` | The machine's script (below)                                                                                   | The lease, then a live boot entry                                                                          |
| `GET /boot/files/<file>`        | The iPXE builds in `tftp_root`, for UEFI HTTP boot (only when `tftp_root` is set)                              | Nothing: it serves only `tftp_root`                                                                        |
| `GET /cache/t/<grant>/<sha256>` | A cached image                                                                                                 | A live grant; a boot grant also needs the lease (503 otherwise). Any other `/cache/` path is 404 (b7ed8a2) |

`/boot/ipxe` answers one of these scripts, all `text/plain`:

| Case                                                        | Status | Script                                                                                                                                               |
| ----------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Not the active boot server                                  | 503    | `echo This boot server does not hold the site boot lease.`, `sleep 10`, `reboot`                                                                     |
| No live boot entry for the MAC                              | 404    | `echo No boot is prepared for this machine.`, `exit` (the firmware falls through to its next boot device). A well-formed MAC is listed as discovered |
| Discovery token not available yet                           | 503    | `echo The boot credential is not available yet.`, `sleep 10`, `chain ${self}`                                                                        |
| Sanboot entry, image not cached or not an `iso` cache image | 503    | `echo The installer image is not on this boot server yet.`, `sleep 30`, `chain ${self}`                                                              |
| Sanboot entry, image ready                                  | 200    | `sanboot --no-describe <url>/cache/t/<grant>/<sha256>`                                                                                               |
| Ramdisk entry                                               | 200    | `kernel <kernel> initrd=initrd <bc.* parameters> <cmdline>`, `initrd --name initrd <initrd>`, `boot`                                                 |

For a ramdisk entry the agent fetches the discovery token with a `SecretRequest` bound to the entry's workflow, once
per token reference (a second fetch in the same boot reuses it), then signs a fresh `DiscoveryAssertion` with its node
key under purpose `discovery-assertion`. The assertion's `not_after` is 15 minutes out, or the entry's `not_after` if
that is sooner; its IP is the `ip` query parameter, else the connection's address; its product UUID the `uuid`
parameter, else the entry's. When `[boot] url` is set, `bc.cache` becomes `<url>/cache/t/<grant>`, a grant that lasts
until the entry lapses (6 hours for an entry without `not_after`). The fields and how the control plane checks them
are on [the discovery attestor](/docs/guides/bare-metal/sites#the-discovery-attestor).

### Proxy DHCP [#proxy-dhcp]

`daemon/internal/pxe` answers beside the site's own DHCP server and never hands out an address (`yiaddr` stays 0):

* It reads requests of type DISCOVER (answered with OFFER) and REQUEST (answered with ACK), and only from clients
  whose vendor class (option 60) starts with `PXEClient` or `HTTPClient`. Everything else is ignored.
* From the request it takes the MAC (`chaddr`), the client architecture (option 93), whether the client is iPXE
  (user class, option 77, contains `iPXE`), the SMBIOS UUID (option 97, 17 bytes with type 0) and the relay (`giaddr`).
* An answer carries the server id (option 54, `server_ip`), the vendor class (`HTTPClient` for HTTP boot clients,
  else `PXEClient` with PXE discovery control 8 in option 43, so the firmware boots the named file with no menu),
  option 66 and `siaddr` when there is a TFTP next-server, and the boot file in option 67 (and in the BOOTP file field
  when it is under 128 bytes).
* A client without an address and without a relay is answered on broadcast `255.255.255.255:68`; a relayed one at
  the address it came from.

Which file a client gets:

| Client                  | Option 93 | Answer                                   |
| ----------------------- | --------- | ---------------------------------------- |
| iPXE (any architecture) | any       | `<url>/boot/chain.ipxe`                  |
| BIOS PXE                | 0         | `undionly.kpxe` from TFTP at `server_ip` |
| x86-64 UEFI PXE         | 7, 9      | `ipxe.efi` from TFTP                     |
| arm64 UEFI PXE          | 11        | `ipxe-arm64.efi` from TFTP               |
| x86-64 UEFI HTTP boot   | 16        | `<url>/boot/files/ipxe.efi`              |
| arm64 UEFI HTTP boot    | 19        | `<url>/boot/files/ipxe-arm64.efi`        |
| Any other architecture  |           | No answer                                |

Before any of this, the agent must be the active boot server and the MAC must have a live boot entry. Otherwise the
request goes unanswered, and a MAC without an entry is listed as discovered.

The TFTP server is read-only (a write request gets error 4), octet mode only, with `blksize` (capped at 1,468 bytes so
a block fits a 1,500-byte MTU) and `tsize` negotiation. It retransmits each block up to 5 times, 2 seconds apart, from
one ephemeral socket per transfer. A name that resolves outside `tftp_root` is "file not found". It runs at most 64
transfers at once and 4 per source address, and drops a request beyond either bound unanswered (4d8b493).

You supply the iPXE builds and the ramdisk kernel and initramfs: the repository builds neither.

### Discovery [#discovery]

While it holds the lease, the agent lists every PXE request, and every `/boot/ipxe` fetch with a well-formed MAC, from
a MAC without a live entry in `StatusReport.discovered` (0056e1e). It keeps at most 256 machines, forgets each a day
after it was last seen, takes a UUID only in canonical form and an architecture only as `[a-z0-9_-]{1,32}` (0442561),
and asks for a status report for a new machine at most once every 30 seconds. With proxy DHCP on, an HTTP fetch counts
only for a MAC DHCP already showed. iPXE fetches report the architecture as `${buildarch}-${platform}` (`x86_64-efi`);
DHCP requests as `bios`, `x86_64-efi`, `arm64-efi`, `x86_64-efi-http`, `arm64-efi-http` or `arch-<n>`. What the
control plane does with the list is on [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines).

### The image cache [#the-image-cache]

The cache lives in `data_dir/cache` (60afcc7). Each image is a file named by its SHA-256 in hex with a `.json`
sidecar holding the URL it was pulled from and its last use:

* **Pulling.** Every `cache_images` entry not yet cached is downloaded in the background into a `.pull-*` temporary
  file, hashed on the way, and renamed into place only when the digest matches. Only `http://` and `https://` URLs
  are pulled.
* **Serving.** Only under a grant (b7ed8a2): a boot grant serves any cached image until the boot entry lapses and only
  while the agent holds the lease; a firmware grant serves one digest for 2 hours. Grants are 128-bit random tokens
  held in memory, so a restart invalidates them. Serving updates the last-use time at most once an hour.
* **Evicting** (2091cdc). With `cache_max_bytes` set, each converge pass deletes least recently used images until the
  cache fits, skipping every digest the `site` section still names.
* **Reporting** (4735600). The inventory's `images` section lists ready and pulling images (at most 1,000, the most
  recently used), the bytes used and the budget. The control plane turns it into the site's `image_locations`.

### BMC operations [#bmc-operations]

For each `bmc.*` command the agent:

1. Decodes the `BmcOperationRequest` and refuses (`invalid-payload`, the command fails) an operation outside its
   kind's allowlist (944221c). The allowlist is on [Redfish in bcd site](/docs/guides/bare-metal/bmc#redfish-in-bcd-site).
2. Picks the driver: built-in Redfish for `redfish` or no driver; otherwise a hosted bcx driver with that id that
   lists the operation's concern (51806ab), or `unsupported-driver`.
3. Refuses driver options on Redfish (`invalid-config`, 7753df0).
4. Fetches the credential with a `SecretRequest` bound to the command's workflow step, as `{ username, password }`
   JSON or a bare password paired with the step's `username` argument. A rotation fetches the new password the same
   way, by `newCredentialRef`.
5. For `firmware.update`, pulls the image into the verified cache and hands the BMC a firmware grant URL instead of the
   source (`image-unverified` without `[boot] url` or a valid digest).
6. Remembers the BMC host, and every address its name resolves to, in `data_dir/bmc-hosts.json` (da474d0).
7. Runs the operation. A hosted driver gets the address, pin and options as config and the credentials as secrets,
   never in its arguments.

A BMC that refuses or fails does not fail the command: it succeeds with `ok: false` and a code. Only a malformed
payload fails the command itself.

| `error_code`              | Raised when                                                                                                                                                                                                                                 |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `unreachable`             | The connection failed, timed out (60 s) or the certificate did not match the pinned SPKI                                                                                                                                                    |
| `auth-failed`             | The BMC answered 401 or 403                                                                                                                                                                                                                 |
| `unsupported`             | 404 or 405, an empty collection, no CD virtual media, no RAID controller, no such account, or an operation the client does not have                                                                                                         |
| `bmc-error`               | Any other status of 300 or more (with the first Redfish extended message), an answer that is not JSON, a resource path outside `/redfish/v1`, or a plugin failure without its own code                                                      |
| `invalid-args`            | A power action or boot device it does not know, a missing image URL, password, component or settings, a BIOS attribute the BIOS lacks or with the wrong type, a RAID level other than 0, 1, 5, 6 or 10, a member disk not on the controller |
| `invalid-address`         | Not `https://`, `redfish://` or a bare host, credentials, a query or a fragment in the address, a path other than `/redfish/v1/Systems/<id>`                                                                                                |
| `invalid-config`          | Driver options on a Redfish BMC                                                                                                                                                                                                             |
| `credentials-unavailable` | The credential or new password could not be fetched                                                                                                                                                                                         |
| `unsupported-driver`      | No hosted driver with that id serves the concern                                                                                                                                                                                            |
| `image-unverified`        | `firmware.update` without a boot server URL, without a hex SHA-256, or whose download did not match                                                                                                                                         |

A hosted driver's own codes (such as `bcx-ipmi`'s `pin-unsupported`) pass through as they are.

The Redfish client (`daemon/internal/redfish`) uses TLS 1.2 or later, never an environment proxy, never follows a
redirect, and reads at most 4 MiB of an answer. Power actions map to `ResetType` values:

| `action`           | `ResetType`        |
| ------------------ | ------------------ |
| `on`               | `On`               |
| `off`              | `ForceOff`         |
| `graceful-off`     | `GracefulShutdown` |
| `reset`            | `ForceRestart`     |
| `graceful-restart` | `GracefulRestart`  |
| `cycle`            | `PowerCycle`       |
| `nmi`              | `Nmi`              |

`power.getPower` reports `on` for `On` and `PoweringOff`, `off` for `Off` and `PoweringOn`, and `unknown` otherwise.
`management.rotateCredential` patches the account, then lists `/redfish/v1/Systems` with the new password and answers
`{ verified }`. The per-operation Redfish requests are on
[Redfish in bcd site](/docs/guides/bare-metal/bmc#redfish-in-bcd-site).

### site.httpProxy [#sitehttpproxy]

`site.httpProxy` makes one request on the control plane's behalf to an agentless machine of the site. Whatever the
command says, the agent enforces (0d42de8, a2a853d, 808379f, da474d0):

* an `http` or `https` URL without credentials; `GET`, `POST`, `PUT` or `DELETE` (an empty method is `GET`);
* no path that cleans to `/redfish…`, and no host that is a remembered BMC or inside `[site] bmc_networks`, by name or
  by any resolved address;
* every resolved address private, CGNAT `100.64.0.0/10` or unique-local, never loopback, link-local, multicast,
  unspecified or public; the connection goes to the first checked address, with no second lookup;
* the host is one of `target_node_id`'s addresses in `SiteSpec.proxy_targets` (an IP literal by value, a name by a
  listed name or by resolving only to listed addresses); with no listed target, nothing is allowed;
* a timeout of at most a minute (30 seconds when unset or out of range), a 10-second connect timeout, no redirects, the
  pinned SPKI when given, secret headers fetched with a `SecretRequest`;
* the response body cut at 1 MiB with `truncated` set, its headers sorted by name.

| Error code                | Raised when                                      |
| ------------------------- | ------------------------------------------------ |
| `forbidden-target`        | Any rule above but the lookup and the connection |
| `unreachable`             | The host does not resolve, or the request fails  |
| `credentials-unavailable` | A secret header could not be fetched             |
| `invalid-payload`         | The request cannot be built                      |

What metal checks before it sends one is on
[deploying machines](/docs/guides/bare-metal/deploying#what-sitehttpproxy-allows).

### Files in data\_dir [#files-in-data_dir]

| Path                                       | Holds                                                                                                        |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `identity.json`, the node key, the journal | As on a node                                                                                                 |
| `boot-entries.json`                        | The boot entries by MAC (protobuf JSON, mode 0600), so a restart keeps answering; lapsed entries are dropped |
| `bmc-hosts.json`                           | Every BMC host and address this agent has driven (mode 0600); `site.httpProxy` refuses them                  |
| `cache/<sha256>`, `cache/<sha256>.json`    | Verified images and their sidecars                                                                           |

## bcd ramdisk [#bcd-ramdisk]

### Flags and boot parameters [#flags-and-boot-parameters]

`bcd ramdisk` is not installed on a host; it is the agent in the ramdisk image the site agent boots. Its flags:

| Flag         | Default         | Meaning                                                                             |
| ------------ | --------------- | ----------------------------------------------------------------------------------- |
| `--cmdline`  | `/proc/cmdline` | Where to read the `bc.*` boot parameters                                            |
| `--data-dir` | `/run/bcd`      | Key, identity and journal, on tmpfs: every boot is a new identity and a new journal |

The boot parameters are the contract between the two modes (`daemon/internal/bootparams`). The site agent renders
them in this order; the ramdisk ignores every argument that does not start with `bc.`:

| Parameter        | Value                                                                  | Missing or malformed                             |
| ---------------- | ---------------------------------------------------------------------- | ------------------------------------------------ |
| `bc.gateway`     | The gateway URL, `https://` (`http://` only in fakehw builds)          | `bc.gateway must be the gateway URL`             |
| `bc.server_name` | Optional: the name the gateway certificate carries                     |                                                  |
| `bc.cache`       | Optional: the site cache under this boot's grant                       |                                                  |
| `bc.site`        | The site id                                                            | `bc.site is missing`                             |
| `bc.pins`        | Serving CA, join signing and join encryption pins, comma-separated     | `bc.pins needs three pins`, `bc.pins is missing` |
| `bc.jwk`         | The JoinService encryption key's public JWK, base64url without padding | `bc.jwk is missing`, or a decode error           |
| `bc.token`       | The machine's single-use, MAC-bound `discovery` join token             | `bc.token is missing`                            |
| `bc.assertion`   | The site agent's `SignedDiscoveryAssertion`, base64url                 | `bc.assertion is missing`, or a decode error     |

The token is readable in `/proc/cmdline` by anything on the ramdisk, which runs nothing but bcd. It is single-use,
bound to the MAC, and support bundles redact it.

### Start-up and join [#start-up-and-join]

<Mermaid
  chart="flowchart LR
  P[&#x22;parse bc.* parameters&#x22;] --> C[&#x22;build the configuration&#x22;] --> J{&#x22;identity in data dir?&#x22;}
  J -- no --> K[&#x22;check bc.jwk against its pin&#x22;] --> I[&#x22;inventory and boot id&#x22;] --> D[&#x22;Join: role ramdisk, attestor discovery&#x22;]
  J -- yes --> R
  D --> R[&#x22;run the agent&#x22;]"
/>

1. It reads and parses the command line. A missing parameter exits 2 (`TestRamdiskNeedsBootParameters`).
2. It builds the configuration from the parameters: gateway, server name, site, pins, `data_dir`, a socket at
   `<data-dir>/bcd.sock`, `bc.cache` as the image cache, JSON logs. An `http://` gateway is refused (exit 2) unless
   the binary is a fakehw build, which then speaks cleartext HTTP/2.
3. Without an identity in the data directory it joins: it decodes `bc.jwk` and checks it against the join encryption
   pin (exit 1 on a mismatch), collects the inventory (it joins without hardware claims if that fails, with a
   warning) and the boot id, and calls Join with role `ramdisk`, attestor `discovery`, the token and the assertion
   as the attestation. A failed join exits 1. With an identity already there (bcd restarted within the same boot),
   it skips the join.
4. It runs the agent until the machine kexecs into its installed OS or is powered off.

The control plane's checks on that join are on [the discovery attestor](/docs/guides/bare-metal/sites#the-discovery-attestor)
and [attestation](/docs/guides/nodes/attestation#discovery).

### The join configuration it installs [#the-join-configuration-it-installs]

Before running, the ramdisk prepares the files a deploy writes into the installed OS (797915d): the same
configuration with `data_dir` reset to `/var/lib/bcd`, rendered as a `config.toml`, and `bc.jwk` as
`join-encryption.jwk`. For a ramdisk booted with `bc.server_name`, the installed file reads:

```toml title="/etc/bcd/config.toml (written by deploy.image)"
# Written by bcd (PLAN.md §3.5, §6 Install).
gateway_url = "https://gw.eu.example.com"
gateway_server_name = "gw.internal"
site = "site_01k5w..."
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."
```

So the installed OS's `bcd join` needs nothing more from its user data than the provisioned token.

### Kinds it runs [#kinds-it-runs-1]

| Kind                                                              | Lane      | Flags            | What bcd does                                                                          |
| ----------------------------------------------------------------- | --------- | ---------------- | -------------------------------------------------------------------------------------- |
| `disk.wipe`                                                       | provision | destructive      | Wipes the listed disks (by WWN or serial) or every non-removable one, and verifies     |
| `deploy.image`                                                    | provision | destructive      | Streams a verified image onto the root disk, then user data and the join configuration |
| `boot.kexec`                                                      | control   | self-terminating | Boots the installed OS without POST                                                    |
| `diag.dcgm`                                                       | provision |                  | `dcgmi diag -r <level> -j`; from level 2, refuses GPUs held by a workload              |
| `inventory.refresh`, `diag.nvml`, `logs.collect`, `driver.invoke` | shared    |                  | As on a node                                                                           |

What each does to the disks, step by step, is on [what the ramdisk does](/docs/guides/bare-metal/deploying#what-the-ramdisk-does).
The daemon-side details:

* **Wipe.** `quick` (the default) zeroes the first and last MiB of each disk and reads them back. The other modes use
  `nvme format --ses=1 --force` (or `--ses=2` for crypto erase) for NVMe, `hdparm` security erase for SATA, else `blkdiscard --secure`.
  Every disk is attempted; if any is not verified, the command fails with reason `wipe-unverified` and the result still
  lists every disk's method and outcome. Metal sends it attested, so the result is node-signed.
* **Root disk.** The hint's first non-empty field wins (`wwn`, `serial`, `device`); otherwise the smallest
  non-removable disk of at least 4 GiB (or the hint's `min_size_bytes`, if larger).
* **Image stream.** The ramdisk asks the site cache first, `GET <bc.cache>/<sha256>`, and falls back to the image's
  source URL on any error or non-200. The SHA-256 covers the stored bytes (before gzip or zstd), and after
  decompression the rest of the stream is drained so the hash covers all of it. Data goes straight to the device in
  4 MiB buffers. A copy error, a failed sync or a digest mismatch zeroes the disk's ends again, so nothing bootable is
  left. Then the kernel rereads the partition table (`blockdev --rereadpt`, `udevadm settle`).
* **User data and join files.** User data is fetched with a `SecretRequest` bound to the step and written as a
  cloud-init NoCloud seed or Ignition's `config.ign`. ext filesystems are written in userspace with `debugfs` and FAT
  with `mtools` (1ae1789, d9a900f); only xfs and btrfs fall back to a `nodev,nosuid` mount, written through `os.Root`
  with exclusive creates.
* **kexec.** It journals the command as running with the boot id before it acts, then loads the kernel and initrd
  (the given paths, `/boot/vmlinuz` and `/boot/initrd.img`, or the newest `vmlinuz-*` and its initrd), copied out of
  an ext root with `debugfs` or from a read-only mount, with `root=UUID=<root> ro` plus the requested arguments, and
  runs `systemctl kexec`, falling back to `kexec -e`. It completes when the installed OS rejoins.

| Error code                 | Kind                                      | Raised when                                                                                       |
| -------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `inventory-failed`         | all four                                  | The disk or GPU inventory could not be collected                                                  |
| `disk-missing`             | `disk.wipe`, `deploy.image`, `boot.kexec` | A listed disk is absent, no disk to wipe, or no disk matches the root hint                        |
| `wipe-unverified` (reason) | `disk.wipe`                               | A disk was not wiped and verified                                                                 |
| `unsupported-format`       | `deploy.image`                            | A qcow2 image                                                                                     |
| `download-failed`          | `deploy.image`                            | The image could not be fetched, or its gzip or zstd header is bad                                 |
| `disk-write-failed`        | `deploy.image`                            | Opening, writing or syncing the device failed                                                     |
| `checksum-mismatch`        | `deploy.image`                            | The stored bytes do not match the pinned SHA-256                                                  |
| `disk-too-small`           | `deploy.image`                            | More bytes were written than the disk holds                                                       |
| `user-data-unavailable`    | `deploy.image`                            | The user data secret could not be fetched                                                         |
| `user-data-failed`         | `deploy.image`                            | Writing the user data or the join configuration failed                                            |
| `kexec-failed`             | `boot.kexec`                              | Loading or executing the kernel failed                                                            |
| `dcgm-failed`              | `diag.dcgm`                               | `dcgmi` produced no output or unparseable JSON                                                    |
| `invalid-payload`          | all                                       | A malformed request: a non-http(s) image URL, a digest that is not 32 bytes, a DCGM level above 4 |

The ramdisk image needs `bcd` and the tools these call; the list is on [installing the ramdisk](/docs/daemon/install#the-ramdisk).

## What the tests cover [#what-the-tests-cover]

| Test                                                                                                                                                | Checks                                                                                                                                                                                                                                                                                             |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/pxe` `TestProxyDHCPAnswersPreparedMachines`                                                                                               | Over real UDP on loopback: firmware gets `ipxe.efi`, iPXE gets the chain URL, the UUID and architecture are parsed, an unprepared MAC gets no answer                                                                                                                                               |
| `internal/pxe` `TestTFTPServesFilesUnderItsRoot`, `TestTFTPBoundsItsTransfers`                                                                      | A 3,000-byte file with `blksize` and `tsize` over real UDP; path traversal and write requests refused; the 64 and 4 transfer bounds                                                                                                                                                                |
| `internal/redfish` `TestRedfishOperations`, `TestRedfishAddressNamesTheSystem`, `TestRedfishRefusesForeignResourcePaths`, `TestBiosRaidAndFirmware` | Against an `httptest` TLS Redfish service: power, boot device, virtual media, inspection, rotation and the old password failing, the SPKI pin, plain HTTP refused, a multi-system service, BMC paths kept on the BMC, BIOS typing, RAID layouts and SimpleUpdate                                   |
| `internal/bootparams` `TestRenderParseRoundTrip`                                                                                                    | Render and parse round-trip; missing parameters and a wrong pin count refused                                                                                                                                                                                                                      |
| `internal/agent` `TestBootServerRendersARamdiskScript`                                                                                              | A real agent with a scripted session: 503 without the lease, 404 for an unknown MAC, the token fetched once for the workflow, the script parsed back by `bootparams`, the assertion verified against the agent's key at epoch 7, the PXE answers, and the fence after a minute without the gateway |
| `internal/agent` `TestSiteBMCAndBootEntries`, `TestBMCOperationsKeepToTheirKind`, `TestBMCThroughAHostedDriver`, `TestParseCredentialForms`         | `unreachable` and `unsupported-driver` as `ok: false`, entries persisted, lapsed and cleaned up, site kinds refused on a node, operations outside their kind, firmware only through the verified cache, a hosted driver getting credentials as secrets, `invalid-config`                           |
| `internal/agent` `TestProxyTargetAllowed`, `TestHTTPProxyRefusesTargetsOutsideTheSite`                                                              | Proxy targets by IP and name; loopback, metadata, public and non-HTTP targets, remembered BMCs, `/redfish` paths and `bmc_networks` refused                                                                                                                                                        |
| `internal/agent` `TestDiscoveryListIsCappedAndForgets`, `TestUnknownMachinesAreListedAndSanbootBootsTheCachedISO`                                   | The 256 cap and 24-hour expiry; an unknown machine listed in the status report; a sanboot served under its grant; an uncached sanboot answered 503 and failing the site                                                                                                                            |
| `internal/agent` `TestSiteCacheWarmsAndServes`, `TestDeployImageFromTheSiteCache`, `TestCacheEvictsLeastRecentlyUsedOutsideTheKeepSet`              | Warming and the `images` section, a ramdisk deploying from the cache, LRU eviction outside the keep set                                                                                                                                                                                            |
| `internal/agent` `TestDiskWipeQuickVerifies`, `TestDeployImageStreamsAndChecksTheDigest`, `TestPickRootDiskAndDCGM`, `TestPartitionsAndBootFiles`   | On fakehw sparse disk files: a node-signed quick wipe, `disk-missing`, a gzip image streamed with user data, `checksum-mismatch` leaving zeroed ends, root disk choice, DCGM parsing, `lsblk` parsing and kernel lookup                                                                            |
| `internal/agent` `TestExtWriteInUserspace`, `TestExtBootFilesInUserspace` (Linux)                                                                   | Real ext4 images built with `mkfs.ext4`: a planted symlink replaced, files root-owned 0600, bad paths refused, symlinks resolved inside the image, loops and escapes refused. Skipped without `mkfs.ext4` or `debugfs`                                                                             |
| `internal/agent` `TestWriteInRootStaysInTheImage`, `TestWriteInRootReplacesSpecialFiles`                                                            | The mount fallback never writes outside the mounted root or through a special file                                                                                                                                                                                                                 |
| `cmd/bcd` `TestRamdiskNeedsBootParameters`                                                                                                          | `bcd ramdisk` exits 2 naming `bc.gateway` on a command line without parameters                                                                                                                                                                                                                     |

Under a fake root, `SecureErase` zeroes the file, `RunDCGM` passes every GPU and `Kexec` reboots the fake host, so the
secure-erase tools, `dcgmi` and `kexec` have not run from bcd. No test has started `bcd site` as a process, joined a
site agent or a ramdisk to a real gateway, answered a real PXE ROM, or talked to a real BMC.

  - [Sites and site agents](/docs/guides/bare-metal/sites): Boot leases, the boot server, discovery and the discovery attestor.

  - [BMC drivers and credentials](/docs/guides/bare-metal/bmc): The operations metal sends, IPMI and sanboot.

  - [Deploying machines](/docs/guides/bare-metal/deploying): Walkthroughs, and what the ramdisk does to disks.

  - [Installing bcd](/docs/daemon/install#site-agents-and-the-ramdisk): Setting up a site agent and the ramdisk image.

  - [Delivery](/docs/daemon/delivery): Commands, lanes, signatures and the journal.


# Telemetry and status (/docs/daemon/telemetry)

> How bcd exports its own GPU, host and instance metrics over OTLP/HTTP to the collector NodeSpec.telemetry names, every metric with its unit, type and attributes, how failures behave, and what bcd status reads from the agent's local socket.



bcd reports on itself in two places. Every interval it pushes its own metrics, GPU readings, host pressure, memory
and huge page pools, and the VMs and sandboxes it runs, over OTLP/HTTP straight to a collector (27c7ee2). And on the
machine itself it answers `bcd status` on a local unix socket (a4c305b, ad8791f). Neither goes through the gateway or
the session, and neither reaches PostgreSQL: principle P8 keeps telemetry off the control plane's path.

This page is the daemon's side: what bcd sends, how, and what happens when the collector is not there. The control
plane's side, the `nodeTelemetryModule` that tells each node where to export and the doctor finding for a pool with a
bad `policy.telemetry`, is in [node metrics from bcd](/docs/operations/observability#node-metrics-from-bcd). The
control plane's own Prometheus series are a different thing and are listed in the
[metrics reference](/docs/reference/metrics).

> **Mostly tested against fakes.** 
  The exporter, the metric set and `bcd status` are tested against an in-process fake collector, a fake GPU sampler
  (`fakehw.GPUSnapshot`) and fake `/proc` and sysfs trees. Two runs go further. The fakehw bcd in `tests/e2e/bcd.test.ts`
  posts its metrics to a local OTLP receiver the test runs. Since 898a23c, `tests/e2e/bcd-host-real.test.ts`
  (`BCD_E2E_REAL=1`) runs the non-fakehw bcd on a real RTX 5090 under WSL and checks that the real `nvidia-smi` numbers
  reach the receiver. No collector product (an OpenTelemetry Collector, Grafana Alloy, a vendor endpoint) has received
  the export yet, and no data-center GPU (ECC, MIG, row remapping) has been sampled live. See
  [what the tests cover](#what-the-tests-cover).

## Where the export is configured [#where-the-export-is-configured]

There is no telemetry key in `config.toml`. The export is configured only by the `telemetry` section of the node's
NodeSpec, so the control plane can point a fleet at a new collector without touching any
host:

```proto title="proto/bettercompute/agent/v1/session.proto"
// Where bcd sends its own telemetry (OTLP). Never the session or Postgres (P8).
message TelemetrySection {
  // Empty disables export.
  string otlp_endpoint = 1;
  uint32 export_interval_ms = 2;
  // Extra "key=value" resource attributes.
  repeated string resource_attributes = 3;
  // Secret ref of the exporter's auth headers, fetched with SecretRequest.
  string headers_ref = 4;
}
```

The control plane fills the section from `TelemetrySpec` (`packages/core/src/domain.ts`). The NodeSpec controller
(`packages/server/src/controllers/nodespec.ts`) merges each module's `telemetry` part field by field and renders it
to the wire:

| `TelemetrySpec` field | Wire field            | How it is rendered                                                            |
| --------------------- | --------------------- | ----------------------------------------------------------------------------- |
| `otlpEndpoint`        | `otlp_endpoint`       | As given                                                                      |
| `intervalSeconds`     | `export_interval_ms`  | `round(intervalSeconds * 1000)`; omitted when unset                           |
| `headersRef`          | `headers_ref`         | As given (a `secret://`, `vault://` or `env://` reference)                    |
| `resourceAttributes`  | `resource_attributes` | An object rendered as `key=value` strings, sorted, so the spec hash is stable |

The only built-in module that sets it is `nodeTelemetryModule` from `@better-compute/observability` (53aa356). It is
not registered by default. Its options, the pool override and the checks it runs are described in
[node metrics from bcd](/docs/operations/observability#node-metrics-from-bcd). Any module can return a
`telemetry` part, and the e2e suites use a small one of their own.

The `telemetry` section converges like any other: see [convergence and status](/docs/daemon/delivery#convergence-and-status).
Every mode that runs the agent has the converger: `bcd node`, `bcd site` and `bcd ramdisk`.

## What bcd checks [#what-bcd-checks]

The control plane checks more than bcd does. `nodeTelemetryModule` allows `http://` only to `localhost`,
`127.0.0.1` or `[::1]`, limits the interval to 5 to 3600 seconds, and keeps `bcd.*` and `service.*` attribute keys
for bcd. bcd itself checks only what it needs to run safely, when it plans the section:

| Input                 | bcd's rule                                                                                                                                                     | On failure                                                              |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `otlp_endpoint`       | Empty, or an `http://` or `https://` URL with a host and no user info (`telemetry.MetricsURL`). `/v1/metrics` is appended unless the path already ends with it | The section fails terminally: `FAILED`, and the exporter is not changed |
| `resource_attributes` | Each entry is `key=value` with a non-blank key. Key and value are trimmed; the value may contain `=`                                                           | The section fails terminally                                            |
| `export_interval_ms`  | 0 means the default, 60 s. Anything below 5 s is raised to 5 s                                                                                                 | Never fails                                                             |
| `headers_ref`         | Not checked at plan time. It is fetched when the exporter starts                                                                                               | See [headers](#headers)                                                 |

So bcd accepts plain `http://` to any host. The e2e suites rely on this to reach a receiver on the WSL host's
address. A deployment that wants TLS everywhere gets it from the module's check, not from bcd.

A terminal failure is tested in `TestTelemetryExportsGPUMetrics`: an `ftp://x` endpoint leaves the `telemetry`
section `FAILED`, and the agent keeps running.

## How the exporter runs [#how-the-exporter-runs]

The converger keeps one exporter goroutine, under the agent's context:

* **Start and restart.** When the section differs from the one applied (`proto.Equal`), the plan has one action:
  `export telemetry to <endpoint>`, or `stop the OTLP export` when the endpoint is empty. Applying it cancels the
  running exporter, resets the last export and last error, and starts a new one if there is an endpoint. Any change,
  even to one resource attribute, restarts the exporter. An empty section on a node that never exported is a no-op.
* **Stop.** A section with an empty endpoint, or no section, stops the export. The exporter also stops with the
  agent.
* **First export.** The exporter sends once as soon as it starts, then every interval on a ticker.
* **Each tick.** It samples the metrics and posts them, under one deadline equal to the interval. So a slow
  `nvidia-smi` or a slow collector cannot pile up exports. The HTTP client has its own 10 s timeout.
* **One batch per tick.** The exporter keeps no queue. A batch that fails is not sent again. The next tick samples
  fresh values and sends those.

### The request [#the-request]

Each export is one `POST` to `<endpoint>/v1/metrics` with `Content-Type: application/json`. The body is an
`ExportMetricsServiceRequest` in the OTLP JSON encoding (the protobuf JSON mapping). bcd builds it itself and links
no OpenTelemetry SDK. The request has one resource, one scope (`bcd` and bcd's version), and the metrics that have
at least one point:

```json title="An export, abridged (from TestExportEncodesOTLPJSON)"
{
  "resourceMetrics": [{
    "resource": { "attributes": [
      { "key": "bcd.node.id", "value": { "stringValue": "n1" } },
      { "key": "service.name", "value": { "stringValue": "bcd" } }
    ] },
    "scopeMetrics": [{
      "scope": { "name": "bcd", "version": "1.0.0" },
      "metrics": [
        { "name": "bcd.gpu.utilization", "unit": "1",
          "gauge": { "dataPoints": [{ "timeUnixNano": "200000000005", "asDouble": 0.87,
            "attributes": [{ "key": "gpu.uuid", "value": { "stringValue": "GPU-1" } }] }] } },
        { "name": "bcd.gpu.ecc.errors", "unit": "{error}",
          "sum": { "aggregationTemporality": 2, "isMonotonic": true, "dataPoints": [{
            "startTimeUnixNano": "100000000000", "timeUnixNano": "200000000005", "asInt": "5",
            "attributes": [
              { "key": "gpu.uuid", "value": { "stringValue": "GPU-1" } },
              { "key": "type", "value": { "stringValue": "correctable" } }] }] } }
      ]
    }]
  }]
}
```

* Attributes are string values, sorted by key.
* Integer points use `asInt` (a string, as the JSON mapping requires); others use `asDouble`.
* Every point of an export has the same `timeUnixNano`, the time of the export.
* Sums are cumulative and monotonic (`aggregationTemporality: 2`). Their `startTimeUnixNano` is when the exporter
  started, not when the process started, so a restart of the exporter (any section change) starts a new series start.
* No metric carries a `description`.

### Resource attributes [#resource-attributes]

| Attribute                           | Value                                                                        |
| ----------------------------------- | ---------------------------------------------------------------------------- |
| `service.name`                      | `bcd`                                                                        |
| `service.version`                   | bcd's version (`-X main.version`, `0.0.0-dev` in a dev build)                |
| `host.name`                         | The host's hostname                                                          |
| `bcd.node.id`                       | The node id                                                                  |
| `bcd.site.id`                       | The site id from the node's state, read when the exporter starts             |
| The section's `resource_attributes` | Added after these. A key that collides with one of the five above is ignored |

### Headers [#headers]

`headers_ref` names a secret that holds the collector's auth headers. The exporter fetches it with a
`SecretRequest` bound to workflow `host:telemetry` and step `headersRef`. The secret broker releases it only while the
node's current NodeSpec names that reference in `telemetry.headersRef` (binding rule 3 in
`packages/server/src/join/secret-broker.ts`; see [secrets](/docs/daemon/protocol#secrets) and
[sealed secrets](/docs/operations/security#sealed-secrets)).

The secret is parsed by `telemetry.ParseHeaders` in one of two forms:

```text title="A JSON object"
{"Authorization":"Bearer a"}
```

```text title="Name: value or name=value, separated by newlines or commas"
Authorization: Bearer a
X-Scope-OrgID: t1
```

A header name that is empty or contains a space, tab, CR, LF or `:`, or a value that contains CR or LF, is refused.
The headers are set on every request after `Content-Type`, so a `Content-Type` in the secret replaces bcd's.

While the headers cannot be fetched or parsed, the exporter sends nothing. It does not export without them. It
records the error (`headers <ref>: …`) and tries the fetch again on every tick until one succeeds. After that it keeps
the headers for the life of that exporter. A rotated secret reaches bcd only when the exporter restarts: a section
change, or an agent restart.

## Failure behaviour [#failure-behaviour]

| What happens                                                      | What bcd does                                                                                                                                                           |
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The endpoint is not an http(s) URL, or an attribute is malformed  | The `telemetry` section is `FAILED` with reason `ConvergeFailed` (terminal), and bcd logs `section failed`; the previous exporter keeps running                         |
| The collector cannot be reached, times out, or answers 5xx or 429 | The batch is lost. The error is recorded and the next tick sends fresh values                                                                                           |
| The collector answers 4xx other than 429                          | The exporter marks it permanent (`telemetry.PermanentError`). The batch is lost; the exporter still tries again next tick, since the agent loop treats both kinds alike |
| The headers secret cannot be fetched or parsed                    | Nothing is sent; the fetch is tried again every tick                                                                                                                    |
| `nvidia-smi` fails or is missing                                  | The `bcd.gpu.*` gauges from samples are left out of that export. The inventory-based GPU series (ECC, row remapping, XIDs) and the host series are still sent           |
| The kernel has no PSI                                             | The `bcd.host.pressure.*` and `bcd.host.cpu.steal` series are left out                                                                                                  |

Logging is per change, not per tick. The first failure logs a warning (`telemetry export`, with `endpoint` and
`error`). The same error again logs nothing. A different error logs again. The first success after a failure logs
`telemetry export resumed`. The latest outcome is always in [`bcd status`](#bcd-status): the last successful export
and the last error.

A failing export never fails the section, holds back `observed_generation`, or touches the session. The control plane
learns nothing about it. Watch the collector's side (a node's series going stale) or `bcd status` on the machine.

## The metrics [#the-metrics]

All metrics are sampled at the moment of the export, except host pressure (below). A metric with no points in an
export is left out of it. Gauges are OTLP gauges; sums are cumulative monotonic OTLP sums.

### GPUs [#gpus]

GPU points carry `gpu.uuid`, `gpu.index` and `gpu.model`, from the inventory. A sampled GPU the inventory does not
list carries only `gpu.uuid`.

| Metric                       | Unit          | Type          | Attributes                                   | Source                                                                                                            |
| ---------------------------- | ------------- | ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `bcd.gpu.memory.used`        | `By`          | gauge, int    | GPU                                          | `nvidia-smi` `memory.used`                                                                                        |
| `bcd.gpu.memory.total`       | `By`          | gauge, int    | GPU                                          | `memory.total`                                                                                                    |
| `bcd.gpu.memory.reserved`    | `By`          | gauge, int    | GPU                                          | `memory.reserved`                                                                                                 |
| `bcd.gpu.utilization`        | `1`           | gauge, double | GPU                                          | `utilization.gpu` / 100 (0 to 1)                                                                                  |
| `bcd.gpu.memory.utilization` | `1`           | gauge, double | GPU                                          | `utilization.memory` / 100                                                                                        |
| `bcd.gpu.temperature`        | `Cel`         | gauge, double | GPU                                          | `temperature.gpu`                                                                                                 |
| `bcd.gpu.power.draw`         | `W`           | gauge, double | GPU                                          | `power.draw`                                                                                                      |
| `bcd.gpu.power.limit`        | `W`           | gauge, double | GPU                                          | `power.limit`                                                                                                     |
| `bcd.gpu.clock.sm`           | `MHz`         | gauge, double | GPU                                          | `clocks.sm`                                                                                                       |
| `bcd.gpu.clock.memory`       | `MHz`         | gauge, double | GPU                                          | `clocks.mem`                                                                                                      |
| `bcd.gpu.processes`          | `{process}`   | gauge, int    | GPU                                          | Compute processes on the GPU (`--query-compute-apps`); graphics processes are not counted (9a6fee9)               |
| `bcd.gpu.mig.memory.used`    | `By`          | gauge, int    | GPU, `mig.id`                                | Each MIG instance's memory, from the full `-q -x` read                                                            |
| `bcd.gpu.mig.memory.total`   | `By`          | gauge, int    | GPU, `mig.id`                                | Each MIG instance's memory size                                                                                   |
| `bcd.gpu.xid.active`         | `{condition}` | gauge, int    | GPU                                          | bcd's active device fault conditions for the GPU (XIDs seen this boot); 0 when none. Sent for every inventory GPU |
| `bcd.gpu.ecc.errors`         | `{error}`     | sum, int      | GPU, `type` (`correctable`, `uncorrectable`) | The inventory's aggregate ECC counters; only for GPUs that report ECC                                             |
| `bcd.gpu.rows.remapped`      | `{row}`       | gauge, int    | GPU, `type` (`correctable`, `uncorrectable`) | The inventory's row remapping counts; only for GPUs that report them                                              |

The sampled series (the first eleven, and MIG) come from one `nvidia-smi` call per export since 9a6fee9:

```bash
nvidia-smi --query-gpu=uuid,memory.total,memory.used,memory.reserved,memory.free,utilization.gpu,utilization.memory,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,pstate,mig.mode.current --format=csv,noheader,nounits
nvidia-smi --query-compute-apps=gpu_uuid --format=csv,noheader
```

bcd falls back to the full read (`nvidia-smi -q -x` and the rest of `nvml.SMI.Snapshot`, several seconds of driver
time on a large board) when a GPU is in MIG mode, since only `-q -x` has the MIG instances' memory; when the driver
refuses a field (`memory.reserved` before R510); or when a line does not have the expected columns. If `nvidia-smi`
is missing (`ErrNoDriver`), there is no fallback and no sample.

The ECC, row remapping and XID series come from bcd's inventory and fault state, not from a new read, so they only
change when the inventory or the fault set does.

A fakehw build samples `fakehw.GPUSnapshot` for its seed instead of `nvidia-smi` (`gpusampler_fakehw.go`).

### Host [#host]

| Metric                      | Unit     | Type          | Attributes          | Source                                                               |
| --------------------------- | -------- | ------------- | ------------------- | -------------------------------------------------------------------- |
| `bcd.host.pressure.cpu`     | `%`      | gauge, double | none                | PSI `/proc/pressure/cpu` `some avg10`                                |
| `bcd.host.pressure.memory`  | `%`      | gauge, double | none                | PSI `/proc/pressure/memory` `some avg10`                             |
| `bcd.host.pressure.io`      | `%`      | gauge, double | none                | PSI `/proc/pressure/io` `some avg10`                                 |
| `bcd.host.cpu.steal`        | `%`      | gauge, double | none                | Share of CPU time stolen between the last two samples (`/proc/stat`) |
| `bcd.host.memory.total`     | `By`     | gauge, int    | none                | `/proc/meminfo` `MemTotal` (3111284)                                 |
| `bcd.host.memory.available` | `By`     | gauge, int    | none                | `/proc/meminfo` `MemAvailable`                                       |
| `bcd.host.hugepages.total`  | `{page}` | gauge, int    | `page.size` (bytes) | `/sys/kernel/mm/hugepages/hugepages-<n>kB/nr_hugepages`              |
| `bcd.host.hugepages.free`   | `{page}` | gauge, int    | `page.size` (bytes) | `.../free_hugepages`                                                 |

Host pressure is not read at export time. The agent's health loop samples it every 30 s (`HealthInterval`) and also
sends it to the control plane in `StatusReport.pressure`. The export uses the latest sample. So the four pressure
series are missing for the first 30 s after bcd starts, and may be up to 30 s old. The first steal reading is 0, as
it needs a baseline.

### Instances [#instances]

These come only from a host with an instance runtime (VPS and sandbox hosts). The configured series cover every
runtime, plugin runtimes included. The live usage series come only from runtimes that read it (`runtime.UsageReader`).

| Metric                                  | Unit         | Type        | Attributes                                                                                          | Source                                                                                                                    |
| --------------------------------------- | ------------ | ----------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `bcd.instances`                         | `{instance}` | gauge, int  | `runtime`, `state`                                                                                  | Domains each runtime lists, managed or not, counted by state                                                              |
| `bcd.instance.vcpu`                     | `{cpu}`      | gauge, int  | `instance.id`, `runtime`, `managed`                                                                 | Configured vCPUs                                                                                                          |
| `bcd.instance.memory`                   | `By`         | gauge, int  | `instance.id`, `runtime`, `managed`, `hugepages` (backing page size in bytes, `0` for normal pages) | Configured memory                                                                                                         |
| `bcd.instance.memory.balloon.reclaimed` | `By`         | gauge, int  | `instance.id`, `runtime`                                                                            | Configured memory less what the balloon leaves the guest; running, managed guests with `placement.balloon` only (9bdd43a) |
| `bcd.instance.cpu.time`                 | `s`          | sum, double | `instance.id`, `runtime`                                                                            | CPU time since the domain started (0b21b4b)                                                                               |
| `bcd.instance.memory.used`              | `By`         | gauge, int  | `instance.id`, `runtime`                                                                            | Memory the domain uses now                                                                                                |
| `bcd.instance.io.read`                  | `By`         | sum, int    | `instance.id`, `runtime`                                                                            | Disk bytes read                                                                                                           |
| `bcd.instance.io.write`                 | `By`         | sum, int    | `instance.id`, `runtime`                                                                            | Disk bytes written                                                                                                        |
| `bcd.instance.network.receive`          | `By`         | sum, int    | `instance.id`, `runtime`                                                                            | Bytes the guest received                                                                                                  |
| `bcd.instance.network.transmit`         | `By`         | sum, int    | `instance.id`, `runtime`                                                                            | Bytes the guest sent                                                                                                      |

Where each runtime reads its usage (cgroup v2 and tap counters for `cloud-hypervisor`, `virsh domstats` for
`libvirt-kvm`) and how that works under `bcd helper` is described in
[node metrics from bcd](/docs/operations/observability#node-metrics-from-bcd). The balloon itself is described in
[memory ballooning](/docs/daemon/delivery#memory-ballooning). A runtime whose `List` fails is left out of that export.

> **Cardinality.** 
  `instance.id` is an attribute on every instance series, so a host's series count grows with the instances it has
  held. bcd sends only what it lists now: a deleted instance leaves the next export. Plan your collector's or backend's
  staleness and limits for that. No GPU or host series carries a tenant id.

## The status socket [#the-status-socket]

`bcd node`, `bcd site` and `bcd ramdisk` serve their status on a unix socket (a4c305b):

* **Path.** `socket_path` in `config.toml`, default `/run/bcd/bcd.sock`. It must be absolute. `bcd ramdisk` uses
  `<data dir>/bcd.sock`.
* **Permissions.** The directory is created with mode 0700 and the socket is set to 0600, so only the account that
  runs the agent (root, or the unprivileged node user under `bcd helper`) and root can connect.
* **Protocol.** HTTP over the socket. One route: `GET /status`, which answers the agent's `Status` as JSON.
  The server's read-header timeout is 5 s.
* **Lifetime.** A stale socket from an earlier process is removed at start; the socket is removed again on shutdown.
  If the socket cannot be created, bcd logs `status socket unavailable` and runs without it. It is never fatal.

The socket serves only `bcd status` today. `bcd doctor` reads the host itself and does not use it. It is the only
local endpoint bcd has: see [outbound only](/docs/daemon#outbound-only).

## bcd status [#bcd-status]

```text
bcd status [--socket /run/bcd/bcd.sock] [--json]
```

| Flag       | Default             | Meaning                                                                                                          |
| ---------- | ------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `--socket` | `/run/bcd/bcd.sock` | The agent's socket. `bcd status` does not read `config.toml`, so pass this when `socket_path` is not the default |
| `--json`   | `false`             | Print the status as indented JSON instead of text                                                                |

It exits 0 when it read a status, and 1 when it could not. When there is no socket, or the connection is refused, it
prints `bcd status: the node agent is not running (no socket at <path>)`; the client times out after 5 s. In the
Kubernetes DaemonSet, `bcd status` is the `bcd` container's liveness probe (see [Kubernetes](/docs/daemon/kubernetes)).

### Text output [#text-output]

```text
node       node_01k… (site site_1, role node)
bcd        0.4.0, boot 3f6c…
session    connected (epoch 12, clock offset -3 ms)
spec       applied 41, observed 41
  host       CONVERGED
  devices    CONVERGED
  telemetry  FAILED ConvergeFailed: converge: terminal failure: otlp endpoint "ftp://x" is not an http(s) URL
commands   0 running, 0 waiting for a spec
events     2 unacked, 0 waiting, 0 metrics dropped
identity   certificate until 2026-10-06T08:00:00Z, node key in the TPM, trust bundle v3, signed intent enforced
gpus       2
  0 GPU-5f1c…   87%   61 C   540/700 W  71.2/79.6 GiB  held by alloc_01k…
  1 GPU-9a02…    0%   34 C    72/700 W  0.0/79.6 GiB  UNHEALTHY
telemetry  https://otel.example.internal:4318, last export 2026-09-26T08:12:00Z
```

The values above are illustrative; the lines and their order are what `runStatus` prints. The section lines show each
NodeSpec section's last state with the `SECTION_STATE_` prefix removed, then its reason and message when there are
any. The `gpus`, `kubernetes` and `telemetry` lines are described in [bcd status](/docs/daemon#bcd-status). The
telemetry line is printed only while the applied section has an endpoint:

| Telemetry line                                     | Meaning                                                 |
| -------------------------------------------------- | ------------------------------------------------------- |
| `<endpoint>, no export yet`                        | The exporter started and has not succeeded yet          |
| `<endpoint>, last export <time>`                   | The last export succeeded at that time (UTC)            |
| `<endpoint>, last export <time>, failing: <error>` | An export succeeded once, and the latest attempt failed |
| `<endpoint>, no export yet, failing: <error>`      | No export has succeeded since the exporter started      |

`metrics dropped` on the `events` line is not about OTLP. It counts metric-class events the journal dropped when it
was full (see [the journal](/docs/daemon/delivery#the-journal)). An OTLP export that fails is never journaled.

### JSON output [#json-output]

`--json` prints the same `agent.Status` the socket serves:

| Field                                                              | Meaning                                                                                                                                                                                                 |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nodeId`, `siteId`, `role`, `version`                              | Identity and bcd's version                                                                                                                                                                              |
| `bootId`, `incarnation`, `journalId`                               | This boot, this process, and the journal's id                                                                                                                                                           |
| `connected`, `epoch`, `clockOffsetMs`                              | Whether a session is open and welcomed, its epoch, and server time minus local time from the last Welcome                                                                                               |
| `appliedGeneration`, `observedGeneration`                          | The spec generations (see [generations](/docs/daemon/delivery#generations))                                                                                                                             |
| `sections`                                                         | Each section's `section`, `state`, `reason`, `message`; omitted before the first status report                                                                                                          |
| `running`, `deferred`                                              | Command ids running, and waiting for a spec; always arrays                                                                                                                                              |
| `pendingEvents`, `waitingEvents`, `droppedMetrics`                 | Unacked journal events, events waiting for journal room, metric events dropped                                                                                                                          |
| `certificateNotAfter`, `nodeKey`, `trustVersion`, `commandSigning` | Certificate expiry, `tpm` or `file`, the trust bundle version, and whether signed intent is enforced                                                                                                    |
| `gpus`                                                             | Per GPU: `uuid`, `index`, `model`, `healthy`, `held`, `utilizationPct`, `temperatureC`, `powerDrawW`, `powerLimitW`, `memoryUsedBytes`, `memoryTotalBytes`, `processes`; omitted on a host without GPUs |
| `kubernetes`                                                       | `mode` (`joined` or `hosted`), `server`, `nodeName`, `advertised`; omitted when the node is not a Kubernetes worker                                                                                     |
| `telemetry`                                                        | `endpoint`, `lastExport` (omitted before the first success), `lastError` (omitted when the last attempt succeeded); omitted when there is no endpoint                                                   |

The GPU fields are read when you ask. Each `bcd status` runs the GPU sampler (the same `nvidia-smi` query as the
export) with a 10 s timeout. If it fails or times out, the GPUs are still listed from the inventory, with zero
readings and the inventory's memory total. Power is rounded to whole watts. Health is bcd's own verdict (no failed or
pending row remap, no uncorrectable ECC, no XID this boot), the same one that keeps a GPU from new holds.

## Relation to control-plane observability [#relation-to-control-plane-observability]

The two sides do not overlap:

|               | bcd's export (this page)                     | The control plane's metrics                                  |
| ------------- | -------------------------------------------- | ------------------------------------------------------------ |
| What          | One node's GPUs, host and instances          | Sessions, commands, queues, placements, IAM, certificates    |
| Protocol      | OTLP/HTTP JSON, pushed                       | Prometheus text, scraped from `GET /metrics`                 |
| Names         | `bcd.*`, OTLP style                          | `better_compute_*`                                           |
| Configured by | NodeSpec `telemetry` (`nodeTelemetryModule`) | `createMetrics` and `host.metrics` in the host configuration |
| Reference     | The tables above                             | [Metrics reference](/docs/reference/metrics)                 |

What the control plane does learn about a node's health travels in the session instead: `StatusReport` carries host
pressure, section states and conditions such as XIDs, and the gateway turns them into node conditions and
`node.status` events (see [convergence and status](/docs/daemon/delivery#convergence-and-status)). That path drives
scheduling and health policy. The OTLP export is only for your dashboards and alerts. For alerting on the control
plane, see [what to alert on](/docs/operations/observability#what-to-alert-on).

## What the tests cover [#what-the-tests-cover]

| Test                                                                 | What it proves                                                                                                                                      | Against                                              |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `TestExportEncodesOTLPJSON` (`internal/telemetry`)                   | The OTLP JSON shape: path, content type, headers, sorted resource attributes, gauge and sum points, empty metrics left out                          | `httptest` collector                                 |
| `TestExportErrors`                                                   | 400 is permanent, 429 is not; `ftp://` is refused; `/v1/metrics` is appended                                                                        | `httptest` collector                                 |
| `TestParseHeaders`                                                   | Both header forms, and the refused names and values                                                                                                 | Unit                                                 |
| `TestTelemetryExportsGPUMetrics` (`internal/agent`)                  | A NodeSpec starts the export with the node's resource attributes and one point per GPU; an empty section stops it; a bad endpoint fails the section | Fake session, fake GPU sampler, `httptest` collector |
| `TestHostAndInstanceMetrics`                                         | Host memory, huge page pools and instance series                                                                                                    | Fake `/proc`, sysfs and runtimes                     |
| `TestStatusShowsGPUsHoldsAndTelemetry`                               | `Status()` lists GPUs with live readings and holds, and an export with no collector shows its error                                                 | Fake GPU sampler, closed port                        |
| `TestStatusWithoutAnAgent` (`cmd/bcd`)                               | `bcd status` without a socket exits 1 with `not running`                                                                                            | No agent                                             |
| `TestSamplesRealRTX5090WSL`, `TestSamplesFallBack` (`internal/nvml`) | The light sample query parses real captured output; MIG mode and an old driver fall back to the full read                                           | Captured output from a real RTX 5090 under WSL       |
| `TestSnapshotLiveHost` (`BCD_REAL_GPU=1`)                            | The light samples agree with the full read on the live GPUs                                                                                         | Real `nvidia-smi`                                    |
| `tests/e2e/bcd.test.ts`                                              | The fakehw bcd, joined to a dev stack, exports every GPU to a receiver named by NodeSpec; the stored spec has `exportIntervalMs: 5000`              | Real bcd process, fake GPUs                          |
| `tests/e2e/bcd-host-real.test.ts` (`BCD_E2E_REAL=1`, 898a23c)        | The non-fakehw bcd exports the real GPU's live metrics; the total memory matches `nvidia-smi` and the power limit is non-zero                       | Real RTX 5090 under WSL                              |

Nothing has tested ECC, row remapping, MIG memory or XIDs on a real data-center GPU, the export through a TLS
collector with auth headers on a real network, or a collector product. The test layout is in
[testing](/docs/operations/testing).


# Troubleshooting bcd (/docs/daemon/troubleshooting)

> Symptom, cause and fix for the ways bcd fails on real hosts - joins, pins and TLS, the clock, renewals, revocation, sections stuck progressing, GPU holds, nvidia-smi, KVM and the helper socket - with the doctor check that detects each.



This page maps what you see on a host (a log line, a refusal, a section that never converges) to its cause and
fix, drawn from the error strings in `daemon/`. Each entry names the [`bcd doctor`](/docs/daemon/commands#doctor-checks)
check that detects it, where one does. What happens to the fleet when a gateway, the database or the network fails
is on [failure modes](/docs/daemon/failure-modes); this page is about one host.

> **Where these come from.** 
  Most entries come from unit tests against fakes. The GPU, unprivileged-node and section-logging entries come from
  running bcd on a real RTX 5090 under WSL 2 (`tests/e2e/bcd-host-real.test.ts`; f84f1f9, da60bbd). The KVM,
  libvirt, cloud-hypervisor and Kubernetes entries have not been seen on a real host yet.

## Where to look first [#where-to-look-first]

```bash
sudo bcd doctor --config /etc/bcd/config.toml   # one line per check; exit 1 when one fails
sudo bcd status                                 # the running agent: session, sections, certificate
journalctl -u bcd.service -e                    # bcd's JSON log lines
journalctl -u bcd-helper.service -e             # on a node installed with --unprivileged
```

`bcd status` lists every NodeSpec section with its state, reason and message. Since da60bbd the node's own log
says the same: a section that cannot converge logs `section retrying` (WARN) with the error on every pass, about
every 30 seconds, and one that failed for good logs `section failed` (ERROR) with its reason
(`AllocationFailed`, `InstanceFailed`, `UpdateFailed` or `ConvergeFailed`). Before that commit an operator on the
host saw only `PROGRESSING`.

```json
{"level":"WARN","msg":"section retrying","section":"devices","generation":42,"error":"the slice of alloc_01k…: start bcd-alloc-alloc_01k….slice: …"}
```

Session problems log at INFO as `session ended; redialing` with the reason and the back-off.

## bcd will not start [#bcd-will-not-start]

| Symptom                                                                                              | Cause                                                                                             | Fix                                                                                                                          | Doctor                             |
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| `bcd node: config /etc/bcd/config.toml: line N: ...`                                                 | A key, section or value the [strict parser](/docs/daemon/configuration#format) refuses            | Fix the line                                                                                                                 | `config` (fail)                    |
| `dev_insecure_h2c is only accepted by fakehw development builds`                                     | A development setting in a host build                                                             | Remove `dev_insecure_h2c` and use an `https://` gateway                                                                      | `config` passes; the agent refuses |
| `agent: this node has not joined (run bcd join)`                                                     | No `identity.json` in `data_dir`                                                                  | [Join](/docs/daemon/commands#bcd-join)                                                                                       | `identity` (warn)                  |
| `agent: the node key does not match the node certificate; join again: ...`                           | `node.key` and `identity.json` belong to different keys; with a TPM key, "(was the TPM cleared?)" | `bcd join --reset --yes`, then join; see [TPM cleared or unavailable](/docs/daemon/failure-modes#tpm-cleared-or-unavailable) | `identity` (fail)                  |
| `session: the journal belongs to node X, not Y`                                                      | A `journal.db` left by another identity, e.g. after copying `identity.json` by hand               | `bcd join --reset --yes` moves the journal aside with the identity (b4fcc2e)                                                 | none                               |
| `[kubernetes] gpu_api must be "device-plugin" or "dra"`                                              | Any other value                                                                                   | Fix it                                                                                                                       | none                               |
| `[kubernetes] allowed_clusters: "..." is not https://host[:port]#sha256:<64 hex>`                    | A malformed entry                                                                                 | Fix it; no path, a 64-digit hex CA digest                                                                                    | `kubernetes` (fail)                |
| `bcd site: this data directory holds a node identity; join with a site token (bcd join --role site)` | `bcd site` on a node's data directory                                                             | Join a site agent separately                                                                                                 | none                               |
| `bcd status: the node agent is not running (no socket at /run/bcd/bcd.sock)`                         | The agent is not running, or `socket_path` is elsewhere                                           | Start `bcd.service`; pass `--socket`                                                                                         | none                               |

## Joining [#joining]

Server-side refusals are deliberately opaque: every refused join is `ACCESS_DENIED` "Join refused", and the reason
is recorded as a `security.join.refused` event. Read the reasons in [refusals](/docs/guides/nodes/joining#refusals).
A refused join is not retried and does not consume the token.

| Symptom (`bcd join: ...`)                                                                                           | Cause                                                                                                                              | Fix                                                                                                                                                    |
| ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `no join token: set BCD_JOIN_TOKEN or pass the token on standard input` (exit 2)                                    | Nothing in the variable or on standard input                                                                                       | `printf '%s\n' "$TOKEN" \| sudo bcd join`; `sudo` drops the environment unless `-E`                                                                    |
| `that is not a join token (bcjt_...)` (exit 2)                                                                      | The input is not `bcjt_<id>.<secret>`                                                                                              | Pass the token itself                                                                                                                                  |
| `... is readable by others (mode 0644): make it 0600 (a Kubernetes Secret: defaultMode 0400)`                       | `--token-file` others can read                                                                                                     | `chmod 0600`, or `defaultMode: 0400`                                                                                                                   |
| `the JoinService encryption key: open /etc/bcd/join-encryption.jwk: no such file or directory`                      | The key file is missing and no `--server` was given                                                                                | Rerun the install script, or add `--server https://...`; see [the join key and its fallback](/docs/guides/nodes/joining#the-join-key-and-its-fallback) |
| `the JoinService encryption key does not match join_encryption_key_pin`                                             | The file or well-known key is another deployment's, or rotated                                                                     | Reinstall with the current bundle's pins and key                                                                                                       |
| `pki: join response rejected: bad signature`                                                                        | The response was not signed by the key `join_signing_key_pin` names: a wrong pin, or a gateway answering for another control plane | Check the pin against the bundle (`bootstrapPins`)                                                                                                     |
| `pki: join response rejected: response is for another CSR`                                                          | The answer was not for this join's CSR                                                                                             | Retry; persistent, suspect the path to the gateway                                                                                                     |
| `agent: join: ... Join refused`                                                                                     | JoinService refused it                                                                                                             | Read `security.join.refused` for the reason                                                                                                            |
| log `join failed, retrying`                                                                                         | The gateway is unreachable, busy (`RESOURCE_EXHAUSTED`) or timing out; bcd retries up to 20 times                                  | Check `gateway_url` and the network; doctor `gateway`                                                                                                  |
| log `join is pending approval`                                                                                      | The pool requires approval                                                                                                         | Approve the join request; bcd asks again by itself                                                                                                     |
| `this machine already joined as X, not Y`                                                                           | `--node-id` names another node than the identity on disk                                                                           | Use the right node id, or `--reset --yes`                                                                                                              |
| `joining without config.toml needs --serving-ca-pin, --join-signing-pin and --join-encryption-pin from the install` | `--server` bootstrap on a host build without the three pins                                                                        | Pass all three                                                                                                                                         |
| `the control plane's well-known document does not match --serving-ca-pin: refusing to join`                         | The published pin differs from the install's                                                                                       | Treat it as an attack or a stale pin; do not drop the flag                                                                                             |
| `the control plane does not publish its gateway (well-known gateway.url); install /etc/bcd/config.toml instead`     | The server's options set no `gateway.url`                                                                                          | Install a `config.toml`, or configure `gateway` on the server                                                                                          |
| `the TPM: open /dev/tpmrm0: ...`                                                                                    | `--attestor tpm` without a usable TPM                                                                                              | Run as root on a host with the kernel resource manager                                                                                                 |
| `already joined as X` but the node cannot connect                                                                   | The identity loads but its certificate expired, was revoked or is refused at renewal                                               | `bcd join --reset --yes` as root, then join; see [joining again with --reset](/docs/daemon/install#joining-again-with---reset)                         |

Doctor: `gateway` fails when the gateway is not reachable over TCP within 5 seconds; `identity` warns before the
first join.

## Pins and TLS [#pins-and-tls]

bcd trusts no system root for the gateway: the chain must contain a CA whose SPKI equals `serving_ca_spki_pin`,
and the leaf must name the host. These appear as the `reason` of `session ended; redialing`, and as the failure of
`bcd join`.

| Symptom                                                                                                      | Cause                                                                                                             | Fix                                                                                                                                                                                      |
| ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `transport: the gateway chain does not include the pinned serving CA`                                        | The gateway's serving certificate is from another CA, a TLS-inspecting proxy answered, or the pin is wrong        | Issue the gateway certificate from the bundle's serving CA (`pki issue-gateway`); behind an inspecting proxy, use the [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04) |
| `transport: gateway certificate: x509: certificate is valid for A, not B`                                    | `gateway_url` names an IP address or another name than the certificate                                            | Set `gateway_server_name` to the certificate's name                                                                                                                                      |
| `transport: gateway certificate: x509: certificate has expired or is not yet valid`                          | The gateway certificate expired, or this host's clock is wrong                                                    | Renew the gateway certificate; check doctor `clock`                                                                                                                                      |
| `transport: the WebSocket endpoint's certificate is neither the pinned gateway's nor the configured proxy's` | An inspecting proxy whose CA is not `[websocket] proxy_ca_file`                                                   | Point `proxy_ca_file` at the proxy's CA                                                                                                                                                  |
| `transport: gateway proof rejected: ...`                                                                     | Over the fallback, the gateway's in-band proof does not chain to the pinned serving CA, or names another audience | Check `[websocket] audience` and the gateway's certificate                                                                                                                               |
| `the WebSocket fallback needs command-signing roots (every command and spec signed)`                         | `[websocket] url` on a node whose trust bundle has no command-signing roots                                       | Configure `pki.commandSigning` on the server, or drop `[websocket]`                                                                                                                      |

Doctor: none checks TLS; `gateway` tests TCP only.

## The clock [#the-clock]

Certificates, signed intents, exec tickets and renewal proofs all compare times. Since e2a0cf0 doctor asks the
kernel (`adjtimex`) whether the clock is synchronized.

| Symptom                                                                                                        | Cause                                                                                                    | Fix                                                              | Doctor         |
| -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------- |
| `the kernel reports the clock unsynchronized: run an NTP client (systemd-timesyncd or chrony); ...`            | No NTP client keeps the clock                                                                            | Enable `systemd-timesyncd` or `chrony`                           | `clock` (fail) |
| log `protocol: server time went backwards; refusing the session`                                               | A `Welcome` more than 5 minutes before bcd's server time mark: a gateway with a wrong clock, or a replay | Fix the gateway's clock; bcd drops each such session and redials | none           |
| `renewal failed` ending `Renew refused`, with `security.renew.refused` reason `client time outside the window` | `client_time` more than 60 seconds from server time                                                      | Synchronize the clock                                            | `clock`        |

bcd evaluates `not_after` through the offset the last `Welcome` gave it, so its own clock matters for TLS and
proofs, not for command deadlines. See [clock skew](/docs/daemon/failure-modes#clock-skew).

## Renewals, expiry and revocation [#renewals-expiry-and-revocation]

bcd renews at 60% of the certificate's life. A failed renewal logs `renewal failed` (WARN) with the error, the
attempt and `expiresIn`, and retries with back-off from 30 seconds to 10 minutes (or when the gateway's pushback
says).

| Symptom                                                                                         | Cause                                                                                                 | Fix                                                                                                                     | Doctor                             |
| ----------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| `renewal failed` with an error ending `Renew refused`                                           | JoinService refused it; the reason is a `security.renew.refused` event                                | Read the event ([refused renewals](/docs/operations/security#refused-renewals))                                         | none                               |
| `renewal: the certificate's node key is in the TPM, which is unavailable: ...`                  | The TPM, or under `[helper]` the helper that holds it, cannot be reached                              | Restore `/dev/tpmrm0` or start `bcd-helper.service`; the next retry renews                                              | `node key`                         |
| `renewal: this TPM's attestation key is not the one the certificate names ...`                  | Another TPM, or its endorsement seed changed                                                          | Join again: `bcd join --reset --yes --attestor tpm`                                                                     | none                               |
| `the certificate expired at ...; join again as root: bcd join --reset --yes, then bcd join`     | Renewal kept failing until expiry, or the machine was off                                             | Join again                                                                                                              | `identity` (fail)                  |
| log `certificate revoked`, then `bcd node: agent: this node's certificate was revoked` (exit 1) | The control plane revoked the certificate (`nodes.revokeIdentity`, `nodes.delete`, a cloned identity) | systemd restarts bcd, but the gateway refuses the serial at the handshake. Join again with a new token, `--reset --yes` | `identity` stays `ok` until expiry |

See [certificate expiry](/docs/daemon/failure-modes#certificate-expiry), [certificate
revocation](/docs/daemon/failure-modes#certificate-revocation) and
[renewal](/docs/guides/nodes/joining#renewal).

## Sections stuck progressing [#sections-stuck-progressing]

A section is `PROGRESSING` with reason `Retrying` while its converger returns an error that is not terminal, and
the spec is not observed meanwhile. The log line `section retrying` carries the error. Common ones:

| Error in `section retrying`                                                                | Cause                                                                                                                                                                                                                                     | Fix                                                                                   | Doctor               |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -------------------- |
| `helper: dial unix /run/bcd-helper/helper.sock: connect: no such file or directory`        | `bcd helper` is not running                                                                                                                                                                                                               | `systemctl start bcd-helper.service`; see why it stopped below                        | `helper`             |
| `helper: not allowed`                                                                      | The helper refused the connection                                                                                                                                                                                                         | See [helper socket refusals](#helper-socket-refusals)                                 | `helper`             |
| `helper: protocol version N, not M` or `protocol version N; this helper speaks M`          | The node and the helper are different builds                                                                                                                                                                                              | Restart both units after replacing the binary                                         | none                 |
| `the slice of ...: start bcd-alloc-....slice: ...` or `set the limits of ...`              | `systemctl` refused the hold's slice: not root, or systemd is not init                                                                                                                                                                    | Run `bcd node` as root or under `[helper]`; doctor `systemd`                          | `systemd`, `cgroups` |
| `the slice of ...: enable hugetlb under /sys/fs/cgroup...` or `limit 2MB hugepages of ...` | A hold with hugepage limits, and bcd cannot write cgroupfs: no `hugetlb` controller, or `/sys/fs/cgroup` is read-only for the process (the install unit sets `ProtectControlGroups=yes` for `bcd.service`; `bcd-helper.service` does not) | Enable `hugetlb`; check which process writes the slice (the helper, under `[helper]`) | `cgroups` (warn)     |
| `... /etc/cdi/...: read-only file system`                                                  | `/etc/cdi` did not exist when `bcd.service` started: the unit's `ReadWritePaths=-/etc/cdi` skips a missing path, and `ProtectSystem=strict` leaves `/etc` read-only                                                                       | `mkdir -m 0755 /etc/cdi` (the install script does it) and restart `bcd.service`       | none                 |

A `host` section that fails a unit keeps going with the others; see [status and
failures](/docs/daemon/host-config#status-and-failures).

## GPU holds [#gpu-holds]

The `devices` converger fails an assignment it can never bind at once, so the control plane re-places it instead
of waiting. The reason travels in `AllocationFailed`.

| Reason                | Message                                                                                                                                       | Cause                                                                                                                                                                                                                     | Fix                                              | Doctor                 |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ---------------------- |
| `node-unprivileged`   | `bcd node runs without root and without [helper]: it cannot write /etc/cdi or create hold slices (run bcd node as root, or under bcd helper)` | A host build of `bcd node` running as an ordinary user with no `[helper]` (da60bbd). Such a node also claims no `cdi`, `devices.limits` or `binding.kubernetes` capability in `Hello`, so capacity should not place there | Run it as root, or install with `--unprivileged` | `gpu-binding` (warn)   |
| `device-missing`      | The devices it names                                                                                                                          | A GPU, MIG instance or slot the assignment names is not on the host                                                                                                                                                       | Check `bcd status` GPUs and the inventory        | `gpu-bus`, `gpus`      |
| `overlap`             | The conflicting hold                                                                                                                          | A device is already held by another allocation, or passed to a VM                                                                                                                                                         | Release the other hold                           | none                   |
| `resources-exhausted` | What does not fit                                                                                                                             | The hold's limits do not fit what the host has left                                                                                                                                                                       | Re-placed by the control plane                   | `hugepages`, `cgroups` |
| `unsupported-binding` | `binding ... is not supported`                                                                                                                | A binding this bcd does not implement                                                                                                                                                                                     | Upgrade bcd                                      | none                   |

Before da60bbd an unprivileged node retried the hold every 30 seconds on systemd's "Access denied" for the slice,
logging nothing above debug, and the reservation stayed `binding` forever. A log warning `NVIDIA's CDI spec does
not name held devices` means the vendor spec predates the GPUs; the [container-toolkit](/docs/daemon/host-config#container-toolkit)
unit regenerates it.

## nvidia-smi errors [#nvidia-smi-errors]

bcd reads GPUs through `nvidia-smi`. Doctor's `gpus` check fails with the tool's error and, since f84f1f9, a hint
for the failures seen on real hosts:

| Error contains                                | Hint bcd adds                                                                     | Fix                                                                                             |
| --------------------------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `Driver/library version mismatch`             | The kernel module and user-space libraries differ, usually after a driver upgrade | Reboot, or unload and reload the `nvidia` modules                                               |
| `couldn't communicate with the NVIDIA driver` | The `nvidia` module is not loaded                                                 | `modprobe nvidia`; under Secure Boot the module must be signed by an enrolled key (see `dmesg`) |
| `No devices were found`                       | The driver sees no GPU                                                            | Compare `lspci -d 10de:` with `dmesg` (`RmInitAdapter failed`, Xid 79)                          |
| a timeout or `signal: killed`                 | `nvidia-smi` did not answer within a minute: a GPU may be wedged (Xid 79 or 119)  | `gpu.reset` or a reboot                                                                         |

Related checks around the GPUs:

| Doctor check            | Symptom                                                                  | Fix                                                                          |
| ----------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| `nvml` (warn)           | `nvml: nvidia-smi not found (no NVIDIA driver on this host)`             | Install the driver, or ignore on a host without GPUs                         |
| `nouveau` (fail)        | `nouveau` is loaded and keeps the NVIDIA driver off the GPUs             | Blacklist it, rebuild the initramfs, reboot                                  |
| `gpu-bus` (fail)        | More GPUs bound to `nvidia` than `nvidia-smi` sees: one fell off the bus | `gpu.reset` or a reboot; check `dmesg`                                       |
| `gpu-bus` (warn)        | GPUs on the bus with no driver bound, or no driver answering             | Install the driver ([nvidia-driver](/docs/daemon/host-config#nvidia-driver)) |
| `fabric-manager` (fail) | NVSwitches but `nvidia-fabricmanager` inactive: CUDA fails on every GPU  | Start it ([fabric-manager](/docs/daemon/host-config#fabric-manager))         |
| `persistence` (warn)    | Persistence mode off                                                     | Enable `nvidia-persistenced`                                                 |
| `gpu` (fail)            | `row remapping failed (RMA)` or uncorrectable ECC errors                 | Take the GPU out of service                                                  |
| `gpu-host` (warn)       | A WSL 2 paravirtual GPU (`/dev/dxg`)                                     | Expected: no VFIO, `gpu.reset`, MIG or DCGM there                            |

## KVM and instances [#kvm-and-instances]

| Symptom                                                                                                                          | Cause                                                                                                                                                      | Fix                                                                                                                                                       | Doctor                                |
| -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| `the CPU supports virtualization but /dev/kvm is missing: load kvm_intel or kvm_amd (modprobe), and check it is not blacklisted` | The module is not loaded                                                                                                                                   | `modprobe kvm_intel` or `kvm_amd`                                                                                                                         | `kvm` (warn)                          |
| `this is a VM without nested virtualization: ...`                                                                                | bcd runs in a VM                                                                                                                                           | Enable nested virtualization on its host                                                                                                                  | `kvm` (warn)                          |
| `no VT-x/AMD-V flags: enable virtualization in the firmware setup`                                                               | Disabled in firmware                                                                                                                                       | Enable Intel VT-x or AMD SVM                                                                                                                              | `kvm` (warn)                          |
| A non-empty `instances` section fails with `this host runs no instance runtime`                                                  | Neither `virsh` nor `cloud-hypervisor` on `PATH`, and no runtime plugin                                                                                    | Install a runtime                                                                                                                                         | `libvirt` (warn)                      |
| Instances without disks, user data or network policy                                                                             | `qemu-img`, `genisoimage`/`xorriso` or `nft` missing                                                                                                       | Install them                                                                                                                                              | `qemu-img`, `seed`, `nftables` (fail) |
| cloud-hypervisor instances do not boot                                                                                           | No `/usr/share/cloud-hypervisor/CLOUDHV.fd`                                                                                                                | Install the firmware                                                                                                                                      | `cloud-hypervisor` (warn)             |
| A cloud-hypervisor VMM cannot open `/dev/kvm`                                                                                    | Each VMM runs as its own uid with `DevicePolicy=closed` and `DeviceAllow=/dev/kvm rw`; bcd adds the `kvm` group to the unit only when a `kvm` group exists | Make `/dev/kvm` group `kvm` (the usual udev rule)                                                                                                         | none                                  |
| GPUs cannot be passed to instances                                                                                               | No IOMMU, `vfio-pci` not loaded, or a GPU shares its IOMMU group with a NIC or disk                                                                        | Enable the IOMMU; `modprobe vfio-pci`; move the GPU or enable ACS                                                                                         | `iommu`, `vfio`, `iommu-groups`       |
| Under `[helper]`, `libvirt` fails                                                                                                | The `bcd` account is in the `libvirt` group, `dynamic_ownership = 0` is missing, or QEMU's user is not in the `bcd` group                                  | Rerun the install script with `--unprivileged`; see [upgrading an unprivileged libvirt host](/docs/daemon/install#upgrading-an-unprivileged-libvirt-host) | `libvirt` (fail)                      |
| Under `[helper]`, no instance gets a NIC                                                                                         | `[helper] instance_bridges` is empty                                                                                                                       | Set the bridges                                                                                                                                           | `cloud-hypervisor` (warn)             |

## Helper socket refusals [#helper-socket-refusals]

On a node installed with `--unprivileged`, every privileged operation goes to `bcd helper`. The node only sees the
answer; the helper's log says why.

| Node sees                                          | Helper logs                                                  | Cause                                                                                                  | Fix                                                                      |
| -------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| `helper: not allowed`                              | `helper: connection refused` reason `uid N gid M is not bcd` | The caller is neither root nor the `[helper] user` with its primary group                              | Run `bcd node` as that user (the `unprivileged.conf` drop-in)            |
| `helper: not allowed`                              | reason `the peer does not run an installed bcd binary`       | `bcd node` runs from a path other than `[update] entrypoint`, a slot binary or the helper's own binary | Run the installed binary                                                 |
| `helper: malformed request`                        | `helper: malformed request`                                  | A request with unknown fields: a different build                                                       | Restart both units on the same binary                                    |
| `helper: dial unix ...: no such file or directory` | (nothing)                                                    | The helper is not running                                                                              | `journalctl -u bcd-helper.service`; it may have refused to start (below) |

`bcd helper` refuses to start, and logs to standard error, when:

| Message                                                                                                   | Cause                                                                                                                                             | Fix                                                              | Doctor          |
| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | --------------- |
| `/usr/local/lib must be a directory root owns that no one else can write (the helper socket lives there)` | A directory on the way to the slots, the socket or the entrypoint is group-writable, such as `root:staff` 2775 on older Debian (c139b9c, 9e71b1f) | `chmod g-w` that directory                                       | `helper` (fail) |
| `/etc/bcd/config.toml must be a regular file root owns that no one else can write`                        | The config was given to `bcd`, made group-writable or replaced by a symlink (3fa9021)                                                             | `chown root /etc/bcd/config.toml`, `chmod 0640` with group `bcd` | none            |
| `helper.user: user: unknown user bcd`                                                                     | The account does not exist                                                                                                                        | Rerun the install script, which creates it                       | none            |
| `runs as root`                                                                                            | Started without root                                                                                                                              | Run it from `bcd-helper.service`                                 | none            |

Host units through the helper fail when the helper cannot bind them to a node:

| Symptom                                                                                       | Cause                                                                         | Fix                                         | Doctor          |
| --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------- | --------------- |
| `helper` warns `host units bind to the node the first signed spec names (trust on first use)` | The node joined before `bcd join` kept root's signed join record              | Re-join as root to pin it                   | `helper` (warn) |
| `root's join record names X, but this machine's identity is Y`                                | The identity changed without a root join                                      | Re-run the join as root                     | `helper` (fail) |
| `bcd helper applies host units unsigned unless the node itself presents a trust bundle ...`   | Neither `command_root_sha256` nor the join record names command-signing roots | Pin them with `--command-root-sha256`       | `helper` (fail) |
| `bcd is in the tss group: it can send the TPM any command ...`                                | An install from between 0ba2a79 and 8870b8d                                   | `gpasswd -d bcd tss`, restart `bcd.service` | `tpm` (fail)    |

See [privilege separation](/docs/daemon#privilege-separation) and [signed specs](/docs/daemon/host-config#signed-specs).

## Kubernetes and self-update [#kubernetes-and-self-update]

| Symptom                                                                                         | Cause                                                       | Fix                                                        | Doctor                               |
| ----------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------ |
| `joining a Kubernetes cluster needs [kubernetes] allowed_clusters in bcd's config ...`          | A `k8s-join` unit with no allow-list (9f860ad)              | Add the cluster as `https://host:port#sha256:<CA digest>`  | `kubernetes` (fail on a joined node) |
| `joined to ... but the kubelet is not running`                                                  | The kubelet stopped                                         | Start it                                                   | `kubelet` (fail)                     |
| GPUs withdrawn with `ForeignGpuExposure`                                                        | Another path hands out GPUs                                 | See [GPU isolation](/docs/daemon/kubernetes#gpu-isolation) | `gpu-isolation` (fail)               |
| `no [update] release_pins: self-update is refused (reinstall to upgrade)`                       | No release keys pinned                                      | Regenerate the script with `--release-pins`                | `update` (warn)                      |
| `bcd X failed (...); bcd update --clear-failed lets a rollout retry it`                         | A version did not confirm and was swapped back              | Fix the cause, then `bcd update --clear-failed`            | `update` (warn)                      |
| `/usr/local/bin/bcd does not link to .../current: bcd.service would not start the current slot` | The install script replaced the entrypoint link with a file | The next self-update adopts it; or relink                  | `slots` (fail)                       |

See [self-update](/docs/daemon/install#self-update) and [bcd on Kubernetes](/docs/daemon/kubernetes).


# Attestors (/docs/extending/attestors)

> Write an AttestorVerifier that admits machines on your own evidence, defer a join for a second round, use evidence once, and register it with JoinService.



An attestor decides one question at join time: may this machine join, and as which node?
The built-in ones cover join tokens, expired certificates, installed images, PXE-booted ramdisks, cloud instance
identities and TPMs. Write your own when your machines can prove who they are some other way: a vendor-signed
delivery manifest, an asset database, a hardware security module.

An attestor has two halves:

| Half     | Contract                                 | Runs in                                  | Does                                                             |
| -------- | ---------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------- |
| verifier | `AttestorVerifier` from `defineAttestor` | JoinService, inside the join transaction | checks evidence, answers where the machine joins, or defers      |
| producer | `NodeAttestor`                           | the joining agent                        | produces the evidence, and answers a challenge in a second round |

The verifier is what you register. The producer for real machines is Go in `bcd`; the TypeScript producers in
`@better-compute/pki` exist for the FakeDaemon and tests.

> **The agent half.** 
  `bcd` joins with the built-in attestors only: `join-token` (the default), `provisioned` (`--node-id`), `tpm`
  (`--attestor tpm`, the only challenge it answers) and, from `bcd ramdisk`, `discovery`. It has no plug-in point for
  third-party producers. A custom attestor can be
  registered and exercised end to end today with TypeScript agents (`runAgentJoin` in `@better-compute/testing`), but a
  real machine needs a `bcd` that sends its `JoinRequest.attestor` and `attestation`.

## How a join reaches your attestor [#how-a-join-reaches-your-attestor]

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant A as Agent
  participant G as Gateway
  participant J as JoinService (api)
  participant V as Your verifier
  participant D as Postgres
  A->>G: Join(csr, claims, role, attestor, attestation, sealed_token?)
  G->>J: relay (it cannot read the token or forge the answer)
  J->>J: pick the attestor by id, check the role is in its roles
  J->>J: open the sealed token against the CSR, if any
  J->>D: BEGIN
  J->>V: verify(evidence, context)
  alt deferral
    V-->>J: { deferred: 'approval' | 'challenge', ... }
    J->>D: COMMIT (the attestor's own rows only)
    J-->>A: signed JoinResponse with the deferral
  else result
    V-->>J: AttestationResult
    J->>D: node, certificate, bindings, events, audit; COMMIT
    J-->>A: signed JoinResponse with the certificate chain
  end"
/>

`JoinRequest.attestor` names the attestor; empty means `join-token`. An unknown id is `UNSUPPORTED`, and a role the
attestor does not list is refused before `verify` runs. The gateway only relays: it cannot read the sealed token, sign
the response, or get a certificate issued without the node key's proof of possession.

## The contract [#the-contract]

```ts title="acme-asset.ts"
import { createHash, createPublicKey, verify } from 'node:crypto';
import { ComputeError, defineAttestor, type AttestationResult } from 'better-compute/core';

const vendorKey = createPublicKey(process.env.ACME_VENDOR_KEY_PEM!);
const refuse = (reason: string): never => {
  throw new ComputeError('ACCESS_DENIED', 'Join refused', undefined, {
    reason: 'acme-asset',
    refusal: `acme-asset:${reason}`,
  });
};

export const acmeAssetAttestor = defineAttestor({
  id: 'acme-asset',
  version: '1.0.0',
  roles: ['node'],
  async verify(evidence, context): Promise<AttestationResult> {
    if (!context.csrSha256) return refuse('no-csr-binding');
    let body: { assetTag?: unknown; sig?: unknown };
    try {
      body = JSON.parse(new TextDecoder().decode(evidence.payload));
    } catch {
      return refuse('malformed');
    }
    if (typeof body.assetTag !== 'string' || typeof body.sig !== 'string') return refuse('malformed');
    // The vendor signed the asset tag together with this join's CSR, so a captured proof is useless.
    const statement = Buffer.concat([Buffer.from(body.assetTag), context.csrSha256]);
    if (!verify('sha256', statement, vendorKey, Buffer.from(body.sig, 'base64url'))) return refuse('bad-signature');

    const { rows } = await context.tx.query<{ tenant_id: string; pool_id: string }>(
      'SELECT tenant_id, pool_id FROM plg_acme_assets WHERE asset_tag = $1',
      [body.assetTag],
    );
    const asset = rows[0] ?? refuse('unknown-asset');
    return {
      tenantId: asset.tenant_id,
      role: 'node',
      poolId: asset.pool_id,
      bind: createHash('sha256').update(body.assetTag).digest(), // one node per asset, for good
    };
  },
});
```

`defineAttestor` returns the verifier frozen with `extension: 'attestor'`. `id` is what agents name, `roles` the agent
roles it may admit (`node`, `site`, `ramdisk`), and `verify` receives:

<TypeTable
  type="{
  'evidence.attestor': { type: 'string', description: 'The attestor id from the request.' },
  'evidence.payload': { type: 'Uint8Array', description: 'JoinRequest.attestation: your evidence, in whatever format your producer writes. For join-token only, the opened token.' },
  'evidence.token': { type: 'string', description: 'The opened sealed join token, whenever the request carried one (provisioned and discovery need it).' },
  'context.tx': { type: 'SqlTx', description: &#x22;JoinService's transaction. Lock and write your rows here; they commit or roll back with the join.&#x22; },
  'context.clock': { type: 'Clock', description: 'Prefer compute_now() in SQL for anything compared with stored times.' },
  'context.claims': { type: 'HardwareClaims', description: '{ productUuid?, serial?, macs?, gpuUuids?, hostname?, arch? } as the agent claims them. Claims, not proof.' },
  'context.csrSha256': { type: 'Uint8Array', description: 'SHA-256 of JoinRequest.csr. Bind signed evidence to it, so it cannot be replayed with another key.' },
  'context.csrSpkiSha256': { type: 'Uint8Array', description: &#x22;SHA-256 of the CSR's SubjectPublicKeyInfo: the node key the certificate will certify. tpm checks that the key its AK certified is this one (cf24876).&#x22; },
  'context.role': { type: 'AgentRole', description: 'The role the agent asks for.' },
}"
/>

### What a result means [#what-a-result-means]

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant the node belongs to. With a pool it must be the pool owner.', required: true },
  role: { type: 'AgentRole', description: &#x22;Must equal the requested role, or the join is refused ('role').&#x22;, required: true },
  poolId: { type: 'string', description: &#x22;The pool it joins. The pool must exist, run bcd (agent 'bcd'), belong to tenantId, and sit in siteId when both are given. A site agent never takes a pool.&#x22; },
  siteId: { type: 'string', description: 'Required when there is no pool; the site must exist.' },
  trust: { type: 'NodeTrust', description: &#x22;Defaults to tenant-owned on a custom site, else platform.&#x22; },
  nodeId: { type: 'string', description: 'A reclaim: the node the machine proved it is. It is re-keyed in place (identity generation + 1, older certificates revoked) and keeps its state.' },
  tokenId: { type: 'string', description: &#x22;A join token you consumed. JoinService then enforces that token's expect binding against the claims.&#x22; },
  certTtlSeconds: { type: 'number', description: 'Caps the certificate lifetime: min(this, the service TTL of at most 86,400 s, and 7,200 s for ramdisks).' },
  allowedKinds: { type: 'readonly string[]', description: &#x22;Restricts the command kinds the certificate allows: exact kinds or prefix.* globs. A malformed entry fails the join with INVALID_CONFIG.&#x22; },
  bind: { type: 'Uint8Array', description: 'A 32-byte stable hardware identity (a hash). Bound to the node in node_bindings after issuing; the same subject can never admit another node.' },
  nodeKey: { type: &#x22;'tpm'&#x22;, description: &#x22;Where you proved the node key lives: 'tpm' when the machine's TPM certified the CSR's key as one it made and never lets out. Absent means not proven. JoinService records it as nodeKey in node.joined and the join audit row ('unattested' when absent).&#x22; },
  nodeKeyUri: { type: 'string', description: &#x22;A urn:bc: statement about the node key that JoinService writes into the certificate after its SPIFFE ID (tpm: urn:bc:tpm-ak:<nameAlg>:<digest>, the AK that certified it; d94f1f3). Renewals carry it forward and must prove it again for the new key, so JoinService writes it only when JoinCrypto.knowsKeyStatement accepts it (0e42018): pkiJoinCrypto knows only urn:bc:tpm-ak:, and a join returning any other nodeKeyUri is refused (node-key-statement), never issued a certificate that could not renew.&#x22; },
}"
/>

A custom attestor has no statement of its own to return. Since 0e42018 JoinService refuses a result whose
`nodeKeyUri` the configured `JoinCrypto` cannot prove at renewal (`knowsKeyStatement`, which `pkiJoinCrypto` answers
true only for a well-formed `urn:bc:tpm-ak:`; a `JoinCrypto` without the method knows none): the join fails with
`ACCESS_DENIED`, recorded as
`security.join.refused` reason `join:node-key-statement`. Before, such a join got a certificate carrying the
statement, and every renewal of it was refused (`key-statement-unknown`), so the node went offline when its first
certificate ran out. Leave `nodeKeyUri` unset unless you return `urn:bc:tpm-ak:` for a key a TPM's AK certified;
`tests/server-join-tpm-renew.test.ts` joins through an attestor returning `urn:bc:other-key:1` and sees it refused.

JoinService then checks the rest itself: the token's `expect` binding when you returned a `tokenId`, a trusted hardware
key already enrolled (a reclaim through a token needs `allowReclaim`), and that a reclaimed node keeps its tenant,
site, pool and role. It inserts the node in `adopting` (or re-keys the existing one), issues the certificate, writes
`node.joined` with `attestor: '<your id>'` and `nodeKey` (your result's, or `unattested`) and an audit row, all in the
same transaction.

`allowedKinds` is how the `discovery` attestor gives a ramdisk only `inventory.refresh`, `logs.collect`, `diag.*`,
`disk.wipe`, `deploy.*` and `boot.kexec`. Entries must be registered kind names (or `prefix.*`): matching is exact,
so `wipe` would not allow `disk.wipe`. It is stored on the certificate row (`node_certs.allowed_kinds`), renewed with it, and enforced both
when a command is enqueued and when the gateway dispatches it.

`bind` is how `tpm` ties an endorsement key to a node: on the next join, the attestor finds the bound node and returns
its `nodeId`, so a reinstall or a lost key reclaims the same node. A subject bound to another node is refused
`bound-elsewhere`. `nodes.delete` and retirement drop the node's bindings.

## Refusing [#refusing]

Throw `ComputeError('ACCESS_DENIED', 'Join refused')` for every refusal, whatever the reason: the agent must not learn
which check failed. Put the reason in `details`, which stays on the server:

* `details.refusal`: `<attestor>:<reason>`, recorded in the `security.join.refused` event (rate-bounded per attestor
  and reason, 60 a minute by default) with hashes of the CSR and claimed hardware key, never the evidence itself.
* `details.tenantId`: only once the evidence authenticated a tenant (a token's secret matched). The event then lands in
  that tenant's stream; otherwise it goes to the root tenant, so an unauthenticated caller cannot write into another
  tenant's events.

Any other error (`INVALID_INPUT`, a plain `Error`) also rolls the transaction back and fails the join, but it is not
recorded as a refusal, so reserve it for bugs, not for evidence you did not like. The attestors in `@better-compute/pki` build their refusals with an
internal `refuser(attestor, onRefusal)` helper and take an `onRefusal` hook for your own metrics; copy the pattern.

## Two-round joins [#two-round-joins]

Return an `AttestationDeferral` instead of a result when you cannot decide yet. JoinService commits the transaction, so
your own rows persist, issues nothing, spends nothing of its own, and answers with a signed deferral. The agent sends
`Join` again with **the same CSR**.

<TypeTable
  type="{
  deferred: { type: &#x22;'approval' | 'challenge'&#x22;, description: 'approval: an operator must decide; the agent retries the same request. challenge: the agent must answer your bytes.', required: true },
  requestId: { type: 'string', description: 'approval: the join request an operator approves or rejects.' },
  retryAfterSeconds: { type: 'number', description: 'approval: when the agent retries (default 30, at least 1).' },
  challenge: { type: 'Uint8Array', description: 'challenge: attestor-specific bytes, delivered inside the signed response. Required and non-empty, or the join fails INVALID_CONFIG.' },
  notAfter: { type: 'Date', description: 'challenge: answers after this are refused.' },
}"
/>

* **Approval.** `discovery` (with `approval: 'unknown-machines'` or `'always'`) and `tpm` (with `enrollment`) record a
  pending row in `join_requests` and defer; an operator decides with
  [`joinRequests.approve`](/docs/reference/api/join-requests#approve) or `reject`, and the retry consumes the approved
  request in its own transaction. The `join_requests` helpers are internal to pki, so a third-party attestor that wants
  approvals keeps its own table (`plg_<id>_*`) or builds on the built-ins.
* **Challenge.** `tpm` answers the first round with a `MakeCredential` challenge and an opaque state sealed with the
  host's `challengeKey`, writing no row, so any API replica can finish the join. The agent activates the credential and
  quotes its PCRs with a nonce bound to the CSR; the second round verifies all of it.

Two rules follow from the commit:

1. **Consume only on the final round.** Anything single-use (a join token's use, a replay marker) must be spent in the
   round that returns a result, never in a round that defers, or the retry finds it already spent. The built-ins split
   it: `checkSingleUseToken` locks and checks a token in every round, and `spendToken` decrements it only at the end.
2. **Bind the rounds to the CSR.** Put `csrSha256` into whatever you hand out, and check it when the answer comes back,
   so a challenge answered for one key cannot finish another key's join.

The agent loop is `runAgentJoin` in `@better-compute/testing`, which is also the behaviour `bcd` must implement: it
keeps one key and CSR for the whole exchange, waits `retryAfterSeconds` on `pending`, passes a `challenge` to the
producer of the same id, and verifies every response against the pinned signing key and the CSR before using it. A
deferral is never taken for a join.

## Single-use evidence [#single-use-evidence]

Evidence that anyone on the machine can read, or that could be captured in flight, must admit one join at most.
`attestation_uses` records it:

```sql
CREATE TABLE attestation_uses (
  attestor text NOT NULL,
  subject bytea NOT NULL CHECK (length(subject) = 32), -- sha256 of what must be used once
  node_id text,
  used_at timestamptz NOT NULL DEFAULT compute_now(),
  expires_at timestamptz,                              -- NULL keeps it forever
  PRIMARY KEY (attestor, subject)
);
```

Insert in the final round's transaction and refuse on conflict:

```ts
const subject = createHash('sha256').update(`${provider}\0${accountId}\0${instanceId}`).digest();
const used = await context.tx.query(
  'INSERT INTO attestation_uses (attestor, subject, expires_at) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING',
  ['acme-asset', subject, null],
);
if (used.rowCount !== 1) refuse('reused');
```

That is what pki's internal `useOnce` does. `cloud-iid` admits each (provider, account, instance) once, forever, so a
machine that lost its certificate re-joins through `x509pop`; `discovery` marks each signed site assertion until it
expires. Set `expires_at` when the evidence cannot be presented after some time: `pruneAttestationUses(tx)` from
`@better-compute/postgres` deletes expired rows in batches (5,000 by default), and rows without an expiry stay.

## The built-ins as examples [#the-built-ins-as-examples]

| Attestor      | Read it for                                                                                                                            | Factory                                                                      |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `join-token`  | the simplest case: lock a row, verify in constant time, spend one use                                                                  | built into JoinService; one you register with id `join-token` replaces it    |
| `x509pop`     | a proof bound to the CSR, the node id and the client time, a replay cache, a reclaim by `nodeId`                                       | `x509popAttestor({ graceMs?, windowMs?, replayCache? })`                     |
| `provisioned` | a single-use token plus a host-supplied resolver (`resolveTarget`) that maps it to a node                                              | `provisionedAttestor({ resolveTarget? })`                                    |
| `discovery`   | two pieces of evidence (a token and a site-signed assertion), a fencing epoch, `certTtlSeconds` and `allowedKinds`, approval deferrals | `discoveryAttestor({ resolveWorkflowNode?, approval?, … })`                  |
| `cloud-iid`   | pluggable verifiers per cloud (`CloudIdentityProvider`), a host `resolve`, `attestation_uses`                                          | `cloudIidAttestor({ providers, resolve, roles? })`                           |
| `tpm`         | a stateless challenge round, `bind`, allow-lists and enrollment                                                                        | `tpmAttestor({ challengeKey, ekRoots?, pcrPolicy?, resolve?, enrollment? })` |

They live in `packages/pki/src/attestors/`, with their agent-side producers next to them: `x509popProducer`,
`createCloudIidEvidence`, `signDiscoveryAssertion` (the site agent's half) and `tpmProducer` over a software TPM.
[Attestation](/docs/guides/nodes/attestation) explains what each one proves and how to configure it.

## Registering [#registering]

Pass verifiers to `betterCompute({ attestors })`, or ship them in a module's `extensions`. The registry collects both,
and `betterCompute()` builds the JoinService behind its gateway services with every registered extension whose
`extension` is `'attestor'`:

```ts title="compute.ts"
import { betterCompute } from 'better-compute/server';
import { x509popAttestor } from 'better-compute/pki';
import { acmeAssetAttestor } from './acme-asset';

export const compute = betterCompute({
  // database, secret, baseURL, iam, pki: { nodeCa, join }, …
  attestors: [x509popAttestor(), acmeAssetAttestor],
});
```

Hosts that run JoinService themselves call `createJoinService({ database, clock, work, events, crypto, attestors })`
from `better-compute/server`. It keeps the built-in `join-token` unless the list has one with that id, and skips
anything that is not an attestor, so a registry's whole extension list can be passed as is. Wrap a verifier with
`meteredAttestor(attestor, registry)` from `@better-compute/observability` to count verifications by outcome
(`admitted`, `deferred-<kind>`, `refused`) and time them.

## Testing [#testing]

Run the verifier through the real JoinService on PGlite, as `tests/pki-attestors-join.test.ts` does, and drive it with
`runAgentJoin` and a `NodeAttestor` producer:

```ts title="acme-asset.test.ts"
import { createJoinService, pkiJoinCrypto } from '@better-compute/server';
import { computeFixture, runAgentJoin } from '@better-compute/testing';

// trustDomain, nodeCa, signingKey, encryptionKey and vendorSign are your test's own (see the pki test above).
const fixture = await computeFixture({ clock: 'sim' });
const service = createJoinService({
  database: fixture.database,
  clock: fixture.clock,
  work: fixture.compute.context.work,
  events: fixture.compute.context.events,
  crypto: pkiJoinCrypto({ trustDomain, nodeCa, signingKey, encryptionKey }),
  attestors: [acmeAssetAttestor],
});

const joined = await runAgentJoin({
  send: (request) => service.join(request),
  signer: { publicKey: signingKey.publicKey },
  wait: async () => {},
  claims: { productUuid: '4c4c4544-0042-3510-8052-b7c04f4e4e01' },
  attestor: {
    id: 'acme-asset',
    async produce({ nonce }) {
      // nonce is sha256(csr): sign the asset tag with it, as the vendor tooling would
      return { attestor: 'acme-asset', payload: vendorSign('ASSET-0042', nonce!) };
    },
  },
});
expect(joined.rounds).toBe(1);
```

Test the refusals too: a replayed proof, a proof for another CSR, an unknown asset, the wrong role, and that each one
is the same `ACCESS_DENIED`. The control-plane simulator in `@better-compute/testing`
(`packages/testing/src/sim/control-plane.ts`) registers a minimal custom attestor, `sim-restricted`, the same way, to
join simulated nodes with restricted certificates.

  - [Attestation](/docs/guides/nodes/attestation): What each built-in attestor proves and how to configure it.

  - [Joining nodes](/docs/guides/nodes/joining): Join tokens, sealed tokens and what JoinService records.

  - [PKI](/docs/operations/pki): The node CA and the JoinService keys a join needs.


# Drivers (/docs/extending/drivers)

> Declare a driver with defineDriver, bind drivers to machines through hardware profiles, and run one out of process with the bcx driver SDK.



A driver is the code that does one hardware job for a machine: read and set its power
through the BMC, pick its boot device, write an image to its disk, or run its VMs. Each job is a
driver **kind** (Ironic calls it an interface), and one driver may implement several kinds together, which is what
Ironic calls a hardware type: a `redfish` driver serves `power`, `management` and `virtual-media` for the same BMC.

Code registers drivers; data selects them. A driver is a `defineDriver()` declaration. Which driver a machine uses for
each kind is data: a binding on the node, inherited from its pool's hardware profile.

> **What runs today.** 
  * **Declarations work.** `defineDriver`, the driver kinds, `drivers.list`, hardware profiles and per-node bindings are
    implemented and tested.
  * **The built-in drivers run in bcd, partly.** They live inside `bcd site` and `bcd ramdisk`, the Go daemon's v0.2
    modes, which are written but not released. Redfish (with BIOS settings, RAID layouts and firmware updates), PXE
    and iPXE, image streaming and kexec are there. IPMI power, boot device and inspection run in `bcx-ipmi`, a bcx plugin the
    site agent hosts (51806ab, 7818f19), and an IPMI machine enrolls and commissions to `ready` (a3ec8d0); vendor BMCs
    are not there. See [what bcd does today](/docs/daemon#what-bcd-does-today).
  * **No control-plane code calls a driver's operations yet.** Metal's BMC steps build commands for the site agent
    instead. Direct calls from TypeScript (`ctx.drivers.for(node)`) are planned for v0.5.
  * **bcx works in TypeScript and in bcd.** `@better-compute/driver-sdk` (v0.5 draft) serves and hosts plugins end to
    end, with real subprocesses in its tests. bcd hosts sha256-pinned, sandboxed plugins from its `[plugins] dir` and runs
    `driver.invoke` in them (`daemon/internal/bcx`, Go unit tests only); see
    [driver plugins in bcd](#driver-plugins-in-bcd). The control plane still does not merge the drivers a daemon
    advertises into its registry.

## Driver kinds [#driver-kinds]

`DriverKinds` in `@better-compute/core` is empty. Packages open it by declaration merging, so a kind type-checks only
where the package that declares it is imported:

| Kind            | Declared by | Operations                                                                                            | Built-in drivers                                                                                                                                                                                            |
| --------------- | ----------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `power`         | metal       | `getPower()`, `setPower(action)`                                                                      | v0.2                                                                                                                                                                                                        |
| `management`    | metal       | `setBootDevice(device, { persistent, uefi })`, optional `rotateCredential(newPassword, { username })` | v0.2                                                                                                                                                                                                        |
| `boot`          | metal       | `prepareRamdisk(spec)`, `prepareInstance(boot)`, `cleanUp()`                                          | v0.2                                                                                                                                                                                                        |
| `deploy`        | metal       | `deploy(spec)`, `tearDown({ wipe })`, both `AsyncIterable<Progress>`                                  | v0.2                                                                                                                                                                                                        |
| `inspect`       | metal       | `inspect()`, a `NodeInventory`                                                                        | v0.2                                                                                                                                                                                                        |
| `virtual-media` | metal       | `insert(iso)`, `eject()`                                                                              | v0.2                                                                                                                                                                                                        |
| `console`       | metal       | `open()`, a byte stream                                                                               | draft until v0.3                                                                                                                                                                                            |
| `bios`          | metal       | `getSettings()`, `applySettings(settings)`                                                            | draft until v0.5                                                                                                                                                                                            |
| `raid`          | metal       | `applyLayout(layout)`, `clear()`                                                                      | draft until v0.5                                                                                                                                                                                            |
| `firmware`      | metal       | `components()`, `update(component, image)`                                                            | draft until v0.5                                                                                                                                                                                            |
| `runtime`       | vps         | `traits`, `list()`, `apply(spec)`, `reboot(id)`, `remove(id)`, optional `console(id)`                 | v0.3 (in bcd: `libvirt-kvm`, 362317c, and `cloud-hypervisor`, 2598f0a, both written and tested against fake tools only); bcd also serves a bcx plugin of this kind as a runtime (`list`, `apply`, `remove`) |

Importing only `@better-compute/core` leaves `power` and `runtime` unusable, and `tests/stubs-driver-kinds.test.ts`
checks exactly that. [BMC drivers and credentials](/docs/guides/bare-metal/bmc) describes what each metal kind is for.

A new kind is the same declaration in your own package:

```ts title="kinds.ts"
declare module '@better-compute/core' {
  interface DriverKinds {
    'acme-cooling': { getFanCurve(): Promise<number[]>; setFanCurve(curve: number[]): Promise<void> };
  }
}
```

## Declaring a driver [#declaring-a-driver]

```ts title="drivers/acme-bmc.ts"
import { defineDriver, field, manifest } from 'better-compute/core';
import 'better-compute/metal'; // merges the metal kinds into DriverKinds

export const acmeBmc = defineDriver({
  id: 'acme-bmc',
  version: '1.0.0',
  runsOn: 'site',
  config: manifest({
    address: field.url(),
    credentials: field.secret(),
    timeout: field.duration().default('30s'),
  }),
  capabilities: ['virtual-media'],
  implements: {
    power: (config, ctx) => ({
      async getPower() {
        return 'on';
      },
      async setPower(action) {
        ctx.emit({ phase: 'power', message: `setting ${action}` });
      },
    }),
    'virtual-media': (config) => ({ insert: async (iso) => {}, eject: async () => {} }),
  },
  retrySafe: { power: ['getPower'], 'virtual-media': ['insert', 'eject'] },
  async validate(config, ctx) {
    await ctx.secrets.resolve(config.credentials); // a live check, e.g. log in once
  },
});
```

`defineDriver` returns the definition frozen with `extension: 'driver'`.

<TypeTable
  type="{
  id: { type: 'string', description: 'The driver id bindings name, such as redfish or ipmi.', required: true },
  version: { type: 'string', description: 'The implementation version. kind:id@version must be unique in a deployment.', required: true },
  runsOn: { type: &#x22;'control-plane' | 'site' | 'node' | 'ramdisk'&#x22;, description: 'Where the driver executes.', required: true },
  config: { type: 'Manifest<C>', description: &#x22;The per-binding config. Declare credentials with field.secret(), so they stay SecretRefs.&#x22;, required: true },
  capabilities: { type: 'readonly string[]', description: 'Free-form flags such as virtual-media or uefi-http.' },
  implements: { type: '{ [kind]: (config, ctx) => DriverKinds[kind] }', description: 'One factory per kind; it may be async.', required: true },
  retrySafe: { type: 'Partial<Record<kind, readonly string[]>>', description: 'Per kind, the operations safe to run again after an interruption.' },
  validate: { type: '(config, ctx) => Promise<void>', description: 'Live checks beyond the manifest; throw to refuse.' },
}"
/>

Every factory receives the parsed config and a `DriverContext`:

<TypeTable
  type="{
  node: { type: 'NodeRef', description: 'The node acted on, when there is one.' },
  idempotencyKey: { type: 'string', description: 'Stable across retries of the same operation.' },
  signal: { type: 'AbortSignal', description: 'Aborted when the caller gives up.' },
  deadline: { type: 'number', description: 'Epoch ms the operation must finish by, when set.' },
  traceparent: { type: 'string', description: 'W3C trace context of the calling step.' },
  clock: { type: 'Clock', description: 'Use it instead of Date.now(), so simulations stay deterministic.' },
  secrets: { type: 'SecretResolver', description: 'resolve(ref) returns the value; keep it in memory only.' },
  log: { type: 'Logger', description: 'debug, info, warn and error with structured fields.' },
  emit: { type: '(progress: Progress) => void', description: '{ phase?, message?, percent?, bytes? } for long operations.' },
}"
/>

Mark an operation retry-safe only when running it twice does no harm. A power cycle is not retry-safe; reading the
power state is. The flag decides whether an interrupted call may be repeated, and, over bcx, whether an error the
plugin returns is marked `retryable`.

## Registering and listing [#registering-and-listing]

Pass drivers to `betterCompute({ drivers })`, or ship them in a module's `extensions` (see
[modules](/docs/extending/modules)). The registry refuses a second `driver:<id>@<version>` with `INVALID_CONFIG`.

[`drivers.list`](/docs/reference/api/drivers#list) returns what the deployment registered, so `metal.enroll` callers
and the console can offer valid choices. It returns declarations only, never a configuration or a credential:

```ts
const { items } = await compute.api.drivers.list(credential, { tenantId: 'root', kind: 'power' });
// [{ id: 'acme-bmc', version: '1.0.0', runsOn: 'site', kinds: ['power', 'virtual-media'],
//    capabilities: ['virtual-media'], retrySafe: { power: ['getPower'], 'virtual-media': ['eject', 'insert'] } }]
```

It is decided as `compute:drivers:read` on the tenant, filters by `kind` (lower-case letters, digits and hyphens), and
sorts by id, then version, with kinds, capabilities and operations sorted too.

## Binding drivers to machines [#binding-drivers-to-machines]

Machines of one pool usually share their BMC and boot hardware, so bindings live in a hardware profile that
`metal({ hardwareProfiles })` names and a pool selects with `policy.hardwareProfile`:

```ts
import { metal } from 'better-compute/metal';

const bareMetal = metal({
  hardwareProfiles: {
    'gpu-redfish': {
      description: 'HGX hosts with Redfish BMCs',
      drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
    },
  },
});
```

A binding is a driver id, or `{ driver, config }` when the binding carries config. At `metal.enroll`,
`resolveDriverBindings` builds the node's bindings in three layers, each overriding the one before per kind:

1. The enrolled BMC binds `power`, `management` and `virtual-media` to `bmc.driver`, with `{ address, credentials }`
   as config (the credentials as a `secret://` reference).
2. The hardware profile: the enroll input's `hardwareProfile`, else the pool's `policy.hardwareProfile`. An unknown
   name is `INVALID_INPUT`.
3. Per-node overrides.

The result is stored on the node (`nodes.drivers`) and served in its NodeSpec. `metal.enroll` checks only that
`bmc.driver` is present and a well-formed id string; PLAN.md has bindings validated against the registry, which is not done yet. The
hardware profile also carries the BIOS settings and RAID layout that commissioning applies (drafts until v0.5); see
[profiles and images](/docs/guides/bare-metal/profiles-and-images).

### From a profile step to a driver [#from-a-profile-step-to-a-driver]

A profile step names a driver operation as `<concern>.<operation>` with `on:` set to an agent. When no registered
step has that id, the workflow engine turns it into one `driver.invoke` command whose args carry the concern, the
operation, the step's `with`, and the driver and config bound in the profile's own `drivers`:

```json title="driver.invoke payload"
{
  "workflowId": "wf_…",
  "stepId": "main.3",
  "attempt": 0,
  "targetNodeId": "node_…",
  "args": { "concern": "bios", "operation": "applySettings", "args": { "…": "…" }, "driver": "acme-bmc" }
}
```

`targetNodeId` appears when the command goes to the site agent rather than the machine itself. `provisionModule()`
registers the `driver.invoke` kind: lane `provision`, destructive (so always signed, since driver operations include
power and deploy), not retry-safe, roles `node`, `site` and `ramdisk`, payload `DriverInvokeRequest`, and a 10-minute
default timeout (each step's own timeout sets `notAfter`). `tests/e2e/bcx.test.ts` runs such steps end to end: a
module profile binds `power.getPower` and `power.setPower` to a bcx plugin's driver, and both go out as
`driver.invoke` through the real engine, queue and gateway to a FakeDaemon that runs them in the plugin process
(`fakeDriverInvoke` from `@better-compute/testing`). bcd runs `driver.invoke` with the same semantics in the
plugins it hosts (see [driver plugins in bcd](#driver-plugins-in-bcd)); no end-to-end test runs bcd for it yet. Metal's own steps
(`power.setPower`, `media.insert`, ...) are registered control-plane steps that relay typed commands instead; see
[workflows](/docs/guides/bare-metal/workflows).

## BMCs on the site agent [#bmcs-on-the-site-agent]

The control plane never dials into a site. Metal's BMC steps run in the control plane, where the node's BMC address and
credential reference are, and send one `BmcOperationRequest` (`driver`, `address`, `credentialsRef`, `operation`,
`argsJson`, optional `tlsSpkiSha256` and `driverOptionsJson`) to the site's agent under a kind such as `bmc.query`, `bmc.boot`,
`bmc.power`, `bmc.rotate` or `bmc.firmware`. The site agent fetches the credential by reference, runs `<concern>.<operation>` on the named driver,
and answers a `BmcOperationResult`. PLAN.md drives BMCs through bmclib (Redfish, IPMI and vendor BMCs); `bcd site`
has its own Redfish client built in (see [Redfish in bcd site](/docs/guides/bare-metal/bmc#redfish-in-bcd-site)),
and since 51806ab it sends a `bmc.*` operation whose `driver` is anything else to a bcx plugin it hosts:

* It picks the hosted driver whose id is the request's `driver` and whose manifest lists the operation's concern
  among its kinds (`power.setPower`: concern `power`, operation `setPower`). No such driver answers
  `unsupported-driver` ("no hosted driver ipmi implements power"), and nothing is invoked.
* It calls `Invoke` with the concern as `kind`, the operation, the BMC `address`, `tlsSpkiSha256` (hex) and the
  request's `driver_options_json` (as `options`) as `config_json`, the remaining arguments as `args_json`, the
  command id as the idempotency key, the command's `traceparent` and its `not_after` as the deadline. The options are
  nested, so they can never override the address or the key pin (7753df0). Options that are not a JSON object fail the
  command (`invalid-payload`); the built-in Redfish driver takes none, so a Redfish request with options answers
  `invalid-config`.
* The credential is fetched by reference as for Redfish and sent only as secrets (`username`, `password`, and a
  rotation's `newPassword`, which is taken out of the arguments), never in the arguments or the config.
* The kind's operation allowlist, the destructive-batch quorum and the site agent's memory of BMC addresses (which
  `site.httpProxy` refuses) apply exactly as for Redfish. A `DriverError` keeps its code; a result that is not JSON is
  `bmc-error`.

A new BMC family is therefore a bcx plugin on the site agent, whose driver id the node's BMC binding names. Its
operations can also run through profile steps sent as `driver.invoke`. The command kinds and credential handling are
on [BMC drivers and credentials](/docs/guides/bare-metal/bmc).

### bcx-ipmi, the IPMI driver [#bcx-ipmi-the-ipmi-driver]

`daemon/cmd/bcx-ipmi` is the first Go plugin in the repository (51806ab): driver `ipmi`, kinds `power`,
`management` and, since 7818f19, `inspect`, `runsOn: site`, over `ipmitool` with IPMI-over-LAN (`-I lanplus`).

| Operation                                           | `ipmitool` command                                                                                                       | Answer                                                   |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- |
| `power.getPower`                                    | `chassis power status`                                                                                                   | `{"power":"on"}` or `{"power":"off"}`                    |
| `power.setPower` (`action`)                         | `chassis power on`, `off`, `cycle`, `reset` or `soft` (`shutdown` is `soft`)                                             | `{}`                                                     |
| `management.setBootDevice` (`device`, `persistent`) | `chassis bootdev pxe`, `disk`, `cdrom` or `bios` (`hdd` is `disk`, `cd` is `cdrom`), plus `options=persistent`           | `{}`                                                     |
| `inspect.inspect`                                   | `fru print 0` and `chassis power status` (required), then `mc info`, `mc guid`, `lan print` and `sdr list` (best effort) | The Redfish driver's shape, less the host's NICs (below) |

`inspect.inspect` answers:

| Field                                   | From                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `manufacturer`, `model`, `serialNumber` | The builtin FRU (`fru print 0`): `Product Manufacturer`, `Product Name`, `Product Serial`, falling back to the board's (and for the serial the chassis') fields                                                                                                                                                                             |
| `power`                                 | `chassis power status`                                                                                                                                                                                                                                                                                                                      |
| `macs`, `ipAddresses`                   | Always empty: IPMI does not name the host's NICs reliably                                                                                                                                                                                                                                                                                   |
| `productUuid`                           | The `System GUID` of `mc guid` (Get System GUID), lower-cased, when it is a canonical UUID other than all zeros (7cd1dd2). It is the host's SMBIOS UUID on most boards, in whichever byte order the BMC prints it; metal matches every order, so a machine enrolled without MACs is found when it PXE-boots. Left out when the BMC has none |
| `board`                                 | `{ manufacturer, product, serial }` from the FRU's board fields                                                                                                                                                                                                                                                                             |
| `bmc`                                   | `{ firmware, manufacturer }` from `mc info` and `{ mac, ipAddress }` from `lan print`: the BMC's own identity, not the host's                                                                                                                                                                                                               |
| `sensors`                               | `{ count, ok, problems }` from `sdr list`; `problems` names at most 32 sensors whose status is neither `ok` nor `ns`                                                                                                                                                                                                                        |

A failed FRU read or power read fails the operation; a failed `mc info`, `lan print` or `sdr list` only leaves its
part out. Each `ipmitool` run is a separate process with the password through its own pipe.

* **Config** is `{ "address": "10.0.0.5[:623]", "options": { "cipherSuite": 17 } }`: a host name or IP address
  with an optional port (a `scheme://` prefix is stripped; anything else is `invalid-address`), and the BMC's driver
  options, which the site agent nests under `options` exactly as enrollment set them (`bmc.driverOptions`, 7753df0).
  The plugin decodes them strictly: `cipherSuite` is the only key, and any other key is `invalid-config`. The RMCP+
  cipher suite is 17 (HMAC-SHA256, AES-128) by default, or 3, 8 or 12 for older BMCs: only suites with an integrity
  HMAC and AES. Suite 0 (the IPMI authentication bypass), 1 and 2 (no encryption) and anything else are
  `invalid-config`. The manifest's `config_schema_json` says the same. Enroll an old BMC with
  `bmc.driverOptions: { "cipherSuite": 3 }`; see [enrolling a machine](/docs/guides/bare-metal/bmc#enrolling-a-machine).
* **No key pin.** IPMI has no TLS, so a BMC enrolled with a `tlsSpkiSha256` pin is refused (`pin-unsupported`) rather
  than driven unpinned: use a Redfish driver, or remove the pin.
* **Credentials** come as the secrets `username` and `password`. The password reaches `ipmitool` through a pipe it
  reads once (`-f /dev/fd/3`), never its arguments (a process listing) nor its environment (readable in `/proc` by
  any process of the same user); an IPMI 2.0 password has at most 20 bytes. `ipmitool` runs by absolute path (the
  first of `/usr/bin`, `/usr/sbin`, `/usr/local/bin` and `/usr/local/sbin` that has it) with a fixed
  `PATH=/usr/sbin:/usr/bin:/sbin:/bin` and nothing else in its environment (933c51b). A username outside
  `[A-Za-z0-9._@-]{1,64}`, a missing password, a longer one, or one with whitespace (`ipmitool -f` keeps only the
  file's first word; b12c217) is `credentials-unavailable`.
* **Arguments** are checked against fixed sets before anything runs: an unknown action or device is `invalid-args`,
  and any other operation is `unsupported-operation`.
* **Failures.** Each `ipmitool` run has 30 seconds (`unreachable`, retryable, on timeout); a failing `ipmitool` is
  `bmc-error` (retryable) with the first line of its output. `Health` is `NOT_SERVING` when `ipmitool` is not
  installed.

It covers power, boot device and inspection: virtual media, firmware, BIOS, RAID and credential rotation stay
Redfish's, though a catalog ISO still boots on an `ipmi` machine over PXE (`sanboot`; see
[sanboot instead of virtual media](/docs/guides/bare-metal/bmc#sanboot-instead-of-virtual-media)). goreleaser builds it without cgo and ships it as `bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz` (348ff56),
and the api's downloads route serves that archive (97b2ac2; see
[serving the binaries](/docs/daemon/install#serving-the-binaries)); `daemon install-script --install-plugins bcx-ipmi`
installs it with its pin after checking the archive against the signed release manifest (adf2ae9; see
[installing plugins from the release](/docs/daemon/install#installing-plugins-from-the-release)). To use it, have it
in a site agent's `[plugins] dir` (below) and enroll the machine's BMC with driver `ipmi`, without a key pin, and in a pool
with an agent with its PXE `macs` or its `productUuid` (PXE discovery then proposes the MAC for `metal.confirmMac`;
3d26428, 8de6e2f), since IPMI inspection finds no MACs. Metal knows what `ipmi` serves and refuses
up front, with `UNSUPPORTED` (reason `bmc-driver-unsupported`), any verb whose workflows need more (a3ec8d0); see
[IPMI through bcx-ipmi](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi) and
[what metal checks before a workflow](/docs/guides/bare-metal/bmc#what-metal-checks-before-a-workflow). Its tests
run it against a fake `ipmitool` only.

A driver of your own is checked the same way once metal knows its operations: `metal({ bmcDrivers: { 'acme-bmc': ['power.getPower', 'power.setPower', 'management.setBootDevice'] } })`. Otherwise the kinds its site agents report
(the `bc.dev/driver-kind.<id>.<kind>` labels each agent's Hello writes) decide, and a driver neither names is not
checked in advance.

## Out-of-process drivers: bcx [#out-of-process-drivers-bcx]

bcx is one protocol, `bettercompute.driver.v1.DriverPlugin`, for drivers that run in their own process, in any
language, next to `bcd` or the control plane. It is go-plugin style: the host execs the plugin, the plugin prints one
handshake line, and the host dials it over gRPC on HTTP/2 with mutual TLS.

| RPC        | Does                                                                                                                 |
| ---------- | -------------------------------------------------------------------------------------------------------------------- |
| `Describe` | the drivers the plugin serves, as `DriverManifest`s, its version and the bcx protocol version (1)                    |
| `Validate` | checks a config against the manifest and the driver's live checks; answers `{ valid, issues[] }` with JSON pointers  |
| `Invoke`   | runs one operation; streams `Progress`, then exactly one `result_json` or `DriverError { code, message, retryable }` |
| `Stream`   | a duplex byte stream (consoles); the first frame is `open`                                                           |
| `Health`   | `SERVING` or `NOT_SERVING`                                                                                           |

A `DriverManifest` is `{ id, version, kinds[], runs_on, config_schema_json, capabilities[], retry_safe_operations[] }`,
with the config schema as JSON Schema 2020-12 (`manifest().jsonSchema()`) and retry-safe operations as
`<kind>.<operation>`. The proto is draft and excluded from `buf breaking` until v0.5 ships.

<Mermaid
  chart="sequenceDiagram
  participant H as Host (externalDriver or bcd)
  participant P as Plugin (serveDriver)
  H->>H: check sha256 pin, make an ephemeral P-256 cert
  H->>P: exec with BCX_MAGIC_COOKIE, BCX_SOCKET_DIR, BCX_CLIENT_CERT
  P->>P: listen on a unix socket (tcp 127.0.0.1 on Windows)
  P-->>H: stdout: bcx|1|unix|<socket>|grpc|<cert>
  H->>P: dial with mTLS, both certificates pinned
  H->>P: Describe
  P-->>H: DriverManifests, protocol_version 1
  H->>P: Invoke(driver, kind, operation, config, args, secrets, key, deadline)
  P-->>H: Progress ... then result_json or DriverError"
/>

### The plugin side [#the-plugin-side]

`serveDriver` serves **the same `defineDriver` object** you would register in process:

```ts title="acme-bmc-plugin.ts"
#!/usr/bin/env node
import { serveDriver } from '@better-compute/driver-sdk';
import { acmeBmc } from './drivers/acme-bmc.js';

await serveDriver(acmeBmc, { pluginVersion: '1.0.0' }); // or serveDriver([acmeBmc, acmeMedia], …)
```

Run by hand, without the magic cookie, it prints that it is a plugin and exits with code 1. Started by a host, it
prints the handshake line on stdout, writes JSON log lines to stderr (the host forwards them to its own log), shuts
down gracefully on SIGTERM or when its stdin closes, and ignores SIGINT.

| Option            | Default                         | Does                                                                                                                             |
| ----------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `pluginVersion`   | `''`                            | `DescribeResponse.plugin_version`                                                                                                |
| `streams`         | none                            | operations, as `<kind>.<operation>`, that return a byte stream; each is advertised as capability `bcx.stream:<kind>.<operation>` |
| `network`         | `unix`, or `tcp` on Windows     | where to listen                                                                                                                  |
| `log`             | JSON lines on stderr            | the plugin's logger                                                                                                              |
| `handleSignals`   | `true`                          | the SIGTERM, stdin and SIGINT behaviour above                                                                                    |
| `shutdownGraceMs` | 5,000                           | how long in-flight calls may run after shutdown starts                                                                           |
| `env`, `stdout`   | `process.env`, `process.stdout` | for tests                                                                                                                        |

Throw `driverError(code, message)` for an error of your own: the code must be lower-case kebab-case, and the host sees
`DRIVER_ERROR` with `details.driverCode`. A thrown `ComputeError` travels as its code in kebab-case (`INVALID_CONFIG`
becomes `invalid-config`); any other error is `driver-error` with its message.

### The host side [#the-host-side]

```ts title="compute.ts"
import { externalDriver } from '@better-compute/driver-sdk';

const acme = externalDriver({ command: ['/opt/bc/plugins/acme-bmc'], sha256: '9f3e…' });
export const compute = betterCompute({ drivers: [...builtins, ...(await acme.start())] /* , … */ });
// on shutdown: await acme.close();
```

`start()` verifies the pin, execs the plugin, reads the handshake within `startTimeoutMs`, dials it and calls
`Describe`. It resolves with one `RemoteDriverDefinition` per served driver, a normal `DriverDefinition` whose parts
are backed by the plugin:

* `implements[kind](config, ctx)` returns a proxy whose every method calls `Invoke`. Awaiting a call sends progress to
  `ctx.emit`; iterating it (`for await (const p of deploy(spec))`) yields the progress instead.
* `config` is a Standard Schema built from the plugin's JSON Schema. The host checks structure (types, enums,
  patterns, lengths, ranges, required and unknown fields) and passes the value through unchanged; the plugin parses it
  again, applying defaults.
* `validate` calls the plugin's `Validate`; `retrySafe` and `capabilities` come from the manifest.

`startExternalDrivers([a, b])` starts several hosts and returns all their drivers.

| Option            | Default                                      | Does                                                                                    |
| ----------------- | -------------------------------------------- | --------------------------------------------------------------------------------------- |
| `command`         | required                                     | the command line, run without a shell; `command[0]` is resolved against `cwd` or `PATH` |
| `sha256`          | required                                     | hex SHA-256 of the pinned file, checked before every start and restart                  |
| `env`             | none                                         | extra environment for the plugin                                                        |
| `cwd`             | `process.cwd()`                              | the plugin's working directory                                                          |
| `startTimeoutMs`  | 30,000                                       | time to print the handshake and answer `Describe`                                       |
| `callTimeoutMs`   | none                                         | deadline of calls whose context carries no `deadline`                                   |
| `restart`         | `{ initialDelayMs: 250, maxDelayMs: 30000 }` | crash restart back-off, doubling                                                        |
| `shutdownGraceMs` | 5,000                                        | between stdin EOF and SIGTERM, and SIGKILL                                              |
| `log`             | warnings on stderr                           | plugin stderr, exits and restarts                                                       |

### Wire rules [#wire-rules]

* **Pinning.** `sha256` pins `command[0]`. When `command[0]` is `node`, `nodejs` or `bun`, it pins the script instead:
  the first later argument not starting with `-` (after `bun`'s optional `run`). Interpreter options must use the
  `--name=value` form, and only the entry file is pinned, so bundle a JS plugin into one file. A mismatch is
  `INVALID_CONFIG`, and the host stops restarting.
* **Mutual TLS.** Each side makes an ephemeral self-signed P-256 certificate. The plugin accepts exactly the host's
  certificate from `BCX_CLIENT_CERT`, not merely one signed by the same key; the host trusts only the certificate in the
  handshake line.
* **Environment.** The plugin does not inherit the host's environment: it gets `PATH`, the temp directories, `HOME`, a
  few OS basics, your `env`, and the three `BCX_*` variables.
* **Arguments and results.** `args_json` is a JSON array of the positional arguments (trailing `undefined`s dropped);
  a non-array value from another host is taken as the single argument. Results are JSON, and `undefined` travels as
  `null`.
* **Deadlines.** The host sends `ctx.deadline`, or now plus `callTimeoutMs`. The plugin turns it into `ctx.signal` and
  ends the call with `timeout` even when the operation ignores the signal; the host gives up one second later.
  Aborting the caller's `ctx.signal` cancels the call.
* **Secrets.** The host resolves only top-level config fields the schema declares secret (`x-bc.kind: 'secret'` or
  `writeOnly`), and only when their value is a `SecretRef`, and sends them in the request. In the plugin,
  `ctx.secrets.resolve(ref)` resolves only refs delivered that way.
* **Streams.** For an advertised stream operation the proxy opens `Stream` instead of `Invoke`. The plugin answers the
  `open` frame with an empty `data` frame, then bytes flow both ways until `close`. The deadline bounds opening the
  stream, not its lifetime.
* **Supervision.** A crash or a lost connection fails calls in flight with a retryable `DRIVER_ERROR`
  (`driverCode: 'unreachable'`), restarts the plugin with back-off, and makes new calls wait for the restart. A
  restarted plugin that describes other drivers is logged and the registered ones are kept. `close()` closes stdin,
  sends SIGTERM (not on Windows), then SIGKILL after `shutdownGraceMs`.

### Driver plugins in bcd [#driver-plugins-in-bcd]

bcd hosts bcx plugins in `bcd node` and `bcd site`, with the Go side of the protocol in `daemon/internal/bcx`.
(`bcd ramdisk` accepts `driver.invoke` too, but it has no config file, so no plugin directory and no plugins.) Point
the agent at a plugin directory in `config.toml`; there is no default in bcd, and `daemon install-script --plugins`
writes the one below (see [installing a plugin node](#installing-a-plugin-node)):

```toml title="/etc/bcd/config.toml"
[plugins]
dir = "/usr/local/lib/bcd/plugins"
```

The directory holds the plugin binaries and a `pins` file in `sha256sum` format, one `<sha256 hex>  <file name>` line
per plugin (`#` comments and blank lines are skipped). Keep both root-owned, like `config.toml`:

```text title="/usr/local/lib/bcd/plugins/pins"
9f3e0c…  acme-bmc
```

At start, before its session opens, the agent runs every plugin the `pins` file names:

1. It opens the binary, hashes it through that open file and refuses to run it unless the SHA-256 equals its pin.
   On Linux the plugin is then executed from the same open file (fd 3, `/proc/self/fd/3`), never by path again, so
   what runs is what was hashed even if the file at the path is replaced in between.
2. It makes an ephemeral self-signed P-256 certificate and a fresh 0700 socket directory, and execs the plugin with
   only `PATH`, `BCX_MAGIC_COOKIE`, `BCX_SOCKET_DIR` and `BCX_CLIENT_CERT` in its environment, sandboxed as
   [below](#the-plugin-sandbox). The plugin's stderr goes to bcd's.
3. It waits up to 10 seconds for the handshake line (`bcx|1|unix|<socket>|grpc|<cert>`), dials the socket over gRPC
   with mTLS, pinning the plugin's certificate byte for byte, and calls `Describe`; a plugin that reports a bcx
   protocol other than 1 is stopped.
4. `Hello.drivers` advertises each driver the plugins describe, so the gateway labels the node
   `bc.dev/driver.<id>=<version>`.

A missing `pins` file runs no plugin (logged as a warning); an unreadable pin line, a pin mismatch or a plugin that
fails to start is logged and skipped. Plugins are stopped with the agent. Two paths reach them: `driver.invoke`,
below, and, on `bcd node`, the `instances` converger, which runs a hosted driver of kind `runtime` whose id is not a
built-in runtime as one of its runtimes (`list`, `apply(spec)`, `remove(id)` over `Invoke`; snapshots unsupported).
See [walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx).

bcd supervises every plugin that started (`bcx.Supervise`, 9006eee, PLAN.md §5.5). When a plugin's process exits,
bcd starts it again after a back-off that begins at 1 second and doubles to at most 1 minute (back to 1 second once a
run lasted a minute), and each restart hashes the binary against its pin and sandboxes it again. An operation that
arrives during a restart fails with a retryable `unavailable` error. A plugin that comes back with other drivers is
logged, and the node keeps advertising the first set until bcd itself restarts. A plugin that fails its first start
is still only logged and skipped, and bcd does not call `Health` on its own.

#### The plugin sandbox [#the-plugin-sandbox]

On Linux bcd runs every plugin confined, as PLAN.md §5.5 asks, set by the rest of the `[plugins]` section:

```toml title="/etc/bcd/config.toml"
[plugins]
dir = "/usr/local/lib/bcd/plugins"
# user = "bcx"               # unset (the default): each plugin gets its own uid
privileged = "kata"          # comma-separated: plugins that run as root, unfiltered
sandbox = true               # the default: no_new_privs + seccomp through bcd's shim
cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins"  # in bcd's delegated cgroup
memory_max = 1_073_741_824   # bytes per plugin; 0 or unset: no limit
pids_max = 256               # tasks per plugin; 0 or unset: no limit
```

| Key          | Default | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user`       | none    | When bcd runs as root, unset gives each plugin its own uid and gid from a block bcd keeps for itself (`0x7FFE0000` to `0x7FFEFFFF`, at the top of the range systemd leaves unassigned, clear of system users, systemd's `DynamicUser` range and container ranges), so no plugin can read another's memory. The uid is derived from the plugin's name, moves past any uid another plugin or an existing account has, and plugins are assigned in name order, so a uid is stable across restarts and reordering the `pins` file changes nothing (6827df8). Set, every plugin runs as that account and its primary group. Either way no supplementary groups, and the socket directory is chowned to the plugin. `root` (or `0`) is a configuration error, and an account that resolves to uid 0 or does not exist keeps that plugin from running (logged). Ignored when bcd is not root |
| `privileged` | none    | Plugins, by name and comma-separated, that run as root without the shim or the filter: a runtime plugin that makes namespaces and mounts (Kata). Each is logged at start as a privileged plugin. Every other plugin is confined                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `sandbox`    | `true`  | Runs each unprivileged plugin through bcd itself as a shim (the hidden `bcd __bcx-exec <name>` command, `bcx.RunShim`): on one locked OS thread it sets `no_new_privs`, installs a seccomp filter synchronised to every thread (`SECCOMP_FILTER_FLAG_TSYNC`), then execs the verified binary from fd 3. `false` drops the shim; the uid still applies                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `cgroup`     | none    | A directory under `/sys/fs/cgroup` inside bcd's delegated cgroup; each plugin, privileged ones included, runs in its own child `bcx-<name>`. bcd prepares it at start (below)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `memory_max` | none    | Written to the child cgroup's `memory.max`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `pids_max`   | none    | Written to the child cgroup's `pids.max`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

The seccomp filter (amd64 and arm64) answers `EPERM` to what no driver plugin needs and a compromised one could abuse:
kexec, loading and deleting kernel modules, mounts (`mount`, `umount2`, `pivot_root` and the new mount API),
namespaces (`unshare`, `setns`, and `clone` with any `CLONE_NEW*` flag), swap, reboot, `bpf`, `io_uring_*`, `ptrace`
and `process_vm_readv`/`writev`, the kernel keyring, `perf_event_open`, `userfaultfd`, `open_by_handle_at`,
`iopl`/`ioperm` (amd64), setting the clock, `acct`, `quotactl`, `lookup_dcookie` and `syslog`. `clone3` answers
`ENOSYS`, since its flags are out of a filter's reach and libc and Go fall back to `clone`. So a plugin cannot make a
user and network namespace to reach `nf_tables`. A system call from another ABI (x32, a foreign architecture) kills
the process. On other architectures the shim installs no filter.

Every plugin also runs in its own process group, is killed if bcd dies (`Pdeathsig`), and is killed as a group on
close, so whatever it started goes with it. The sandbox is Linux only: on another OS bcd runs a plugin by path and
refuses to start it when a shim, a user or a cgroup is asked for, so there a plugin runs only with
`sandbox = false`. The Go tests check that a sandboxed plugin's `perf_event_open` and `unshare(CLONE_NEWUSER)` get
the filter's `EPERM`, that `/proc/<pid>/exe` is the verified binary, and that the keys parse and validate.

With `cgroup` set, bcd prepares the directory before it starts any plugin (d902063): it creates it and enables the
memory and pids controllers in it and in its parent. cgroup v2 gives no controllers to the children of a cgroup that
holds processes itself, so when bcd finds itself running directly in the parent (systemd older than 254, without
`DelegateSubgroup=main`), it first moves itself into a `main` leaf beside the plugins. A path outside
`/sys/fs/cgroup`, or a cgroup bcd cannot write (not delegated), means no plugin runs, and the log says why. To set a
node up for all this, see [installing a plugin node](#installing-a-plugin-node).

`driver.invoke` then runs in the plugins, as the FakeDaemon's `fakeDriverInvoke` does:

| Step            | bcd                                                                                                                                                                                                                                                                        |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pick the driver | the one hosted driver whose manifest lists the concern among its kinds, restricted to `driver` when the args name one; none is `driver-not-found`, several are `ambiguous-driver` (the step must name one)                                                                 |
| Secrets         | every top-level `config` field whose value is a `secret://`, `vault://` or `env://` reference is fetched with a node-signed `SecretRequest` bound to the workflow step and sent as a `SecretValue`; the config keeps the reference. A failed fetch is `secret-unavailable` |
| Request         | `InvokeRequest` with the concern as `kind`, the operation, `config_json`, `args_json` (the step's args), the idempotency key `wf:<workflowId>:<stepId>#<attempt>`, the command's `traceparent`, and the command's `not_after` as the deadline, which also cancels the call |
| Result          | `DriverInvokeResult { result_json }`; a `DriverError` keeps its code, any other failure is `driver-error`                                                                                                                                                                  |

A Go plugin serves the protocol with `bcx.Serve(ctx, impl, os.Stdout)`, where `impl` is a
`driverv1connect.DriverPluginHandler`: it checks the magic cookie, listens on `plugin.sock` in the host's directory,
accepts only the host's certificate from `BCX_CLIENT_CERT`, and prints the handshake line. It is the start of a Go
plugin SDK, internal to the daemon module for now. The Go unit tests run a pinned plugin (the test binary executed
again as one), a refused unpinned binary, and `driver.invoke` through a hosted driver with a secret; no end-to-end
test runs bcd with a plugin against the control plane yet.

The control plane does not merge `Hello.drivers` into its driver registry. What it does with them today is label
the node: at every `Hello` the gateway writes each advertised driver as the system label
`bc.dev/driver.<id>=<version>`, each kind a driver serves as `bc.dev/driver-kind.<id>.<kind>=1` (14ceb1e; kinds
outside `[a-z][a-z0-9-]`, 1 to 32 characters, skipped, at most 32 per driver), and each `Hello` capability as
`bc.dev/cap.<name>=1` (such as `bc.dev/cap.instance.network-policy`), and drops the ones no longer advertised, so
placement can require a runtime, a driver's kind or a capability with `PlacementRequest.nodeLabels`, `'*'` matching
any value. The kind labels are meant for BMC steps that check a site agent's driver serves a concern before they run
(`bc.dev/driver-kind.ipmi.inspect`); no metal step reads them yet (see [placement](/docs/guides/capacity/placement#eligibility-filters)). The FakeDaemon
only advertises an `nvidia` driver of kind `config` on GPU hosts. PLAN.md's v0.5 acceptance includes "a third-party TS driver runs on a
node via bcx" and a Kata `runtime` plugin in Go (walkthrough 4).

#### Installing a plugin node [#installing-a-plugin-node]

Two things in the stock `bcd.service` stop the sandbox: its capability bounding set lacks `CAP_SETUID` and
`CAP_SETGID`, and `ProtectControlGroups=yes` keeps the cgroup tree read-only. The install script handles both
(4225772):

```bash
better-compute daemon install-script --pool pool_vps --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex> \
  --plugins --privileged-plugins kata > install-bcd.sh
```

1. `--plugins` writes a `[plugins]` section with `dir = "/usr/local/lib/bcd/plugins"` (root-owned, 0755, so each
   plugin's uid can reach its binary) and `cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins"`, unless the
   node's `config.toml` already has one. `--privileged-plugins NAMES` implies `--plugins` and adds `privileged`.
2. On every run, the script installs the drop-in `/etc/systemd/system/bcd.service.d/plugins.conf` when
   `config.toml` has a `[plugins]` section, and removes it when not: `CAP_SETUID CAP_SETGID CAP_KILL` added to the
   bounding set, `Delegate=yes` with `DelegateSubgroup=main`, `ProtectControlGroups=no`, and `RestrictNamespaces=no`
   only when `privileged` names a plugin.
3. Copy the plugin binaries into the directory and list them in its `pins` file, both root-owned, then restart
   `bcd.service`. The script installs only the plugins a release ships (`--install-plugins bcx-ipmi` with
   `--release-manifest`, adf2ae9); any other plugin is yours to place.

An upgrade keeps the `[plugins]` section, and with it the drop-in. See
[the plugin drop-in](/docs/daemon/install#the-plugin-drop-in) for the unit settings.

## Testing a driver [#testing-a-driver]

Unit-test the definition in process: call `implements[kind](config, ctx)` with a context you build and assert on the
operations. Then test the plugin as a real subprocess, as `tests/driver-sdk-*.test.ts` do with
`tests/fixtures/bcx-echo-driver.mjs`:

```ts title="acme-bmc.test.ts"
import { externalDriver, pinnedFile, sha256File } from '@better-compute/driver-sdk';

const command = [process.execPath, 'dist/acme-bmc-plugin.js'];
const host = externalDriver({ command, sha256: await sha256File(pinnedFile(command)) });
const [acme] = await host.start();

const power = await acme.implements.power({ address: 'https://10.0.0.10', credentials: 'secret://root/bmc' }, context);
expect(await power.getPower()).toBe('on');
expect(await host.health()).toEqual({ serving: true, message: '' });
await host.close();
```

`tests/fixtures/bcx-loader.mjs` lets a plain `node` subprocess load the workspace's TypeScript sources, so the fixture
runs without a build. The suites cover the handshake and pinning, Describe, Validate, Invoke with progress, errors,
deadlines, cancellation, secrets and streams, and supervision (crash restart, a pin changed under a running host,
graceful close):

```bash
pnpm vitest run tests/driver-sdk-handshake.test.ts tests/driver-sdk-host.test.ts tests/driver-sdk-supervision.test.ts
```

  - [BMC drivers and credentials](/docs/guides/bare-metal/bmc): The metal kinds, BMC commands, sealed credentials and rotation.

  - [Profiles and images](/docs/guides/bare-metal/profiles-and-images): Hardware profiles, deploy profiles and the image catalog.

  - [Steps and profiles](/docs/extending/steps-and-profiles): Workflow steps that drive agents.


# Overview (/docs/extending)

> Every seam of Better Compute is a typed contract you register through options. What each extension point is for, where it runs and how stable it is.



Better Compute has no plugin loader. Every place you can change its behaviour is a typed contract in
`@better-compute/core`, and you plug an implementation in by passing it to `betterCompute()`. Two rules from PLAN.md
§5.1 shape all of them:

* **Code registers kinds; data configures instances.** A driver, a provider or an attestor is code you register
  once. A BMC binding, a provider account or a join token is a row, validated by the manifest that code declares.
* **Registration is explicit.** Nothing is discovered from the file system or `node_modules`. Everything arrives in
  an options array, and the registry refuses what it cannot check: a duplicate `kind:id@version`, a method whose
  action is missing from the IAM catalog, or a migration named for another module are all `INVALID_CONFIG` when
  `betterCompute()` or `initialize()` runs.

## The extension points [#the-extension-points]

| Extension point                                                 | Declared with                    | Import from                  | Registered through                              | Runs in                                          |
| --------------------------------------------------------------- | -------------------------------- | ---------------------------- | ----------------------------------------------- | ------------------------------------------------ |
| [Module](/docs/extending/modules)                               | `defineModule`                   | `better-compute`             | `modules`                                       | control plane                                    |
| [Plugin](/docs/extending/modules#plugins) (a module with an id) | `definePlugin`                   | `better-compute`             | `plugins`                                       | control plane                                    |
| [API method](/docs/extending/methods)                           | `defineMethod`                   | `better-compute/core`        | a module's `api()`                              | control plane                                    |
| [Driver](/docs/extending/drivers)                               | `defineDriver`                   | `better-compute`             | `drivers`, or a module's `extensions`           | control plane, site agent, node, ramdisk         |
| Out-of-process driver (bcx)                                     | `serveDriver`, `externalDriver`  | `@better-compute/driver-sdk` | `drivers`                                       | its own process                                  |
| [Attestor](/docs/extending/attestors)                           | `defineAttestor`                 | `better-compute`             | `attestors`                                     | JoinService, in the control plane                |
| [Provider](/docs/extending/providers)                           | `defineProvider`                 | `better-compute/vps`         | `providers`                                     | control plane                                    |
| [Scheduler plugin](/docs/extending/scheduler-plugins)           | `defineSchedulerPlugin`          | `better-compute`             | `scheduler.plugins`, or a module's `extensions` | the placer, in api and worker processes          |
| [Event sink](/docs/extending/sinks)                             | `defineSink({ kind: 'events' })` | `better-compute`             | `sinks.events`, or a module's `extensions`      | the `sinks` worker role                          |
| [Audit sink](/docs/extending/sinks#audit-sinks)                 | `defineSink({ kind: 'audit' })`  | `better-compute`             | `sinks.audit`, or a module's `extensions`       | the `audit-sinks` reconciler, `controllers` role |
| [Step and profile](/docs/extending/steps-and-profiles)          | `defineStep`, `defineProfile`    | `better-compute/provision`   | a module's `extensions` and `profiles`          | control plane or one agent command               |

The root entry `better-compute` exports `defineModule`, `definePlugin`, `defineDriver`, `defineSchedulerPlugin`,
`defineAttestor` and `defineSink`. `defineMethod`, `defineStep`, `defineProfile` and `defineProvider` come from the
`better-compute/core`, `/provision` and `/vps` subpaths.

Other contracts in `core` are ports rather than plug-ins: `SqlExecutor` (the store), `IamPort`, `Clock`,
`Notifier`, `LeaseProvider`, `SecretStore`, `BlobStore`, `CertificateAuthority`, `ImageSource` and `NodeAdapter`. You
replace them through their own options (`database`, `iam`, `clock`, `notifier`, `secrets`, `pki`), not through
`extensions`; the [configuration reference](/docs/operations/configuration) lists them.

## Where each one plugs in [#where-each-one-plugs-in]

<Mermaid
  chart="flowchart LR
  subgraph cp[&#x22;Control plane (betterCompute)&#x22;]
    REG[&#x22;Registry&#x22;]
    ENV[&#x22;Operation envelope&#x22;]
    RT[&#x22;Worker runtime&#x22;]
    PL[&#x22;Placer&#x22;]
    JS[&#x22;JoinService&#x22;]
  end
  MOD[&#x22;Modules and plugins&#x22;] --> REG
  MET[&#x22;Methods&#x22;] --> ENV
  REC[&#x22;Reconcilers&#x22;] --> RT
  SCH[&#x22;Scheduler plugins&#x22;] --> PL
  ATT[&#x22;Attestors&#x22;] --> JS
  SNK[&#x22;Event and audit sinks&#x22;] --> RT
  DRV[&#x22;Drivers, steps&#x22;] --> WF[&#x22;Workflow engine (provision)&#x22;]
  PRV[&#x22;Providers&#x22;] --> VPS[&#x22;vps module&#x22;]
  WF -->|&#x22;commands&#x22;| AG[&#x22;bcd agents&#x22;]"
/>

A module is the unit everything else hangs off. It can carry API groups, reconcilers, migrations, command kinds,
NodeSpec sections, IAM catalog fragments, CLI commands, profiles and `extensions` (drivers, steps, scheduler
plugins, attestors, sinks). The top-level options (`drivers`, `providers`, `attestors`, `scheduler.plugins`,
`sinks`) register extensions that do not need a module of their own. Both paths end in the same registry and the
same `kind:id@version` uniqueness check, and every consumer reads both: scheduler plugins a module lists join
`scheduler.plugins` in the placer, and its event and audit sinks join `sinks.events` and `sinks.audit`.

```ts title="compute.ts"
import { betterCompute, defineSchedulerPlugin, defineSink } from 'better-compute';
import { acmeWidgets } from '@acme/bc-widgets'; // definePlugin({ id: 'acme-widgets', … })

export const compute = betterCompute({
  // …database, secret, baseURL, iam
  plugins: [acmeWidgets()],
  scheduler: { plugins: [defineSchedulerPlugin({ id: 'rack-spread', scorers: [/* … */] })] },
  sinks: {
    events: [defineSink({ kind: 'events', id: 'warehouse', async deliver(events) { /* … */ } })],
  },
});
```

## What is stable [#what-is-stable]

Better Compute is pre-1.0. The contracts in `@better-compute/core` froze at the end of phase P0 and change only by
additive edits: new optional members in minor versions, and a new interface selected by version for anything
breaking. PLAN.md §5.1 keeps every `define*` helper **experimental until v1.0**, so pin your versions.

| Extension point                                           | State today                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Modules, plugins, methods                                 | Built. Every built-in feature (`capacity`, `events`, `fleet`, `rollout`, `audit`, `iam`, `vps`, `metal`, `provision`) is a module on this contract.                                                                                                                                                                                                                                                                                              |
| Reconcilers, command kinds, NodeSpec sections, migrations | Built, with the v0.1 worker runtime.                                                                                                                                                                                                                                                                                                                                                                                                             |
| Scheduler plugins                                         | Built: filters, scorers, atom pickers and admission hooks run in both placers.                                                                                                                                                                                                                                                                                                                                                                   |
| Sinks                                                     | Built: event sinks on OutboxDispatch with durable cursors and dead letters; audit sinks on the built-in `audit-sinks` module. Usage sinks are refused with `INVALID_CONFIG` (usage goes to IAM billing through the bridge); see [sinks](/docs/extending/sinks#usage). Sinks and scheduler plugins a module lists in `extensions` run like those in the options.                                                                                  |
| Attestors                                                 | Built: `join-token`, `x509pop`, `provisioned`, `discovery`, `cloud-iid` and `tpm` are all `defineAttestor` values. bcd produces `join-token` joins, `provisioned` joins for an OS a deploy installed (`bcd join --node-id`), `discovery` joins from `bcd ramdisk` with its site agent's signed assertion, and `tpm` joins with the host's TPM (`bcd join --attestor tpm`, 8545688); its producers for `x509pop` and `cloud-iid` are not written. |
| Drivers, steps, profiles                                  | Built in TypeScript (`metal`, `provision`). bcd runs the metal command kinds that steps send (Redfish BMC operations, boot entries and `site.httpProxy` on `bcd site`; wipes, image writes, kexec and DCGM on `bcd ramdisk`), and `driver.invoke` in the bcx plugins it hosts; not released and tested by Go unit tests only.                                                                                                                    |
| Providers                                                 | Built: `hetzner` and `proxmox` provider packages, checked by `providerConformanceCases()`.                                                                                                                                                                                                                                                                                                                                                       |
| bcx out-of-process drivers                                | A v0.5 draft. `@better-compute/driver-sdk` works for TypeScript plugins, and bcd hosts sha256-pinned plugins (see [drivers](/docs/extending/drivers#driver-plugins-in-bcd)); the proto is unstable until v0.5 ships.                                                                                                                                                                                                                             |

> **The daemon half is mostly missing.** 
  Several extension points end in a command that a bcd agent runs: driver operations on a site
  agent, steps on a ramdisk, NodeSpec sections on a node. The Go daemon's node agent is built and tested end to end,
  and its site-agent and ramdisk modes are written with Go unit tests; none of it is released. It converges the
  `devices`, `access`, `instances` (on `libvirt-kvm` with `virsh` or `cloud-hypervisor` with its binary, both tested
  against fake tools only, or a fakehw build's file-backed runtime), `site` and `daemon` sections and runs its built-in
  and metal kinds, `driver.invoke` in the bcx plugins it hosts, and on a site agent `bmc.*` operations through a hosted
  BMC driver such as `bcx-ipmi`, but no `host` section. The TypeScript `FakeDaemon` in `@better-compute/testing`
  speaks the real protocol and runs the other paths in tests and in `better-compute dev --sim N`. See
  [what bcd does today](/docs/daemon#what-bcd-does-today).

## Rules every extension follows [#rules-every-extension-follows]

These hold for anything you write against the contracts, and the built-in modules follow them too:

1. **Modules never import each other's internals.** They meet through the ports on `ModuleContext`: the `vps`
   module books host capacity through `ctx.capacity.reserve`, not by importing `@better-compute/capacity`. The
   repository enforces the direction with `scripts/check-deps.mjs`.
2. **IAM is never awaited inside a compute transaction.** The envelope makes the one decision before your handler's
   transaction opens. Composite methods decide their sub-calls with `ctx.check` before they open a transaction.
3. **Write state, events and work in one transaction.** `ctx.events.publish`, `ctx.work.enqueue`,
   `ctx.commands.enqueue` and `ctx.authz.upsert` all take the caller's `tx`, so a rollback leaves nothing behind.
4. **Business time comes from the database.** SQL uses `compute_now()`, never `now()`; TypeScript uses `ctx.clock`.
   `scripts/check-sql-now.mjs` fails CI on a bare `now()` in `packages/*/src`.
5. **Everything is retried.** Reconcilers are level-triggered, sinks are at-least-once, commands carry idempotency
   keys. Write each piece so that running it twice changes nothing.

## Conformance [#conformance]

A contract comes with a conformance suite: plain `{ name, run }` cases you run against your implementation. The
runner lives on its own subpath of `@better-compute/core`, so the root entry carries no test code:

```ts title="acme-cloud.test.ts"
import { runConformance } from '@better-compute/core/conformance';
import { providerConformanceCases } from 'better-compute/vps';
import { acmeCloudSubject } from './acme-cloud-subject'; // your provider, against recorded responses

const report = await runConformance(providerConformanceCases(), () => acmeCloudSubject());
// report: { passed: string[], failed: { name, error }[] }; a factory gives each case a fresh subject
```

`storeConformanceCases()` from `@better-compute/postgres/conformance` does the same for a `SqlExecutor`. The acceptance tests in
`tests/extensions/` register third-party-style plugins with zero edits to `@better-compute/*`; see
[walkthroughs](/docs/extending/walkthroughs).

  - [Modules](/docs/extending/modules): defineModule and definePlugin: groups, migrations, reconcilers, NodeSpec sections and the module context.

  - [Methods](/docs/extending/methods): defineMethod: actions, resources, validation, admission, composites and errors.

  - [Drivers](/docs/extending/drivers): defineDriver, driver kinds, and out-of-process drivers over bcx.

  - [Attestors](/docs/extending/attestors): defineAttestor: how a joining agent proves what it is.

  - [Providers](/docs/extending/providers): defineProvider: cloud and hosting APIs behind VPS and metal pools.

  - [Scheduler plugins](/docs/extending/scheduler-plugins): Filters, scorers, atom pickers and admission hooks for the placer.

  - [Sinks](/docs/extending/sinks): Event and audit sinks, their cursors and dead letters, and where usage goes.

  - [Steps and profiles](/docs/extending/steps-and-profiles): defineStep and defineProfile for provisioning workflows.

  - [Walkthroughs](/docs/extending/walkthroughs): End-to-end extensions built without touching Better Compute.


# Methods (/docs/extending/methods)

> defineMethod declares an API method with its IAM action, the resource it is decided on, validation, and optional admission or composite behaviour. The envelope does the rest.



A method is a typed function behind one route. You declare what it needs, and the operation envelope does the
security work around it: authenticate the caller, check the tenant, make exactly one IAM decision, open the
transaction, run your handler and write the audit row. Your handler never calls IAM for its own admission and never
opens the outer transaction.

## The definition [#the-definition]

```ts title="@better-compute/core"
interface MethodDefinition<I, O> {
  action: `compute:${string}` | null;
  kind: 'read' | 'mutation' | 'public';
  resource(input: I): MethodResource | undefined;
  validate(input: unknown): I;
  handler(context: MethodContext, input: I): Promise<O>;
  admission?: MethodAdmission<I, O>;
  transaction?: 'auto' | 'manual';
  poll?: MethodPoll<I, O>;
}
```

<TypeTable
  type="{
  action: { type: 'compute:<area>:<verb> | null', description: 'The IAM action decided. Plugins: compute:<plugin-id>:<verb>. null only for public methods.', required: true },
  kind: { type: &#x22;'read' | 'mutation' | 'public'&#x22;, description: 'Decides how the call is decided, whether it writes, and whether it is audited.', required: true },
  resource: { type: '(input) => { type, id, action? }', description: 'The resource the action is decided on, without the IAM type prefix.', required: true },
  validate: { type: '(raw) => I', description: 'Parses the raw input; must reject unknown and invalid fields with INVALID_INPUT.', required: true },
  handler: { type: '(ctx, input) => Promise<O>', description: 'Runs after the decision, in a transaction unless transaction is manual.', required: true },
  admission: { type: 'MethodAdmission', description: 'Request-as-resource admission for creates and extends. Mutations only.' },
  transaction: { type: &#x22;'auto' | 'manual'&#x22;, description: 'manual: composite mutations that open their own transactions and use check/invoke.', default: &#x22;'auto'&#x22; },
  poll: { type: '{ done(output); refresh(ctx, input, output) }', description: 'Async-with-wait: how the envelope waits when the input asks wait: { timeoutMs }. Mutations only; see below.' },
}"
/>

`defineMethod` (from `better-compute/core`) freezes the definition and keeps its input and output types, which is
what types `compute.api` and the client. The registry checks the shape when the module registers: `resource`,
`validate` and `handler` must be functions, `kind` one of the three, a public method's `action` `null`, every other
action must match `^compute:[a-z][a-z0-9-]{0,63}:[a-z][a-z0-9-]{0,63}$`, and a plugin's must start with
`compute:<id>:`. Anything else is `INVALID_CONFIG`.

## How a call runs [#how-a-call-runs]

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant C as Caller
  participant E as Envelope
  participant I as IAM
  participant D as Postgres
  C->>E: route + credential + JSON input
  E->>E: validate(input)
  E->>I: authenticate (cached 30 s)
  E->>E: tenant pre-check
  E->>I: one decision on (action, resource(input))
  E->>D: BEGIN
  E->>D: handler(ctx, input)
  E->>D: audit_events row (mutations)
  E->>D: COMMIT
  E-->>C: { data }"
/>

1. **Public methods** (`kind: 'public'`, `action: null`) are validated and run; the credential is passed along
   only when one was presented.
2. **Everything else** needs a credential (`UNAUTHENTICATED` without one, before any IAM call). The input is
   validated, then the caller is authenticated through the IamPort, cached for 30
   seconds by the credential's hash.
3. **Tenant pre-check.** A caller whose tenant is neither the input's `tenantId` nor the root tenant gets
   `ACCESS_DENIED` before compute's database is read or IAM is asked anything.
4. **One decision.** Mutations call `iam.require`, never cached. Reads call `iam.authorize`, whose allows are cached
   for 10 seconds per session. A refusal is `ACCESS_DENIED` and no transaction is opened.
5. **The handler** runs in one transaction: read-only for reads, so a read that writes fails. An allowed mutation
   also gets an `audit_events` row with its admission context, in the same
   transaction, so a rolled-back mutation leaves no audit row either.

A thrown error rolls the transaction back and becomes the error response.

## Actions and resources [#actions-and-resources]

`resource(input)` names what the action is decided on, without the IAM type prefix: `{ type: 'tenant', id }`,
`{ type: 'pool', id: input.poolId }`. The envelope adds the prefix (`compute-` by default) and decides in the tenant
the input names. So a method's input needs a `tenantId`, unless it is decided on the fleet (`{ type: 'fleet', id:
'global' }`), which is always decided in the root tenant. A method with neither is `INVALID_CONFIG` at call time.

Keep `resource` a pure projection of the validated input. For methods that have a row in the method table,
`initialize()` probes `resource` with a placeholder input and compares the kind it decides on with the row.

| Method kind   | Decides on         | Examples                                                               |
| ------------- | ------------------ | ---------------------------------------------------------------------- |
| Tenant-scoped | `tenant`           | `webhooks.*` (`compute:webhooks:manage`), `quotas.get`, `pools.create` |
| Object-scoped | the object         | `reservations.get` on `reservation`, `pools.get` on `pool`             |
| Fleet-wide    | `fleet` / `global` | `fleet.apply`, `rollouts.create`, `quotas.set`                         |

A plugin can decide on its own resource types, declared in its `catalog.resourceTypes`, as long as IAM can resolve
them: write their rows into the authz projection with `ctx.authz.upsert(tx, { type, id, tenantId, attributes,
ownerId })` when you create them. See [actions and resources](/docs/guides/iam/actions-and-resources).

### Per-input actions [#per-input-actions]

Some rows of the method table decide a different action for some inputs: `tunnels.open` decides
`compute:nodes:console` when it targets a node. `resource()` returns that action in `resource.action`, and the
envelope decides it, but only if the row lists that `(action, type)` pair as an alternative. A method without a
row, which includes every plugin method, cannot override its action; `resource.action` other than its own action
is `INVALID_CONFIG`.

## Validation [#validation]

`validate` receives the raw JSON object (or `{}` for an empty body) and returns the typed input. It must reject
unknown fields, not strip them. The `manifest` builder does both:

```ts
import { field, manifest } from 'better-compute';

const rotateInput = manifest({
  tenantId: field.string({ pattern: '^[A-Za-z0-9._:-]{1,128}$' }),
  webhookId: field.string({ pattern: '^whk_[0-9a-hjkmnp-tv-z]{26,52}$' }),
  keepPreviousSeconds: field.int({ min: 0, max: 86_400 }).optional(),
});
// validate: (raw) => rotateInput.parse(raw)
// → INVALID_INPUT "Unknown field x", "keepPreviousSeconds: …", "Required", …
```

`parse` throws `ComputeError('INVALID_INPUT')` naming the first problem. Validation runs before authentication,
so a malformed request never costs an IAM call. Another validator works too, but wrap its failures in
`ComputeError('INVALID_INPUT', …)`: any other error thrown from `validate` reaches the caller as `INTERNAL_ERROR`.

A `validate` function may carry its input's JSON Schema as a `jsonSchema` property; the generated API reference and
tool schemas read it. For a manifest, attach `m.jsonSchema()` to `m.parse`. For a hand-written validator,
`describedBy(schema, validate)` from `@better-compute/core` checks the schema's structure first (field types,
required and unknown fields, enums), reporting every structural issue at once as `INVALID_INPUT` with
`details.issues` (`{ message, path }[]`), then runs your validator for the domain rules; `strictObject(properties,
required)` builds an object schema that refuses unknown fields:

```ts
import { describedBy, strictObject } from '@better-compute/core';

const validate = describedBy(
  strictObject({ tenantId: { type: 'string' }, count: { type: 'integer' } }, ['tenantId']),
  (raw) => parseCount(raw), // ranges, patterns, "exactly one of"
);
```

Every built-in method carries a schema. `tests/input-schema-agreement.test.ts` composes every module, fails on any
registered method without `validate.jsonSchema`, and checks that each schema agrees with its validator on required,
unknown, enum and string fields.

## The method context [#the-method-context]

The handler's `MethodContext` is the [module context](/docs/extending/modules#the-module-context) plus the call:

| Member            | Meaning                                                                                                                                                                                       |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sql`             | The envelope's transaction. `sql.transaction(fn)` opens a savepoint; `migrate` and `close` are refused                                                                                        |
| `caller`          | `{ identityId, identityKind, tenantId, sessionId, sessionKind, mfa, agentId?, delegationId? }`; absent for public methods                                                                     |
| `credential`      | The credential as presented, for calls you make on the caller's behalf                                                                                                                        |
| `admission`       | The recorded decision of an allowed mutation: tenant, identity, session, action, resource, `decidedAt`                                                                                        |
| `requestId`       | The caller's `X-Request-Id` when it is short and plain, else a generated `req_…`                                                                                                              |
| `traceparent`     | The request's W3C `traceparent` header when it is valid (version `00`, non-zero ids); `ctx.commands.enqueue` then defaults each command's `traceparent` to it unless the command sets its own |
| `signal`          | Aborts when the transport does                                                                                                                                                                |
| `check`, `invoke` | Composite methods only; see below                                                                                                                                                             |

## Admission: the request as a resource [#admission-the-request-as-a-resource]

A create that should be decided on what it asks for, not only on the tenant, uses `admission`. The request is first
written as a draft with its attributes in the authz projection, IAM decides on the draft,
and only an allowed draft becomes real. This is how `reservations.create` lets a policy cap GPU count and duration
in one decision; see [reservations](/docs/guides/capacity/reservations#the-admission-sequence).

```ts title="@better-compute/core"
interface MethodAdmission<I, O> {
  prepare(context: MethodContext, input: I, tx: SqlTx): Promise<
    { status: 'decide'; resource: MethodResource } | { status: 'replay'; result: O }
  >;
  denied?(context: MethodContext, input: I, tx: SqlTx, error: unknown): Promise<void>;
  describe?(input: I): string;
  check?(context: MethodContext, input: I, resource: MethodResource): Promise<void>;
  abandoned?(context: MethodContext, input: I, tx: SqlTx, error: unknown): Promise<void>;
}
```

<Mermaid
  chart="flowchart TD
  P[&#x22;tx 1: prepare()&#x22;] -->|replay| R[&#x22;return the stored result, no decision&#x22;]
  P -->|decide| Q[&#x22;iam.require on the returned resource&#x22;]
  Q -->|allowed| C[&#x22;check(), no transaction open&#x22;]
  C -->|passes| H[&#x22;tx 2: handler() + audit row&#x22;]
  H -->|throws| A[&#x22;own tx: abandoned(error)&#x22;]
  A --> Z[&#x22;throw the handler's error&#x22;]
  C -->|throws| Y[&#x22;throw its error; the draft stays&#x22;]
  Q -->|refused| D[&#x22;own tx: denied(error)&#x22;]
  D --> X[&#x22;throw ACCESS_DENIED or CONFIRMATION_REQUIRED&#x22;]"
/>

* **`prepare`** runs in the first transaction and never calls IAM. It looks up the idempotency key (scoped to the
  tenant and the caller's identity), writes or reuses the draft and its projection row, and says what to decide on.
  A finished request by the same identity is a `replay`: its stored result is returned without a second decision. A
  different request under the same key must throw `CONFLICT`. `draftId(prefix, tenantId, identityId, key)` from
  `better-compute/core` derives a deterministic id for the draft.
* **The decision** is one `require` on the resource `prepare` returned. `resource(input)` may return `undefined`
  for admission methods, which learn their resource only in `prepare`.
* **`check`** (optional) runs after the decision allowed the request and before the handler, with **no transaction
  open**, so it may call IAM: for example `ctx.iam.billingCheck` for a budget. Its `ctx.sql` runs each statement on
  its own. A throw refuses the request with that error and does **not** call `denied`: the draft stays as it is, so
  a retry with the same key is decided again later (say, once a budget's window resets). Composite `ctx.check` runs
  it too.
* **`handler`** runs in a second transaction with `ctx.admission` set, and the audit row is written there.
* **`abandoned`** (optional) runs in a third transaction of its own when an allowed request's handler threw, for
  example because the resource changed between the transactions (d14564e). The handler's rollback also undid any
  cleanup it did, so this is where to undo, best effort, what `prepare` staged in the first transaction and would
  otherwise outlive the request, such as pending authz attributes an `extend` set, which would be merged into every
  later decision on the resource. Keep a retry with the same key possible. If it throws, the failure is logged and the
  caller still gets the handler's error. `reservations.extend` and `sandboxes.extend` pass their `denied` clearing
  here too (a2288ec).
* **`denied`** runs in a transaction of its own after a refusal, to mark the draft denied or clear pending changes.
  If it throws, the failure is logged and the refusal is still what the caller gets.
* **Delegated sessions.** When the caller is an agent in a delegated session, the envelope asks IAM to open a
  confirmation, with `describe(input)` as the reason shown to the person. If the delegation holds the action for
  confirmation, the error becomes `CONFIRMATION_REQUIRED` with `details: { confirmationId, expiresAt?, action,
  resource }`, and `denied` receives it so the draft can wait as `awaiting-confirmation`. See
  [delegated confirmation](/docs/guides/iam/delegated-confirmation).

A replay-only admission needs no draft at all. `projects.create` derives its id from the key and returns
`{ status: 'replay', result }` when the project exists, else `{ status: 'decide', resource: { type: 'tenant', id } }`.
`pools.create` does the same when an `idempotencyKey` is given.

Only mutations take an `admission`, and a method cannot combine it with `transaction: 'manual'`.

## Async-with-wait [#async-with-wait]

A mutation whose work finishes later (a queued placement) can let callers wait for it in the same request (PLAN.md
section 9.1). Declare `poll`, and accept `wait: { timeoutMs }` in `validate`:

```ts title="@better-compute/core"
interface MethodPoll<I, O> {
  done(output: O): boolean;
  refresh(context: MethodContext, input: I, output: O): Promise<O>;
}
```

After the mutation commits (plain or admitted, and on a replay too), the envelope checks `done(output)`. When it is
false and the input asks for a positive `wait.timeoutMs`, it calls `refresh` in read-only transactions, backing off
from 50 ms to 1 s, until `done`, the timeout (at most 30 seconds, wall clock) or the request's abort, and answers the
last output. A timeout is not an error: the output itself must say the work is still pending, as a reservation's
`operation.state: 'running'` does. `reservations.create` and `instances.create` are the built-in users; see
[waiting for placement](/docs/guides/capacity/reservations#waiting-for-placement) and
[waiting for the instance](/docs/guides/vps/instances#waiting-for-the-instance). Keep `wait` out of the request hash
your idempotency check compares, as both do, so a retry with a different wait replays. The registry refuses (`INVALID_CONFIG`)
a `poll` on a method that is not a mutation, or one without both functions.

## Composite methods [#composite-methods]

A mutation declared `transaction: 'manual'` composes other methods, each decided for the same caller. `fleet.apply`
is the one built in: it applies a whole fleet document through `pools.create`, `quotas.set` and the rest, all or
nothing.

```ts title="packages/fleet/src/module.ts (trimmed)"
apply: defineMethod<ApplyInput, ApplyResult>({
  action: 'compute:fleet:admin',
  kind: 'mutation',
  resource: () => ({ type: 'fleet', id: 'global' }),
  transaction: 'manual',
  validate: parseApply,
  async handler(ctx, input) {
    const first = await ctx.sql.transaction((tx) => plan(ctx, tx, input));
    // Phase 1: one decision per call, as the caller, with no transaction open.
    for (const call of first.calls) await ctx.check!(call.path, call.input);
    // Phase 2: re-plan and run everything in one transaction.
    await ctx.sql.transaction(async (tx) => {
      const second = await plan(ctx, tx, input);
      if (second.planHash !== first.planHash) throw new ComputeError('CONFIG_DRIFT', 'plan again');
      for (const call of second.calls) await ctx.invoke!(tx, call.path, call.input);
    });
    return { plan: first, applied: first.calls.length };
  },
}),
```

* The composite itself is decided once, like any mutation. Its handler then runs **with no transaction open**;
  `ctx.sql` runs each statement on its own and each `ctx.sql.transaction` is a real transaction.
* **`ctx.check(path, input)`** makes the target's tenant pre-check, validation and one decision exactly as the
  envelope would, and records a grant keyed by the path and the canonical JSON of the input. It refuses to run inside
  a transaction (`INVALID_STATE`).
* **`ctx.invoke(tx, path, input)`** consumes one grant (else `ACCESS_DENIED`), runs the target's handler in a
  savepoint of `tx` without another decision, and writes the sub-call's own audit row. A failed sub-call leaves
  nothing behind. Grants live only for the current call. When the target has an `admission.check`, `invoke` runs the
  very input object that check ran on, not a fresh validation of the raw input (7911c5f), so what the check recorded
  in it reaches the handler: `sandboxTemplates.create` and `update` record their per-volume decisions there
  (`decidedVolumes`), which is how `fleet.apply` can create or change a template that mounts volumes.
* `check` refuses (`UNSUPPORTED`) other composites, admissions with a `denied` hook (reservations, instances) and
  list methods that decide their items one by one. Replay-only admissions work: `prepare` runs in a short
  transaction at `check` and again inside `invoke`.
* The composite's own audit row is written in its own transaction after the handler succeeds.

Paths are the routes below the base path: `pools/create`, or `plugins/acme/widgets/poke`.

## Serving plugin methods [#serving-plugin-methods]

A plugin's methods are served under its id. For plugin `acme`, group `widgets`, method `poke`:

  **HTTP:**

    ```bash
    curl -X POST https://compute.example.com/api/compute/plugins/acme/widgets/poke \
      -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" \
      -H "Content-Type: application/json" \
      -H "X-Better-Compute: 1" \
      -d '{"tenantId":"acme","widgetId":"w-1"}'
    # → { "data": { "poked": "w-1" } }
    ```
  
  **Server:**

    ```ts
    const result = await compute.api.plugins.acme.widgets.poke(
      { headers: request.headers },
      { tenantId: 'acme', widgetId: 'w-1' },
    );
    ```
  
  **Client:**

    ```ts
    import { createComputeClient } from 'better-compute/client';
    import type { compute } from './compute';

    const client = createComputeClient<typeof compute>({ baseURL, token });
    await client.plugins.acme.widgets.poke({ tenantId: 'acme', widgetId: 'w-1' });
    ```
  
Every route takes `POST` with `Content-Type: application/json`, the `X-Better-Compute: 1` header (without either
the answer is `CSRF_REJECTED`) and a JSON object of at most 64 KiB. Success is `{ data }`; failure is `{ error: { code, message, retryAfterMs?, details? } }`
with the code's HTTP status. `compute.api` runs the same envelope in process.

## Errors [#errors]

Throw `ComputeError` from `better-compute/core`. Its code and message are shown to the caller; `details` carries
structured context.

```ts
throw new ComputeError('CONFLICT', 'Widget changed meanwhile', undefined, { widgetId });
throw new ComputeError('RATE_LIMITED', 'Slow down', undefined, { retryAfterMs: 20_000 });
```

A numeric `details.retryAfterMs` moves to the body's top-level `retryAfterMs`, and the HTTP answer gets a
`Retry-After` header in seconds. Anything that is not a `ComputeError`, and any `INTERNAL_ERROR`, reaches the
caller as `INTERNAL_ERROR` with the message `Internal server error`, so a leaked connection string stays in your
logs. `isComputeError(error, code?)` recognizes errors from a second copy of the package too.

| Code                                     | Status | Throw it when                                                                                            |
| ---------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------- |
| `INVALID_INPUT`                          | 400    | Validation fails (the manifest does this for you)                                                        |
| `NOT_FOUND`                              | 404    | The object does not exist in the input's tenant. Use it for other tenants' objects too, so nothing leaks |
| `INVALID_STATE`                          | 409    | The object exists but cannot do this now (`cancel` on an active reservation)                             |
| `CONFLICT`                               | 409    | An idempotency key reused for a different request, or a lost race worth retrying                         |
| `CAPACITY_UNAVAILABLE`, `QUOTA_EXCEEDED` | 409    | Nothing fits, or a quota is spent                                                                        |
| `CONFIG_DRIFT`                           | 409    | Live state moved since a plan was made                                                                   |
| `UNSUPPORTED`                            | 422    | A valid request this deployment or object does not support                                               |
| `RATE_LIMITED`                           | 429    | With `details.retryAfterMs`                                                                              |
| `DRIVER_ERROR`, `PROVIDER_ERROR`         | 502    | A driver or provider call failed                                                                         |
| `UNAVAILABLE`                            | 503    | A dependency is unreachable for now and nothing changed; with `retryAfterMs`                             |
| `NOT_IMPLEMENTED`                        | 501    | A stub: `notImplemented(feature, phase)` names the phase that delivers it                                |
| `INVALID_CONFIG`                         | 500    | The deployment is misconfigured, not the request                                                         |

Leave `UNAUTHENTICATED`, `ACCESS_DENIED`, `CONFIRMATION_REQUIRED`, `CSRF_REJECTED` and the transport codes to the
envelope. The [error reference](/docs/reference/errors) lists every code.

## Adding a method to Better Compute itself [#adding-a-method-to-better-compute-itself]

This applies when you change the repository's own groups, not your plugin's. Better Compute's API surface is frozen
toward v1.0: `pnpm lint:repo` runs `scripts/api-surface.mjs`, which compares every method of core's `apiSurface` and
the registry (route, kind, IAM action, resource, alternatives, list flag, status, phase and input JSON Schema) and every
error code's HTTP status with the recorded `generated/api-surface.json`. A new method, a new optional field or a
wider schema is an addition and passes; removing a served method, changing its route, action or resource, or
narrowing its input (a new required field, a tighter bound, a lost enum value) fails. A method that is still planned
may change freely.

After adding or widening a method, regenerate the docs (`node apps/docs/scripts/generate.mjs`, which the check reads)
and record the addition with `pnpm api:surface`, so the new shape is frozen too. Until it is recorded the check only
reports it. See [the API surface freeze](/docs/operations/testing#the-api-surface-freeze). A new (action, resource)
pair is also covered by the authz matrix, from the §9.2 table in core, in both IAM modes.


# Modules (/docs/extending/modules)

> A ComputeModule bundles API groups, migrations, reconcilers, command kinds and NodeSpec sections. How to write one, what its context offers, and how the host runs it.



A module is the unit of extension. Everything Better Compute does beyond its envelope is a module: `capacity` and
`inventory` are built in, and `events`, `fleet`, `rollout`, `audit`, `iam`, `vps`, `metal` and `provision` are
modules you pass in. Yours use the same contract, `ComputeModule` from `@better-compute/core`, and reach the rest of
the system only through the ports on their context.

## What a module declares [#what-a-module-declares]

```ts title="@better-compute/core"
interface ComputeModule<Name extends string = string, A extends ApiGroups = {}> {
  name: Name;
  requires?: string[];
  migrations?: Migration[];
  extensions?: Extension[];
  api?(context: ModuleContext): A;
  reconcilers?: Reconciler[];
  commandKinds?: CommandKindSpec[];
  nodeSpec?(context: ModuleContext, node: NodeRecord): Promise<NodeSpecPart>;
  statusHandlers?: readonly ModuleStatusHandler[];
  catalog?: { actions?: string[]; resourceTypes?: Record<string, ComputeResourceType> };
  commands?: CliCommandSpec[];
  profiles?: ProfileDocument[];
  selfCheck?(context: ModuleContext): Promise<Finding[]>;
  start?(context: ModuleContext): Promise<void>;
  stop?(): Promise<void>;
  tenantDeleted?(context: ModuleContext, tenantId: string): Promise<'DONE' | 'SYNC_AGAIN'>;
}
```

<TypeTable
  type="{
  name: { type: 'string', description: 'Matches ^[a-z][a-z0-9-]{0,63}$. core, server and plugins are reserved; a name registered twice is INVALID_CONFIG.', required: true },
  requires: { type: 'string[]', description: 'Names of modules that must also be registered, in any position.' },
  migrations: { type: 'Migration[]', description: 'Named <name>/<nnnn>_<slug>, numbers strictly increasing. See Migrations.' },
  extensions: { type: 'Extension[]', description: 'Drivers, steps, scheduler plugins, attestors and sinks built with a define* helper, unique by kind:id@version.' },
  api: { type: '(ctx) => ApiGroups', description: 'Returns { group: { method: defineMethod(...) } }. Called once, while the registry is built.' },
  reconcilers: { type: 'Reconciler[]', description: 'Level-triggered workers on the work queue. See Reconcilers.' },
  commandKinds: { type: 'CommandKindSpec[]', description: 'Agent command kinds this module sends. Each kind is registered once across all modules.' },
  nodeSpec: { type: '(ctx, node) => NodeSpecPart', description: &#x22;The module's sections of a node's desired state.&#x22; },
  statusHandlers: { type: 'ModuleStatusHandler[]', description: &#x22;(8de6e2f) Consumers of a node's coalesced StatusReport: (tx, { nodeId, tenantId, work, events? }, report) => Promise<unknown>, run by the gateway in the status transaction under its own savepoint. better-compute serve and the dev stack run every registered module's after the gateway's XID health handler and before the configuration's reports.status; a host that calls createGateway itself adds them. metal declares its PXE discovery handler here.&#x22; },
  catalog: { type: '{ actions?, resourceTypes? }', description: 'Merged into the IAM catalog. Every action matches compute:<area>:<verb>.' },
  commands: { type: 'CliCommandSpec[]', description: 'CLI commands the module adds: { name, summary, group? }.' },
  profiles: { type: 'ProfileDocument[]', description: 'Provisioning profiles the module ships; see steps and profiles.' },
  selfCheck: { type: '(ctx) => Finding[]', description: 'Read-only checks reported by selfCheck() and better-compute doctor.' },
  start: { type: '(ctx) => Promise<void>', description: 'Awaited by initialize() after migrations, in registration order.' },
  stop: { type: '() => Promise<void>', description: 'Awaited by stop(), in reverse registration order.' },
  tenantDeleted: { type: &#x22;(ctx, tenantId) => 'DONE' | 'SYNC_AGAIN'&#x22;, description: 'Releases what the module holds for a deleted IAM tenant. Called by the built-in tenancy reconciler until it answers DONE. Must be idempotent.' },
}"
/>

`defineModule(module)` returns its argument unchanged; it only keeps the name and API types literal, so the typed
client and `compute.api` know your groups.

## A complete example [#a-complete-example]

This plugin is the one the server's own tests register (`tests/support/server.ts`), trimmed and given a reconciler
and a self-check:

```ts title="acme-widgets.ts"
import { definePlugin, field, manifest } from 'better-compute';
import { ComputeError, defineMethod, type Finding } from 'better-compute/core';

const widgetInput = manifest({
  tenantId: field.string({ pattern: '^[A-Za-z0-9._:-]{1,128}$' }),
  widgetId: field.string({ pattern: '^[a-z0-9-]{1,64}$' }),
});
type WidgetInput = { tenantId: string; widgetId: string };

export const acmeWidgets = () =>
  definePlugin({
    id: 'acme',
    catalog: { actions: ['compute:acme:poke', 'compute:acme:read'] },
    migrations: [
      {
        name: 'acme/0001_widgets',
        sql: `CREATE TABLE plg_acme_widgets (
                id text PRIMARY KEY, tenant_id text NOT NULL,
                created_at timestamptz NOT NULL DEFAULT compute_now())`,
      },
    ],
    api: () => ({
      widgets: {
        poke: defineMethod<WidgetInput, { poked: string }>({
          action: 'compute:acme:poke',
          kind: 'mutation',
          resource: (input) => ({ type: 'tenant', id: input.tenantId }),
          validate: (input) => widgetInput.parse(input),
          async handler(ctx, input) {
            await ctx.sql.query('INSERT INTO plg_acme_widgets (id, tenant_id) VALUES ($1, $2)', [
              input.widgetId,
              input.tenantId,
            ]);
            await ctx.events.publish(ctx.sql, {
              type: 'acme.widget-poked',
              tenantId: input.tenantId,
              subject: `widget/${input.widgetId}`,
              data: { widgetId: input.widgetId },
            });
            await ctx.work.enqueue(ctx.sql, 'acme.widgets', input.tenantId);
            return { poked: input.widgetId };
          },
        }),
      },
    }),
    reconcilers: [
      {
        name: 'acme.widgets',
        shardBy: 'none',
        periodic: [{ key: 'sweep', every: '10m' }],
        async run(ctx, key) {
          // Level-triggered: look at the state for `key` and converge it. Safe to run twice.
          return 'DONE';
        },
      },
    ],
    async selfCheck(ctx): Promise<Finding[]> {
      const { rows } = await ctx.sql.query<{ n: string }>('SELECT count(*)::text AS n FROM plg_acme_widgets');
      return Number(rows[0]?.n) > 1_000_000
        ? [{ check: 'acme.widgets', severity: 'warning', message: 'Over a million widgets', fix: 'Prune old widgets' }]
        : [];
    },
  });
```

The method's handler runs inside the envelope's transaction, so the row, the event and the work item commit
together or not at all. Its HTTP route is `POST /api/compute/plugins/acme/widgets/poke`; see
[methods](/docs/extending/methods).

## Registering modules [#registering-modules]

```ts title="compute.ts"
import { betterCompute } from 'better-compute';
import { eventsModule } from 'better-compute/events';
import { createComputeIamBridge } from 'better-compute/iam';
import { acmeWidgets } from './acme-widgets';

const plugins = [acmeWidgets()];
const bridge = createComputeIamBridge({ rootTenantId, plugins }); // merges the plugin's catalog

export const compute = betterCompute({
  // …database, secret, baseURL
  iam: bridge.embedded(iam, { rootTenantId }),
  modules: [...bridge.modules, eventsModule({ secret })],
  plugins,
});
await compute.initialize();
```

`betterCompute()` registers, in this order: the server's own groups, the built-in `node-controllers` and
`site-boot` modules, `inventory`, `capacity`, the `tenancy` module (suspended and deleted tenants), the `quorum`
module (the v1.0 destructive-batch quorum's batches and `commandBatches`, always hosted), a `pki` entry carrying the sealed-secrets migration when the default
secret store is in use, the `tunnels` module when `tunnels` is configured, the `audit-sinks` module when any audit
sink is listed (see [sinks](/docs/extending/sinks#audit-sinks)), then your `modules`, then your `plugins`. The order
is the order of migrations, of `start()` calls and of NodeSpec merging.

The `extensions` a module or plugin lists are read like the top-level options: drivers and attestors are found in
either place, scheduler plugins join `scheduler.plugins` in the placer, and event and audit sinks join `sinks.events`
and `sinks.audit`. A `pool-policy` extension (`PoolPolicyValidator`, 37dcd02) checks the parts of a pool policy its
module reads: `validate(policy, { class, sharing })` runs in `pools.create` and in `pools.update` when the policy
changes, before the write, and refuses by throwing a `ComputeError`; any other error becomes `INVALID_INPUT`
`policy: ...`. Provision registers one for its host units. A usage sink, in `sinks.usage` or in `extensions`, is `INVALID_CONFIG`: usage goes to IAM billing
through the IAM bridge.

Registration checks everything it can without the database and throws `INVALID_CONFIG`:

* module names are unique and well formed, and every `requires` entry is registered;
* every migration of a module is named `<module>/…`, and the whole list passes `assertMigrations`;
* extensions are built with a `define*` helper and unique by `kind:id@version`;
* command kinds are unique, including against the server's built-in kinds;
* routes are unique, group and method names match `^[A-Za-z][A-Za-z0-9_-]{0,63}$` and are not `then`, `catch`,
  `finally`, `toJSON`, `constructor`, `prototype`, `__proto__` or `plugins`;
* catalog actions match `compute:<area>:<verb>`; a plugin's actions must be `compute:<plugin-id>:<verb>`.

`initialize()` then checks every method against the merged catalog and the method table of PLAN.md §9.2, migrates
the store (its base schema first, then module and plugin migrations in order), anchors the clock, and awaits each
module's `start()` in registration order. If a `start()` throws, the modules already started are stopped in
reverse and the error is rethrown. `stop()` stops the worker loops first, then the modules in reverse order.

> **Pass the catalog to IAM too.** 
  A module's or plugin's `catalog` must reach every Better IAM host, not only the compute server. Pass the same
  modules and plugins to `createComputeIamBridge({ modules, plugins })`, which merges their fragments into the catalog
  IAM decides with. See [actions and resources](/docs/guides/iam/actions-and-resources).

## Plugins [#plugins]

A plugin is a module with an `id`, built with `definePlugin`. Its name defaults to the id. Three things differ from
a module:

|            | Module (`modules`)                          | Plugin (`plugins`)                              |
| ---------- | ------------------------------------------- | ----------------------------------------------- |
| Routes     | `POST {basePath}/{group}/{method}`          | `POST {basePath}/plugins/{id}/{group}/{method}` |
| Server API | `compute.api.<group>.<method>`              | `compute.api.plugins.<id>.<group>.<method>`     |
| Actions    | Must match its row in the §9.2 method table | Must be `compute:<id>:<verb>`                   |

A non-plugin method with no §9.2 row is an `undeclared` problem: `initialize()` still succeeds, `better-compute
doctor` reports a `catalog.undeclared` warning, and `scripts/authz-matrix.mjs` fails. The method table is
normative for Better Compute's own surface, so **third-party code ships as plugins**. By convention a plugin's
tables are named `plg_<id>_*`; nothing enforces it, but it keeps plugin tables apart from compute's own.

## Migrations [#migrations]

```ts title="@better-compute/core"
interface Migration {
  name: `${string}/${string}`; // <owner>/<nnnn>_<slug>
  sql: string | ((tx: SqlTx) => Promise<void>);
}
```

A name matches `^([a-z][a-z0-9-]{0,63})/(\d{4})_[a-z0-9][a-z0-9_-]{0,95}$`, and the owner must be the module's
own name: `rollout/0001_rollouts`, `events/0001_webhooks`, `acme/0001_widgets`. Within an owner the numbers must
strictly increase. Names are checked before anything runs.

The store applies them after its base schema (owner `core`). Each migration runs in its own transaction and is
recorded in the append-only `compute_migrations` ledger; concurrent migrators serialize on an advisory lock and skip
what another applied. So a failed migration leaves nothing behind, and an applied one is never run again. Never
edit a migration that has shipped; add the next number instead. Use `compute_now()` for defaults, never `now()`.

## Reconcilers [#reconcilers]

```ts title="@better-compute/core"
interface Reconciler {
  name: string;
  shardBy: 'node' | 'pool' | 'none';
  run(context: ReconcileContext, key: string): Promise<'DONE' | 'SYNC_AGAIN'>;
  periodic?: readonly { key: string; every: Duration }[];
  role?: 'controllers' | 'scheduler'; // default 'controllers'
}
```

A reconciler's name is also its work kind: `ctx.work.enqueue(tx, 'acme.widgets', key)` wakes it for `key`. Names
match `^[a-z][a-z0-9._-]{0,63}$`, are unique across modules, and may not be `command-sweeper`, `outbox-dispatch` or
`event-subscribers`, which the runtime reserves. Keys are non-empty strings of at most 512 characters.

Runs are level-triggered. The work queue coalesces: enqueuing a key that is already queued bumps its generation
instead of adding a row, and a run that finishes deletes the key only at the generation it claimed. A key dirtied
again while its reconciler runs survives, so no wake-up is lost.

| Outcome        | What happens to the key                                                               |
| -------------- | ------------------------------------------------------------------------------------- |
| `'DONE'`       | Removed; for a periodic key, due again `every` after this run                         |
| `'SYNC_AGAIN'` | Due again in 1 second                                                                 |
| a throw        | Due again after a back-off of 1 s, doubling to at most 5 minutes; logged as a warning |

`shardBy` decides which worker runs a key. Every key lands on shard `shardOf(key)`, an FNV-1a hash modulo 256.

* **`node`*&#x2A; and &#x2A;*`pool`**: keys are node or pool ids. Each worker runs the keys of the shards it holds `shard:<n>`
  leases for. Every 5 minutes plus up to 5 minutes of jitter (`worker.resyncMs`), it re-enqueues every node or pool
  of its shards, so a missed wake-up heals.
* **`none`**: one worker at a time runs the reconciler, under a `singleton:<name>` lease.

`role` picks the worker role that runs it (v0.4). The default, `controllers`, suits almost everything. Placers
declare `scheduler`, as the pool-lease placer `capacity.placer` does: they then run only in processes started with
the `scheduler` role (`serve --roles scheduler`), under that role's own rendezvous membership (worker rows
`scheduler/<id>`, leases `scheduler/shard:<n>`), so placement scales apart from the controllers. A role joins a
shard membership only when it runs a sharded reconciler of its own, so a scheduler process with no placer registered
holds no shard leases. `serve` with the default `--roles all` runs both roles.

`periodic` keys are seeded once and then come back `every` after each run (at least `1s`), besides any edge
enqueue. Every transaction a reconciler opens through `ctx.sql.transaction` begins by fencing its lease, so a worker
that lost the lease cannot commit. `ctx.shard` and `ctx.lease` say what the run holds, and `ctx.signal` aborts on
`stop()`.

Reconcilers from the built-in modules, for reference:

| Reconciler                                                            | Module                                   | `shardBy` | Periodic                                                                                         |
| --------------------------------------------------------------------- | ---------------------------------------- | --------- | ------------------------------------------------------------------------------------------------ |
| `capacity.reservations`                                               | capacity                                 | `none`    | edge-triggered, with `notBefore` wake-ups                                                        |
| `capacity.placer` (role `scheduler`)                                  | capacity (`placer: 'pool-lease'`)        | `pool`    |                                                                                                  |
| `capacity.rollup`, `capacity.usage`                                   | capacity                                 | `none`    | `all` every `5m`                                                                                 |
| `capacity.mig`                                                        | capacity (the MIG layout gate)           | `node`    | edge-triggered, with `notBefore` wake-ups                                                        |
| `inventory.hygiene`                                                   | inventory                                | `none`    | `all` every `5m`                                                                                 |
| `tenancy`                                                             | tenancy (built in)                       | `none`    | `sweep` every `30s`, plus one key per deleted tenant                                             |
| `quorum.batches`                                                      | quorum (built in)                        | `none`    | `expire` every `1m`: pending batches past their end expire and their held commands are cancelled |
| `tunnels.ssh-certificates-gc`, `tunnels.ssh-certificates-reauthorize` | tunnels (with `tunnels`)                 | `none`    | `ssh-certificates` every `1h`, `1m`                                                              |
| `events.fanout`, `events.deliver`, `events.prune`                     | events                                   | `none`    | every `1s`, `2s`, `1h` (`events.prune` only unless `retention: false`)                           |
| `audit-sinks`                                                         | audit-sinks (built in, with audit sinks) | `none`    | `tick` every `5s`                                                                                |
| `fleet.drift`                                                         | fleet                                    | `none`    | every `15m` by default                                                                           |
| `rollout`                                                             | rollout                                  | `none`    | `all` every `30s`                                                                                |
| `audit.checkpoint`                                                    | audit                                    | `none`    | `checkpoint` every `1h`                                                                          |
| `iam.inbox`, `iam.reconcile`, `iam.usage`                             | iam                                      | `none`    | `2s`, `5m`, `1h`                                                                                 |
| `vps.instances`, `vps.bootstraps`                                     | vps                                      | `none`    | edge-triggered, with `notBefore` wake-ups                                                        |
| `vps.hosts`                                                           | vps                                      | `none`    | `all` every `30s` (`vps({ hostSyncEvery })`)                                                     |
| `vps.sandboxes`                                                       | vps                                      | `none`    | `all` every `15s`: sandbox expiry, `sandbox.ready` and `sandbox.deleted`, warm pools             |
| `vps.usage`                                                           | vps                                      | `none`    | `all` every `5m`                                                                                 |

All but `capacity.placer` run in the `controllers` role. Tests and the simulator step them without timers:
`await compute.jobs['acme.widgets'].tick()` runs one batch.

## Command kinds [#command-kinds]

A module that sends work to agents registers the kinds it sends. `ctx.commands.enqueue` refuses a kind nobody
registered.

For fleet-wide destructive work (v1.0), the queue also takes batches: `ctx.commands.createBatch?(tx, { tenantId,
scope, reason, notAfterMs?, createdBy? })` stores a batch of `{ subject, kind }` leaves and answers
`{ batchId, held, required, size }`, and `enqueue(tx, { ..., batchId })` puts a command into it; the command's
`(subject, kind)` must be in the scope, and it is held until operators approve the batch. `createBatch` is optional
on `CommandQueue`, so test fakes need not implement it. See
[the destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10).

```ts title="packages/rollout/src/module.ts"
export const daemonUpgradeKind: CommandKindSpec = {
  kind: 'daemon.upgrade',
  lane: 'control',
  retrySafe: true,
  selfTerminating: true,
  timeout: '15m',
  roles: ['node', 'site'],
  payload: DaemonSectionSchema, // a protobuf-es message descriptor
};
```

| Field             | Meaning                                                                            |
| ----------------- | ---------------------------------------------------------------------------------- |
| `lane`            | `control`, `provision` or `bulk`; the daemon runs each lane on a fixed worker pool |
| `retrySafe`       | Safe to run again after a crash between accept and completion                      |
| `selfTerminating` | Ends the agent's process or boot; completes when the agent rejoins                 |
| `attested`        | The result is node-signed and verified before any transition (v0.2)                |
| `destructive`     | Must be signed from v0.2                                                           |
| `timeout`         | A duration; the CommandSweeper fails the command after it                          |
| `roles`           | Agent roles allowed to run it: `node`, `site`, `ramdisk`                           |
| `payload`         | The protobuf message the payload encodes as                                        |

`enqueue(tx, { nodeId, kind, payload, idempotencyKey, notAfter, … })` writes a `pending` row in your transaction;
the same idempotency key on the same node returns the existing command. When the CommandSweeper expires or times
out a command, it wakes the node's controller and, if the key's first segment names a registered reconciler
(`acme.widgets:<key>:…`, or `wf:<id>:…` for workflows), that reconciler for `<key>`. See
[command kinds](/docs/reference/command-kinds) and [commands](/docs/guides/nodes/commands).

## NodeSpec sections [#nodespec-sections]

`nodeSpec(ctx, node)` returns this module's part of the node's NodeSpec, a
`NodeSpecPart` from `@better-compute/core` (`Partial<NodeSpec>` whose `site` is a `Partial<SiteSpec>`). The
NodeSpec controller calls every module's `nodeSpec` in registration order and merges the parts:

* `host.config`, `telemetry` and `access` merge key by key, later modules winning; `access.sshCaPublicKeys` is a
  union;
* `devices.assignments` merge by `allocationId` and `instances` by `id`;
* `site` merges field by field, the last contribution per field winning, so one module's `boot` and another's
  `cacheImages` do not erase each other (the site-boot module contributes `boot`, provision contributes
  `cacheImages` from the pools' `policy.prewarmImages`). A `site` section with no `boot` goes on the wire as
  `boot { active: false }`;
* `daemon` is taken whole from the last module that returns it.

The controller bumps the node's generation only when the merged spec's hash changes. Your module must say when its
contribution may have changed: call `ctx.desired.invalidate(tx, nodeId)` in the transaction that changes it. The
capacity module does this on every binding change; rollout returns a `daemon` section from `daemonSpecFor`. See
[desired state and commands](/docs/guides/concepts/desired-state-and-commands).

## The module context [#the-module-context]

`api()`, `nodeSpec`, `selfCheck`, `start` and every reconciler run get a `ModuleContext`. Method handlers get a
`MethodContext`, which adds the caller; see [methods](/docs/extending/methods#the-method-context).

| Port          | What it does                                                                                                                                                                                                                                                                                                             |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `sql`         | The store. In a method handler it is the envelope's transaction; in a reconciler every `transaction` is lease-fenced                                                                                                                                                                                                     |
| `clock`       | Business time anchored to `compute_now()`; the sim clock in tests                                                                                                                                                                                                                                                        |
| `work`        | `enqueue(tx, kind, key, { notBefore? })` and `enqueueMany`: wake a reconciler                                                                                                                                                                                                                                            |
| `events`      | `publish(tx, { type, tenantId, subject, data })` and `publishMany`: append to the outbox                                                                                                                                                                                                        |
| `commands`    | `enqueue(tx, command)` and `cancel(tx, commandId)`: the durable command queue                                                                                                                                                                                                                                            |
| `desired`     | `invalidate(tx, nodeId)`: recompute a node's NodeSpec                                                                                                                                                                                                                                                                    |
| `authz`       | `upsert`, `remove`, `setPending`: maintain your rows of the authz projection                                                                                                                                                                                                          |
| `capacity`    | `reserve(tx, request, { admission }?)` and `release(tx, allocationId)`: quota plus the inline placer; pass the caller's admission so scheduler plugins' `admit` hooks run                                                                                                                                                |
| `inventory`   | `get(nodeId)`: the latest stored inventory snapshot                                                                                                                                                                                                                                                                      |
| `leases`      | `acquire`, `release`, `fence`: named leases with a fencing epoch                                                                                                                                                                                                                                                         |
| `iam`         | The IamPort; never call it inside a transaction                                                                                                                                                                                                                                               |
| `registry`    | `hasModule`, `commandKind`, `driver`: a read-only view of what is registered                                                                                                                                                                                                                                             |
| `secrets`     | The secret broker: `put`, `resolve`, `delete` over the configured stores                                                                                                                                                                                                                                                 |
| `pki`         | `issueJoinToken(tx, …)`: single-use join tokens for agents a module brings up                                                                                                                                                                                                                                            |
| `workflows`   | `start(tx, …)` and `abort(tx, id)`, present only when a hosted module runs workflows (`provisionModule()`)                                                                                                                                                                                                               |
| `afterCommit` | Runs a hook once the outermost transaction commits, never on rollback; for pokes and in-memory updates only                                                                                                                                                                                                              |
| `caches`      | `dropIdentity`, `dropSession`, `dropDecisions`: invalidate the envelope's IAM caches                                                                                                                                                                                                                                     |
| `typePrefix`  | The deployment's IAM resource type prefix (`ComputeOptions.typePrefix`, default `compute-`), always set by `betterCompute()`, for a module that decides on compute resources itself, such as vps deciding `compute:secrets:manage` on `${typePrefix}tenant/<id>`; optional in the type, so fall back to your own default |
| `log`         | `debug`, `info`, `warn`, `error`                                                                                                                                                                                                                                                                                         |

`usage` (a `UsageRecorder`) is declared on the contract but the v0.1 server leaves it unset. The built-in
capacity and vps modules write usage windows in SQL themselves.

## start, stop and selfCheck [#start-stop-and-selfcheck]

`start(ctx)` is for work that must happen before the process serves: the IAM bridge loads its authz projection
and binds its inbox there, and `eventsModule` anchors its fan-out cursor at the outbox tail so no event committed
after start is missed. It runs in every process mode, so keep it idempotent. `stop()` releases what `start()`
took.

`selfCheck(ctx)` only reads and returns findings:

```ts
interface Finding {
  check: string; // 'events.webhook-dead-letters'
  severity: 'error' | 'warning' | 'info';
  message: string;
  fix: string;
  count?: number;
}
```

`compute.selfCheck()`, the `system.selfCheck` method and `better-compute doctor` collect them from every module. A
module whose `selfCheck` throws becomes one `module.<name>` error finding; any `error` finding makes the report not
`ok`. Report what an operator can act on: a backlog, dead letters, rows left unsealed (the audit module warns about
audit events older than 3 hours that no checkpoint covers).

## tenantDeleted [#tenantdeleted]

When Better IAM deletes a tenant, the built-in `tenancy` module records it in `compute_tenant_states` and enqueues
a teardown. Its `tenancy` reconciler then calls every hosted module's `tenantDeleted(ctx, tenantId)`, plugins
included, and a 30-second `sweep` asks again while any of them answered `SYNC_AGAIN`. Once every hook answers
`DONE` in one pass, it sets `teardown_done_at` and publishes `tenant.torn-down`. See
[tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants).

```ts title="widgets.ts"
const batch = 100;

export const acmeWidgets = () =>
  definePlugin({
    id: 'acme',
    // ...
    async tenantDeleted(ctx, tenantId) {
      const { rowCount } = await ctx.sql.query(
        `DELETE FROM plg_acme_widgets WHERE id IN (
           SELECT id FROM plg_acme_widgets WHERE tenant_id = $1 LIMIT $2)`,
        [tenantId, batch],
      );
      return rowCount === batch ? 'SYNC_AGAIN' : 'DONE';
    },
  });
```

* The hook runs on every sweep until the whole teardown finishes, so a second call must find nothing left and
  answer `DONE`.
* Bound the work per call and answer `SYNC_AGAIN` while more remains, as the built-ins do (100 rows of each kind).
* Write what other modules or watchers must see through `ctx.events` in the same transaction as the change.

The built-in hooks: `capacity` ends the tenant's reservations, withdraws offers to it and archives the pools it
owns; `provision` aborts its workflows, soft deletes its private images and profiles and revokes its image
signers; `metal` retires every machine it owns; `vps` deletes its instances through the ordinary lifecycle.


# Providers (/docs/extending/providers)

> Build a provider adapter with defineProvider, make create idempotent against an API that has no idempotency keys, map errors, and test it against an emulator.



A provider adapter lets Better Compute create machines on someone else's API: a public
cloud, a dedicated-server reseller, or a hypervisor cluster you run without `bcd` on it. The adapter is code, a
`defineProvider()` definition. Each provider account is data: a row that names the
provider and holds a config its manifest validates, with credentials as secret references.

This page is for adapter authors. How operators create accounts and provider pools is on
[providers](/docs/guides/vps/providers), and how instances move is on [instances](/docs/guides/vps/instances).

> **Status.** 
  Provider pools run today: the `vps.instances` reconciler drives an account's client for every instance of a
  `class: 'provider'` pool (`agent: 'none'`). Five adapters ship: `hetzner-cloud` and `hetzner-robot` in
  `@better-compute/provider-hetzner`, `proxmox` in `@better-compute/provider-proxmox`, `gcp` in
  `@better-compute/provider-gcp`, which also implements every optional member below (discovery, ownership labels,
  reservations and networks), and `digitalocean` in `@better-compute/provider-digitalocean` (the same, except
  reservations). Platform-owned provider
  machines run too: `providerAccounts.bootstrap` and the `vps.bootstraps` reconciler call `ProviderImpl.identify` and
  `bootstrap` to turn a rented server (Hetzner Robot) into a bcd host; see [bootstrapping machines](#bootstrapping-machines).
  The provider packages are on npm but sit outside the `better-compute` umbrella, so you install them by name.
  `@better-compute/provider-kubernetes` is not an adapter of this kind: it has no `defineProvider()` and creates no
  machines. It renders the manifests that run bcd as a DaemonSet on a cluster's existing nodes, which then join a pool
  like any bcd host; see [bcd on Kubernetes](/docs/daemon/kubernetes).

## The shape of an adapter [#the-shape-of-an-adapter]

Every shipped adapter follows the same pattern: a factory that takes transport options and returns the definition, a
default instance built with no options, and a client class per account.

```ts title="acme-cloud.ts"
import {
  ComputeError,
  field,
  manifest,
  type CallContext,
  type ExternalMachine,
  type ProviderContext,
  type ProviderImpl,
  type SecretRef,
} from 'better-compute/core';
import { defineProvider, type ProviderDefinition } from 'better-compute/vps';

export const acmeCloudConfig = manifest({
  token: field.secret().describe('Acme API token, as a secret reference'),
  region: field.option(['fra', 'ams']).default('fra'),
});
export interface AcmeCloudConfig {
  token: SecretRef;
  region: 'fra' | 'ams';
}

export interface AcmeCloudOptions {
  /** Tests pass an emulator; production can pass a fetch bound to a pinned CA. */
  fetch?: typeof fetch;
  endpoint?: string;
}

class AcmeCloudClient implements ProviderImpl {
  constructor(
    private readonly config: AcmeCloudConfig,
    private readonly context: ProviderContext,
    private readonly options: AcmeCloudOptions,
  ) {}
  async listOffers(query) { /* … */ }
  async create(request, call: CallContext): Promise<ExternalMachine> { /* … */ }
  async get(id: string, call: CallContext) { /* … */ }
  async delete(id: string, call: CallContext) { /* … */ }
}

export function acmeCloudProvider(options: AcmeCloudOptions = {}): ProviderDefinition<AcmeCloudConfig> {
  return defineProvider<AcmeCloudConfig>({
    id: 'acme-cloud',
    version: '0.1.0',
    provides: ['vps'],
    config: acmeCloudConfig as unknown as ProviderDefinition<AcmeCloudConfig>['config'],
    capabilities: ['power'],
    create: (config, context) => new AcmeCloudClient(config, context, options),
  });
}

export const acmeCloud = acmeCloudProvider();
```

`defineProvider` checks the shape up front and returns the definition frozen with `extension: 'provider'`. Every
refusal is `INVALID_CONFIG` with `details.path`:

| Field          | Rule                                                                                  |
| -------------- | ------------------------------------------------------------------------------------- |
| `id`           | lower-case letters, digits and hyphens, starting with a letter, at most 64 characters |
| `version`      | a version string such as `1.0.0`                                                      |
| `provides`     | `'metal'`, `'vps'` or both, each once                                                 |
| `config`       | a manifest or any other Standard Schema                                               |
| `capabilities` | optional; non-empty strings of at most 64 characters                                  |
| `create`       | a function `(config, context) => ProviderImpl`, which may be async                    |
| anything else  | refused as not a known field                                                          |

`create` runs once per account version: vps caches clients by account id and version, and a config update builds a
fresh one. It receives the validated config, with secret fields still as references, and a `ProviderContext` of
`{ clock, secrets, log }`. Resolve the token on each request with `context.secrets.resolve(config.token)`, never once up
front, so a rotated secret takes effect and the value never lives longer than a request.

## The client contract [#the-client-contract]

| Method                                                     | Called by vps                                                                                                             | Must                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `listOffers(query)`                                        | `providerAccounts.offers`                                                                                                 | return offers matching `{ kind?, region?, gpuModel?, minGpus?, minVcpu?, minMemoryGiB? }`; `[]` for an impossible query                                                                                                                                                                             |
| `create(request, call)`                                    | once per instance, retried until it answers                                                                               | be idempotent by `call.idempotencyKey`                                                                                                                                                                                                                                                              |
| `get(id, call)`                                            | while the instance moves, and at steady state                                                                             | return the machine, or `undefined` when it does not exist                                                                                                                                                                                                                                           |
| `delete(id, call)`                                         | on instance deletion                                                                                                      | treat an already-gone machine as deleted: throw `NOT_FOUND` or return                                                                                                                                                                                                                               |
| `power?(id, action, call)`                                 | on `instances.start` and `stop` (`on`, `graceful-off`), and `instances.reboot` (`graceful-restart`, or `reset` when hard) | act on power; without it a start or stop fails the instance `UNSUPPORTED`, and a reboot is dropped with a `ProviderError` condition                                                                                                                                                                 |
| `reinstall?`, `rescue?`, `list?`                           | not yet                                                                                                                   | optional capabilities                                                                                                                                                                                                                                                                               |
| `discover?(call)`                                          | `vps.discovery`, every `vps({ discovery: { every } })`                                                                    | yield every machine the account can see, including ones better-compute did not create, with as much of `zone`, `machineType`, `vcpu`, `memoryGiB`, `gpus`, `labels`, `networks`, `reservation`, `createdAt` and `claimedBy` as the provider knows                                                   |
| `claim?(id, instanceId, call, { managed? })`               | the first reconcile of an attached instance; with `managed` before an `onDelete: 'delete'` deletion                       | write the ownership label naming `instanceId` (and with `managed`, the label that hands deletion to better-compute); `CONFLICT` when the label names another instance                                                                                                                               |
| `release?(id, instanceId, call)`                           | deleting an attached instance that detaches                                                                               | remove the ownership label only if it names `instanceId`, keep every other label; a gone machine is success                                                                                                                                                                                         |
| `listReservations?(call)`                                  | `vps.reservations`, every `vps({ reservations: { every } })`                                                              | return the account's capacity reservations as `ExternalReservation`s, each with an `offerId` that `create` consumes                                                                                                                                                                                 |
| `networks?`                                                | `vps.networks`                                                                                                            | a `ProviderNetworks`: idempotent, non-blocking `ensureNetwork`, `ensureSubnet`, `ensureFirewall`, `deleteNetwork`, `deleteSubnet`, `deleteFirewall` (each answers `pending`/`deleting` while an operation runs and is called again), and `setTags`, which replaces only the `bc-` tags of a machine |
| `identify?(machine, call)`                                 | `vps.bootstraps`, in `identifying`                                                                                        | return `{ productUuid }`, the machine's SMBIOS UUID (Hetzner Robot reads it in rescue)                                                                                                                                                                                                              |
| `bootstrap?(machine, { joinToken, bcdUrl, sha256 }, call)` | `vps.bootstraps`, in `installing`                                                                                         | metal adapters: deliver a single-use join token without user data                                                                                                                                                                                                                                   |

`create` receives `{ offerId, image, userData, name, network?, labels? }`: the flavor's `offerId`, the instance's image,
the rendered user data, the instance id as `name`, and for an instance on a tenant network a `NetworkRef` of
`{ network, subnet, tags }` (the provider-side names vps chose, used as given). `call` is a `CallContext` of `{ idempotencyKey, signal, clock, secrets, log }`, and
vps passes the instance id as the key, so a retried create after a crash or a timeout must return the same machine.

A machine is an `ExternalMachine`: `{ id, name?, state, offerId?, region?, addresses }`, with `state` one of `pending`,
`running`, `stopped`, `rescue`, `deleting`, `deleted`, `error` or `unknown`, plus optional detail that discovery and
attach use (`zone`, `machineType`, `vcpu`, `memoryGiB`, `gpus`, `labels`, `networks`, `reservation`, `createdAt`, and
`claimedBy`, the instance the ownership label names). Map the provider's statuses onto these
conservatively; Hetzner's `stopping` is still `running` and `rebuilding` is `pending`, and anything unknown is
`unknown`.

vps never calls a client inside a database transaction: it plans in one transaction, calls the provider, then applies
the answer in another guarded by the instance's version. A `get` that returns `undefined` for a live instance marks
it `lost`.

## Idempotent create without idempotency keys [#idempotent-create-without-idempotency-keys]

Few provider APIs take an idempotency key. Each shipped adapter records the key on the machine itself, looks it up
before creating, and resolves races with something the API keeps unique:

| Adapter         | Where the key lives                                                                    | What stays unique                           | Race handling                                                                                                                                                               |
| --------------- | -------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hetzner-cloud` | label `better-compute.io/idempotency-key=<key>`, plus `better-compute.io/managed=true` | the server name                             | a create that loses on the name looks the key up again and returns the winner                                                                                               |
| `proxmox`       | the VM's description, in a VMID slot hashed from the key within `[vmidMin, vmidMax]`   | the VMID                                    | concurrent creates target the same VMID and PVE lets one clone through; a slot held by another key probes the next one; an interrupted clone is finished by the next create |
| `hetzner-robot` | the server's name, `bc-<key>`                                                          | the name                                    | it claims an existing server instead of ordering one; a server someone named for another use is never claimed (`CONFLICT`)                                                  |
| `digitalocean`  | the tag `better-compute-key:<hash of the key>`, plus `better-compute-managed:true`     | nothing the API promises (names may repeat) | it looks the tag up before and after the POST; creates that raced on one key both land, and each keeps the oldest Droplet and deletes the rest                              |

The general shape, from `hetzner-cloud`:

```ts
async create(request, call) {
  const existing = await this.byKey(call.idempotencyKey, call.signal);
  if (existing) return toMachine(existing);
  try {
    return toMachine(await this.post('/servers', { name: serverName(request.name), labels: { [keyLabel]: labelValue(call.idempotencyKey) }, … }));
  } catch (error) {
    const winner = await this.byKey(call.idempotencyKey, call.signal); // a concurrent create won
    if (winner) return toMachine(winner);
    throw error;
  }
}
```

When the API keeps nothing unique, as on DigitalOcean (tags, not labels, and no name is promised unique), a worked example is
`digitalocean`: the key is hashed into a tag, and the create is a lookup, a POST and a second lookup. Whoever sees more
than one Droplet carrying the key's tag converges on the oldest, so every caller of one key ends with one machine:

```ts
// Simplified from packages/provider-digitalocean/src/compute/droplets.ts
async create(request, ctx) {
  // … validate the offer, image and region …
  const existing = await this.byKey(ctx.idempotencyKey, gpu, signal); // a retry finds what its first attempt made
  if (existing.length > 0) return toMachine(await this.converge(existing, signal));
  // … the VPC and the SSH keys …
  let created;
  try {
    created = (await this.client.request('POST', '/droplets', { body: { …, tags: [keyTag(ctx.idempotencyKey), …] }, signal, budget: 'mutate' })).droplet;
  } catch (error) {
    if (!isComputeError(error, 'UNAVAILABLE')) throw error;
    const landed = await this.byKey(ctx.idempotencyKey, gpu, signal); // an answer that never arrived may still have created one
    if (landed.length === 0) throw error;
    return toMachine(await this.converge(landed, signal));
  }
  const found = new Map((await this.byKey(ctx.idempotencyKey, gpu, signal)).map((d) => [d.id, d]));
  if (created && !found.has(created.id)) found.set(created.id, created);
  if (found.size === 0)
    throw new ComputeError('PROVIDER_ERROR', 'DigitalOcean accepted a create but returned no Droplet', undefined, { provider: 'digitalocean', reason: 'badBody' });
  return toMachine(await this.converge([...found.values()], signal)); // a racing create may have posted too
}
```

`byKey` lists by the key's tag (`better-compute-key:<24 hex of sha256(key)>`) and keeps only Droplets this deployment
owns: the managed tag and its deployment tag. So ownership is settled by the lookup, and `converge` keeps the oldest
of what it is given and deletes the others by id, one request each. A tag-scoped bulk delete would be one call, but it
also removes whatever else carries the tag, so `digitalocean` never calls the bulk endpoints. `list` and the
idempotency lookup are scoped by tag; `discover` is the exception: it yields every Droplet in the configured regions
unless `discoverTag` is set. DigitalOcean lists GPU Droplets separately and not with a tag filter, so for a GPU size
`byKey` also pages the GPU list and filters it by tag in code.

Mark every machine you create, and act only on marked machines. On a shared account or cluster that is the only thing
that keeps `delete` away from someone else's server: `proxmox` refuses to delete or power a VM not tagged
`better-compute` with `ACCESS_DENIED`, and `list` in both Hetzner adapters and Proxmox yields only managed machines.

## Errors [#errors]

Throw `ComputeError`s. The reconciler records any failure as a `ProviderError` condition on the instance, with the
code as its reason, then:

| Code                                     | vps does                                                                                                                                                                                      | Use it for                                                                                                                                                        |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UNSUPPORTED`                            | fails the instance                                                                                                                                                                            | an action the provider cannot do (`nmi` on Hetzner Cloud or Proxmox)                                                                                              |
| `INVALID_STATE`, `INVALID_CONFIG`        | fails the instance                                                                                                                                                                            | a request that can never succeed as configured                                                                                                                    |
| `CAPACITY_UNAVAILABLE`, `QUOTA_EXCEEDED` | from `create`, or from `get` while the instance is `scheduling` or `provisioning`: fails the instance with a `ProvisionFailed` condition (no retry; the quota is refunded when it is deleted) | the provider refused the machine for capacity or quota, at once or after accepting the create (GCE reports a stockout on the insert operation, which `get` reads) |
| `CONFLICT`                               | from `claim`: fails an attached instance with `AttachConflict`                                                                                                                                | the ownership label names another instance                                                                                                                        |
| `PROVIDER_ERROR`                         | retries on a later poll                                                                                                                                                                       | API errors, with `details` such as `{ status, code }`; an unreachable endpoint                                                                                    |
| `RATE_LIMITED`                           | retries on a later poll                                                                                                                                                                       | HTTP 429, with `details.retryAfterMs`                                                                                                                             |
| `NOT_FOUND`                              | success, from `delete`                                                                                                                                                                        | a machine that is already gone                                                                                                                                    |

Any other code is retried the same way (`CAPACITY_UNAVAILABLE` too, once the machine exists), and a thrown value that is not a `ComputeError` is recorded as
`PROVIDER_ERROR`. Keep secrets out of messages: the first 500 characters of the message are stored in the condition
and returned with the instance.

`hetzner-cloud` is a compact reference: 404 is absent (`get` answers `undefined`; `delete`, `power` and `reinstall`
throw `NOT_FOUND`), 429 is `RATE_LIMITED` with `retryAfterMs` from `RateLimit-Reset`, a network failure is
`PROVIDER_ERROR` with `details.provider`, and every other status is `PROVIDER_ERROR` with Hetzner's error code.

## Offers [#offers]

An offer is `{ id, kind, region, vcpu, memoryGiB, diskGiB?, gpus?, pricePerHour?, currency?, available }`. Its `id` is
what a provider pool's [flavor](/docs/guides/vps/flavors) names as `offerId`, so keep ids stable across calls.

* Default `region` to the account's own; honour `query.region` when given.
* Return `[]`, not an error, for what the provider does not have: `hetzner-cloud` answers GPU and metal queries with
  nothing, and skips deprecated server types.
* `available` says whether a create would succeed now: Hetzner reads it from the location's datacenters, Proxmox from
  whether an online node (or the pinned one) has the CPUs and free memory.
* Page through the provider's catalog yourself; `hetzner-cloud` follows `meta.pagination.next_page`.

## User data [#user-data]

vps renders user data from the tenant's SSH keys, or passes the tenant's own cloud-init (resolved from a `secret://`
reference when it is one). No join token ever goes into user data, so an adapter never has to filter secrets out of
it; but it must deliver what it is given. `hetzner-cloud` sends it as `user_data`. The PVE API cannot upload cloud-init
snippets, so `proxmoxProvider({ writeSnippet })` takes a function that writes one (over SFTP or a shared mount) and
returns its volume id for `cicustom`; without it, create refuses anything but an empty `#cloud-config`.

## Metal adapters and bootstrap [#metal-adapters-and-bootstrap]

An adapter with `provides: ['metal']` turns machines into `metal-host` nodes that run `bcd`. Its `bootstrap` delivers a
single-use `provisioned` join token, the `bcd` download URL and its digest, and must never put the token in
tenant-readable user data. `hetzner-robot` shows how:

1. Activate rescue with the account's SSH key and wait for SSH through an injected `SshExecutor`.
2. Write the token over SSH **stdin** into a `umask 077` file, so it never appears in argv or the script.
3. Run `installimage` (software RAID 1 over NVMe, else SATA) with a post-install script that places the token at
   `/etc/bcd/join-token` (0600), downloads `bcd` over https, checks its sha256, installs two units and shreds
   itself: `bcd-join.service`, a oneshot running `bcd join --server <controlPlaneUrl> --token-file /etc/bcd/join-token --attestor provisioned --serving-ca-pin … --join-signing-pin … --join-encryption-pin …`
   (`robotJoinArgv(url, pins)`), and `bcd.service` (`bcd node`) after it.
4. Reboot and deactivate rescue.

Robot needs the control plane's URL, `vps({ bootstrap: { controlPlaneUrl } })`, passed to `ProviderImpl.bootstrap`.
It is optional in `vps()`, where a value that could break out of a unit file (anything but https, or spaces,
quotes, `%` or `$`) is `INVALID_CONFIG` at startup; the Robot adapter refuses a missing or unsafe one with
`INVALID_INPUT` before anything reaches the machine. It also needs the three join pins,
`vps({ bootstrap: { pins: { servingCa, joinSigning, joinEncryption } } })` (`sha256/<base64>` SPKI pins from the PKI
bundle, validated at construction; `ProviderImpl.bootstrap` receives them as `pins`), and refuses a bootstrap without
them (`INVALID_CONFIG`). With no `config.toml` on the new system, `bcd join --server` reads the gateway from the
control plane's `/.well-known/better-compute`, refuses the document if its pins differ from the ones on the command
line, joins, and writes `config.toml` itself; the pins never come from the document, which only the control plane's
Web PKI certificate vouches for. A join unit that runs again after the node joined exits 0. The installed `bcd` joins through
the `provisioned` attestor. `scripts/daemon-install-contract.mjs` writes the exact command lines the cloud-init,
Ignition and Robot installs run, and bcd's `TestInstallContract` parses each with its real flags. Without a `resolveTarget` that maps the token to a node,
the token must carry `expect: { productUuid }`, and the machine joins as a new node. See
[attestors](/docs/extending/attestors).

### Bootstrapping machines [#bootstrapping-machines]

`providerAccounts.bootstrap({ tenantId, accountId, poolId, machineId | offerId, idempotencyKey })` (decided as
`compute:providers:manage`) records a request to turn one machine into a node of a `vps-host` or `metal-host` pool
that runs `bcd` and that the tenant owns. Name exactly one of `machineId` (a machine the account already has) and
`offerId` (claim or order one). It needs `vps({ bootstrap: { bcdUrl, sha256 } })` and the host PKI, else
`INVALID_CONFIG`, and an adapter with both `identify` and `bootstrap`, else `UNSUPPORTED`. A retry with the same key
returns the current status; another request under the key is `CONFLICT`.

The `vps.bootstraps` reconciler then drives it through `claiming` (`create`, keyed by the bootstrap id),
`identifying` (`identify`), `installing` (a fresh single-use `provisioned` join token bound to that UUID, 2 h by
default and at most 6 h, handed to `bootstrap`), `joining` (every 30 s it looks for a node of the pool whose applied
inventory reports that UUID) and `joined` or `failed`. The token's secret is never stored, and a retry revokes the
last token before it mints another. Provider errors are retried with back-off up to `maxAttempts` (default 3), and a
machine that does not join within `joinTimeoutMs` (default 1 h) fails with `JOIN_TIMEOUT`. Events
`provider.bootstrap.started`, `installed`, `joined` and `failed` have the subject `provider-bootstrap/<id>`. The table
is `provider_bootstraps` (migration `vps/0003_provider_bootstraps`).

## Registering [#registering]

Pass definitions to `vps({ providers })`. vps refuses anything not built with `defineProvider` and a provider listed
twice (`INVALID_CONFIG`), and registers the definitions as its own extensions:

```ts title="compute.ts"
import { vps } from 'better-compute/vps';
import { acmeCloud } from './acme-cloud';

export const compute = betterCompute({ modules: [vps({ providers: [acmeCloud] })] /* , … */ });
```

Do not also list them in `betterCompute({ providers })`: the registry would see `provider:acme-cloud@0.1.0` twice
and refuse to start. `providerAccounts.create` answers an unknown provider id with `INVALID_INPUT`, and
`details.known` lists the registered ones.

## Testing against an emulator [#testing-against-an-emulator]

Test adapters against an in-memory emulator of the endpoints they use, plugged in through the factory's `fetch`
option. The emulator is plain code that answers `Request`s from a `Map`, which makes races, failures and pagination
easy to inject:

```ts title="acme-cloud.test.ts"
class AcmeEmulator {
  servers = new Map<number, Record<string, unknown>>();
  failNext?: { status: number; code: string };
  fetch = async (input: string | URL | Request, init?: RequestInit): Promise<Response> => {
    const url = new URL(String(input));
    if (new Headers(init?.headers).get('authorization') !== 'Bearer acme-token')
      return Response.json({ error: { code: 'unauthorized' } }, { status: 401 });
    // … route GET/POST/DELETE on url.pathname, apply failNext, paginate
  };
}

const api = new AcmeEmulator();
const provider = acmeCloudProvider({ fetch: api.fetch, endpoint: 'https://acme.test/v1' });
const secrets = { resolve: async () => 'acme-token' };
```

Then run the generic suite. `providerConformanceCases()` from `better-compute/vps` turns the contract into cases, and
`runConformance` from `@better-compute/core/conformance` runs them in any test runner and collects every failure
instead of stopping at the first:

```ts
import { runConformance } from '@better-compute/core/conformance';
import { providerConformanceCases } from 'better-compute/vps';

const report = await runConformance(providerConformanceCases(), () => ({
  provider: acmeCloudProvider({ fetch: new AcmeEmulator().fetch }),
  config: { token: 'env://ACME_TOKEN' },
  offerId: 'small',
  image: 'ubuntu-24.04',
  context: { secrets, clock },
}));
expect(report.failed.map(({ name, error }) => `${name}: ${String(error)}`)).toEqual([]);
```

<TypeTable
  type="{
  provider: { type: 'ProviderDefinition', description: 'The definition under test.', required: true },
  config: { type: 'unknown', description: &#x22;An account config; it must pass the provider's manifest.&#x22;, required: true },
  offerId: { type: 'string', description: 'An offer create accepts; listOffers must list it.', required: true },
  image: { type: 'string', description: 'An image create accepts.', required: true },
  context: { type: 'Partial<ProviderContext>', description: 'secrets for the config refs, a clock, a logger. Without secrets, resolving fails INVALID_CONFIG.' },
  prefix: { type: 'string', description: &#x22;Prefix of the cases' idempotency keys, so concurrent runs against a live account never share one.&#x22;, default: &#x22;'bc-conformance'&#x22; },
  powerObservable: { type: 'boolean', description: 'false for providers that can act on power but not report it (Hetzner Robot): the power case then checks only that the actions succeed.', default: 'true' },
}"
/>

The five cases check that the definition builds a client with `listOffers`, `create`, `get` and `delete`; that offers
are well formed, include `offerId`, and that `{ minVcpu: 1000000 }` matches nothing; that a retried create names the
same machine; that `get` follows a machine, `delete` removes it, a second `delete` fails only with `NOT_FOUND`, and an
unknown id is `undefined`; and that `power`, when offered, stops and starts it. Each case deletes what it created.
Pass a factory, as above, so every case gets a fresh emulator.

The same suite runs against a live account or a recorded transport: the `fetch` hook is where you record real traffic
and replay it later. After the generic cases, test what is specific to your API. The shipped suites are good models:

| Suite                                  | Beyond conformance                                                                                                                                                                                                                   |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tests/provider-hetzner.test.ts`       | labels and names, a concurrent create losing on the name, 401, 422 and 429 mapping, paginated offers with prices, power and rebuild                                                                                                  |
| `tests/provider-proxmox.test.ts`       | VMID slots, concurrent and interrupted clones, disk growth, `writeSnippet`, refusing untagged VMs, tasks and their timeouts, reinstall into the same VMID                                                                            |
| `tests/provider-digitalocean.test.ts`  | the tag scheme, a create racing on one key and converging on the oldest Droplet, the ownership rules of `delete` and `power`, 422 message mapping, SSH keys, GPU Droplets, claim races, the request budget, 401, 403 and 429 mapping |
| `tests/provider-hetzner-robot.test.ts` | idempotent claims and claim races, power through resets, the rescue bootstrap over a fake SSH executor, the script quoting (set `BC_SH` to a POSIX shell to syntax-check it)                                                         |

Inject time as well as transport: `proxmoxProvider` takes `sleep`, `taskPollMs` and `taskTimeoutMs`, and
`hetznerRobotProvider` takes `sleep`, `rescuePollMs` and `rescueTimeoutMs`, so the tests never wait on a real clock.

```bash
pnpm vitest run tests/provider-hetzner.test.ts tests/provider-proxmox.test.ts tests/provider-hetzner-robot.test.ts
```

  - [Providers and accounts](/docs/guides/vps/providers): Accounts, credentials and the shipped adapters from the operator's side.

  - [Instances](/docs/guides/vps/instances): How provider instances are created, polled, powered and deleted.

  - [Drivers](/docs/extending/drivers): The runtime driver kind, for VPS hosts that run bcd.


# Scheduler plugins (/docs/extending/scheduler-plugins)

> Add filters, scorers, atom pickers and admission hooks to the capacity placer with defineSchedulerPlugin, and tune the built-in scorer weights.



The placer of the built-in capacity module decides which node and which
atoms a request gets. You cannot replace the placer, but you can shape every decision it
makes: a scheduler plugin adds **filters** that turn nodes down, **scorers** that rank the rest, **atom pickers**
that choose devices on a node, and an **admission hook** that can refuse a request outright. None of them can break
the booking guarantees: whatever they choose is still committed with guarded statements under the
`EXCLUDE` constraint. [Placement](/docs/guides/capacity/placement) explains the pipeline they plug into.

## The contract [#the-contract]

```ts title="@better-compute/core"
interface SchedulerPluginInput {
  id: string;
  admit?(request: PlacementRequest, context: AdmissionContext): Promise<void>;
  filters?: { id: string; filter(request: PlacementRequest, node: NodeView): true | { reason: string } }[];
  scorers?: { id: string; weight?: number; score(request: PlacementRequest, node: NodeView): number }[];
  preferAllocation?: {
    class: string;
    pick(request: DeviceRequest, free: Atom[], node: NodeView): Atom[] | undefined;
  }[];
}
```

`defineSchedulerPlugin(input)` (from `better-compute`) freezes it and marks it as a `scheduler` extension. Filters,
scorers and pickers are synchronous and pure: no SQL, no clock, no network. They see a read-only `NodeView` and the
`PlacementRequest`, and the same inputs must give the same answer, because the simulator replays placements.

### What a plugin sees [#what-a-plugin-sees]

<TypeTable
  type="{
  id: { type: 'string', description: 'The node id.' },
  'siteId, poolId, ownerTenantId': { type: 'string', description: 'Where the node lives and who owns it.' },
  role: { type: 'NodeRole', description: 'Only host nodes reach plugin filters; others fail eligibility first.' },
  'provisionState, connectivity, trust': { type: 'string', description: 'Lifecycle, session and trust state.' },
  'maintenance, cordoned': { type: 'boolean', description: 'Always false for nodes that reach plugin filters.' },
  labels: { type: 'Record<string, string>', description: 'Node labels, such as rack or fabric.domain.' },
  noScheduleReasons: { type: 'string[]', description: 'Health taints; empty for nodes that reach plugin filters.' },
  devices: { type: 'Device[]', description: 'Every device, including partitioned parents, for topology.' },
  free: { type: 'Atom[]', description: &#x22;Atoms free for the request's interval: { id, class, parentId?, healthy, bookable, attributes }.&#x22; },
  resources: { type: 'Record<string, { allocatable, ratio, allocated }>', description: 'Scalar resources such as vcpu and memoryGiB.' },
  links: { type: 'Record<string, Record<string, GpuLink>>', description: &#x22;links[a][b] is NV, PIX, PXB, PHB, NODE or SYS, when inventory reported it.&#x22; },
}"
/>

The request carries `tenantId`, `poolId`, `projectId`, `reservationId` or `instanceId`, `devices` (class, count,
selector, topology, sharing), `topology`, `gang`, `resources`, `imageId`, `during`, `priority`, `excludeNodes` and
`nodeLabels` (labels every candidate must carry with exactly those values; the built-in `labels` filter drops the
others before plugin filters run, so a plugin never sees them).

## Where each hook runs [#where-each-hook-runs]

<Mermaid
  chart="flowchart TD
  A[&#x22;admit() of every plugin, once per request&#x22;] --> E[&#x22;Built-in: excluded, pool, eligibility, labels, scalars&#x22;]
  E --> F[&#x22;Plugin filters, in plugin order&#x22;]
  F --> K[&#x22;Per device request: plugin pickers, else the default pick; nvlink-all check&#x22;]
  K --> S[&#x22;Built-in scorers + plugin scorers, weighted sum&#x22;]
  S --> T[&#x22;Random pick among the best topN, then guarded commit&#x22;]"
/>

| Hook               | Runs                                                                                               | A failure means                                                            |
| ------------------ | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `admit`            | Once per request, before candidates are gathered, only when the placement has an admission context | The error is thrown to the caller and the request's transaction rolls back |
| `filters`          | Per candidate node, after the built-in eligibility, exclusion and scalar checks, before device fit | The node is rejected with `filter: <filter id>` and your reason            |
| `preferAllocation` | Per device request whose class matches, before the default NVLink- and NUMA-aware pick             | See below                                                                  |
| `scorers`          | Per node that fits                                                                                 | Scores are clamped to 0..1; a non-finite score counts as 0                 |

The first filter that answers `{ reason }` turns the node down, and the reason shows in the `Unschedulable`
message of a pending reservation (`no node fits (12 nodes): 7 × rack-a: …`). Keep reasons short and stable.

### Admission hooks [#admission-hooks]

`admit` receives the request and the admission context of the decision that let
it in: the tenant, identity, identity kind, session kind, agent and delegation, action and resource. It is an
admission decision, so it runs once, in the request's own transaction:

* `reservations.create`, with both placers (the pool-lease placer runs it before answering `queued`);
* whole-node leases (`binding: 'node'`) at create, although they do not go through the placement pipeline;
* bookings a module makes through `ctx.capacity.reserve(tx, request, { admission })` in its caller's request:
  `instances.create` passes its decision, so VPS instances are admitted too.

The system's own re-placements of an admitted request carry no caller and are not re-admitted: pending retries,
BookingRepair, bind-timeout re-places, spot preemption and a module's retries. Filters, pickers and scorers apply to
every attempt.

Use IAM policies for who may ask for what; they see the request's attributes. Use `admit` for rules that are about
placement itself, and throw a `ComputeError` to refuse:

```ts
import { ComputeError, defineSchedulerPlugin } from 'better-compute';

export const gangCap = defineSchedulerPlugin({
  id: 'gang-cap',
  async admit(request, context) {
    if (request.gang && request.gang.size > 8 && context.identityKind === 'agent')
      throw new ComputeError('UNSUPPORTED', 'Agents may book gangs of at most 8 nodes');
  },
});
```

### Atom pickers [#atom-pickers]

For each device request, the first plugin (in `scheduler.plugins` order) with a picker for that exact `class` that
returns an array wins. Return `undefined` to fall through to the next picker and finally to the default pick. The
array must hold exactly `count` distinct atoms taken from `free`; anything else rejects the node with the plugin's
id as the filter. `topology: 'nvlink-all'` is checked on your pick as on the default one.

## Adding a scorer [#adding-a-scorer]

Suppose your racks on power tier `low` are cheaper to run, and you want placements to prefer them without ever
refusing the others:

```ts title="power-tier.ts"
import { defineSchedulerPlugin } from 'better-compute';

export const powerTier = defineSchedulerPlugin({
  id: 'power-tier',
  scorers: [
    {
      id: 'power-tier.low',
      weight: 0.5,
      score: (_request, node) => (node.labels['power-tier'] === 'low' ? 1 : 0),
    },
  ],
  filters: [
    {
      id: 'power-tier.no-drain',
      filter: (_request, node) =>
        node.labels['power-tier'] === 'draining' ? { reason: 'rack is draining' } : true,
    },
  ],
});
```

```ts title="compute.ts"
import { betterCompute } from 'better-compute';
import { powerTier } from './power-tier';

export const compute = betterCompute({
  // …database, secret, baseURL, iam
  scheduler: {
    plugins: [powerTier],
    weights: { 'power-tier.low': 0.75, spread: 0.25 },
    topN: 5,
  },
});
```

Then check it the way the capacity tests do, with the pure planner and hand-built views:

```ts title="power-tier.test.ts"
import { planPlacement } from 'better-compute/capacity';

const plan = planPlacement(request, [nodeA, nodeB], pool, { plugins: [powerTier] });
expect(plan.ranked[0]!.nodeId).toBe('b'); // b carries power-tier=low
expect(plan.ranked[0]!.scores['power-tier.low']).toBe(1);
```

Each ranked plan carries its per-scorer `scores` and the total `score`, and `plan.rejected` lists every node turned
down with the filter that did it.

### Weights [#weights]

A node's total is the sum of weight × score over every scorer. The built-in weights are:

| Scorer          | Default weight | Scores 1 for                                                      |
| --------------- | -------------- | ----------------------------------------------------------------- |
| `binpack`       | 1              | Nodes left fullest after the placement (keeps 8-GPU boards whole) |
| `topology`      | 1              | All picked GPUs on NVLink; 0.5 when a link is unknown             |
| `fragmentation` | 0.25           | Leaving 0 free atoms (0.5 for a power of two)                     |
| `spread`        | 0              | The emptiest nodes (1 − binpack)                                  |
| `image-cached`  | 0.5            | Nodes whose runtime already caches the request's `imageId`        |

A plugin scorer's weight is, in order of precedence: `scheduler.weights[id]`, the scorer's own `weight`, then 1.
`scheduler.weights` also overrides the built-ins by id; `{ spread: 1, binpack: 0 }` turns packing into spreading.
Give your scorers ids of their own: a plugin scorer named like a built-in replaces that built-in's score. Ties
break on node id, and the placer picks at random among the best `topN` (default 5), with a seeded random in the
simulator.

### The image-cached scorer [#the-image-cached-scorer]

`image-cached` is the one built-in scorer that reads state outside the node view. When a request carries an
`imageId` (VPS instances), the inline placer reads the nodes of the pool whose `node_image_cache` row for that image
is `ready`, and scores them 1. The table is fed from the daemon's `images` inventory section; while it does not
exist the set is empty and the scorer is 0 for every node. Requests without an `imageId` skip it. Plugin scorers
cannot read the database; if yours needs outside state, keep it in node labels (set by operators with
`nodes.label`) or device attributes (reported by inventory).

## Performance [#performance]

For requests without selectors, without a gang and without plugin scorers, the inline placer first asks SQL for
every fitting node's free-atom counts and plans only the best 48 (at least 8 × `topN`), falling back to the whole
pool whenever nodes outside that shortlist could still rank in the top `topN`. **Any plugin scorer turns the
shortlist off**, because SQL cannot compute your score: every placement then loads and plans every node of the pool.
Filters and pickers keep the shortlist. On large pools prefer a filter, or a label a selector can match, over a
scorer when either would do.

## Placers [#placers]

`Placer` is a contract in `core`, but the capacity module does not take a custom one. It offers two, chosen with
`capacity.placer`:

|            | `inline` (default)                                                | `pool-lease` (v0.4)                                                                                                                                                                                                                         |
| ---------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Runs       | In the admission transaction, pool row locked only for the commit | In the `capacity.placer` reconciler (`shardBy: 'pool'`, worker role `scheduler`), batches of at most 64; the pool row is locked only after the batch's rows and the index are loaded                                                        |
| Plans from | The SQL shortlist or every node of the pool                       | An in-memory index of the pool (`PoolIndex`), kept across batches for up to 5 s while the pool row is unchanged. It loads counts and bookings up front and each node's full devices and links the first time a placement looks at that node |
| Plugins    | `admit`, filters, pickers, scorers                                | `admit` at create; filters, pickers and scorers in each batch                                                                                                                                                                               |

Instances, gangs and BookingRepair always place inline. Both placers run the same `planPlacement` with the same
plugins, so a plugin behaves the same under either. See [placement](/docs/guides/capacity/placement#two-placers) and
[scaling](/docs/operations/scaling).

> **Registering.** 
  Scheduler plugins reach the built-in capacity module two ways: `betterCompute({ scheduler: { plugins } })`, and the
  `extensions` of any module or plugin you register. Both lists are joined (the options' plugins first) and checked
  for a duplicate id, and the placer runs them all.



# Sinks (/docs/extending/sinks)

> Event and audit sinks receive every committed row at least once, each with its own durable cursor. How dispatch works, the cursor pitfalls, poison events, and usage.



Every change in Better Compute writes an event to the outbox in the same transaction as the
change. An **event sink** is how you get all of those events out of the database and into something else: a
warehouse, a queue, a platform-wide audit feed. The control plane hands each sink every committed event, in commit
order, at least once, and remembers per sink how far it got.

## The contract [#the-contract]

```ts title="@better-compute/core"
interface EventSink {
  readonly extension: 'sink';
  readonly kind: 'events';
  readonly id: string;
  /** At-least-once; events carry ids for deduplication. */
  deliver(events: readonly ComputeEvent[]): Promise<void>;
}

interface ComputeEvent<D = Json> {
  id: string;       // evt_…, stable across redeliveries
  type: string;     // reservation.placed, node.connected, …
  tenantId: string;
  subject: string;  // reservation/rsv_…
  time: string;     // RFC 3339, from compute_now() at insert
  sequence: string; // <txid>:<seq>, the outbox position
  data: D;
}
```

`deliver` resolves when the batch is safely stored on your side, and throws when it is not. That is the whole
protocol.

```ts title="compute.ts"
import { betterCompute, defineSink } from 'better-compute';

const warehouse = defineSink({
  kind: 'events',
  id: 'warehouse',
  async deliver(events) {
    // One idempotent write per batch; event.id is the deduplication key.
    await warehouseClient.upsert('compute_events', events.map((event) => ({ key: event.id, row: event })));
  },
});

export const compute = betterCompute({
  // …database, secret, baseURL, iam
  sinks: { events: [warehouse] },
});
```

The id must be a non-empty string, unique among the event sinks (two sinks with one id would share a cursor, so the
options refuse it with `INVALID_CONFIG`). Pick it once and keep it: the cursor is stored under the id.

A module or plugin can ship a sink instead: an event sink it lists in its `extensions` joins `sinks.events` and is
delivered to the same way (the same uniqueness check applies across both lists).

## How dispatch works [#how-dispatch-works]

OutboxDispatch runs in the `sinks` worker role with one lease and one loop per sink:
`singleton:outbox-dispatch:<sink id>` (a `h-` plus sha256 prefix for an id that is not lease-safe, that is, not 1 to
100 of letters, digits and `. _ @ -`). Exactly one process delivers to each sink at a time, different `sinks` workers
can deliver different sinks, and a slow sink never delays the others. Tests step it with
`compute.jobs['outbox-dispatch'].tick()`, which runs one pass over every sink.

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant W as OutboxDispatch (the sink's lease holder)
  participant O as outbox
  participant S as your sink
  participant C as outbox_cursors
  W->>C: new lease epoch: load this sink's cursor
  loop each pass
    W->>O: read after the sink's cursor (finished transactions only)
    W->>S: deliver(batch)
    alt resolved
      W->>C: save the cursor, forward only
    else threw
      W->>W: bisect or back off; cursor unchanged
    end
  end"
/>

* **Per-sink cursors and leases.** Each sink reads from its own position under its own lease, so a failing or slow
  sink backs off without holding the others back.
* **Batches.** A pass reads up to 4 × `worker.batchSize` events per sink (128 with the default of 32) and hands them
  to `deliver` in one call.
* **Failure.** A throw leaves the cursor where it was. A failing batch is bisected: the next pass hands the sink
  half as many events, until the failing event is alone; the batch size grows back after successes. A lone event
  that fails is retried after 1 second, doubling up to 5 minutes, with a warning logged with the attempt count. See
  [poison events](#poison-events) for when an event is dead-lettered.
* **Handover.** Cursors are saved to `outbox_cursors` after every delivered batch. When a process takes a sink's
  lease under a new epoch it reloads that sink's cursor, so a new holder resumes where the last one saved. A batch delivered whose save
  failed, or whose holder died before saving, is delivered again.
* **A new sink** has no stored cursor and starts at the origin: it receives every event the outbox still holds.

## Cursors and the txid pitfall [#cursors-and-the-txid-pitfall]

```sql
CREATE TABLE outbox (
  seq bigserial PRIMARY KEY,
  txid xid8 NOT NULL DEFAULT pg_current_xact_id(),
  topic text NOT NULL,
  payload jsonb
);
CREATE TABLE outbox_cursors (            -- core/0009_runtime
  name text PRIMARY KEY,                 -- sink:<id>
  stream text NOT NULL DEFAULT 'outbox', -- or authz_changes
  txid xid8 NOT NULL,
  seq bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT compute_now()
);
```

An event's position is the pair `(txid, seq)`, never `seq` alone. Sequence numbers are drawn when a row is
inserted, but transactions commit in a different order: transaction A can take `seq` 41, transaction B `seq` 42,
and B commit first. A reader that saw 42 and moved its cursor past it would never see 41. Every reader in Better
Compute therefore reads like this:

```sql
SELECT o.seq::text AS seq, o.txid::text AS txid, o.topic, o.payload FROM outbox o
 WHERE o.txid < pg_snapshot_xmin(pg_current_snapshot())
   AND (o.txid, o.seq) > ($1::xid8, $2::bigint)
 ORDER BY o.txid, o.seq LIMIT $3;
```

Only transactions older than every transaction still running are read, so nothing can commit behind the cursor
afterwards. If you read the outbox yourself, keep all of it:

* **Filter on `txid < pg_snapshot_xmin(…)`.** Without it you will skip events under load, and no test on an idle
  database will show it.
* **Compare and order the pair.** `(txid, seq) > (…)` as a row comparison, `ORDER BY txid, seq`.
* **Order by the columns, not their text.** A query that selects `txid::text AS txid` and then says `ORDER BY txid,
  seq` sorts the text aliases (`'10' < '9'`). Qualify the columns (`o.txid`) or alias the text differently
  (`txid_text`), as the built-in readers do.
* **Keep the numbers as strings.** `xid8` is 64 bits; carry it as a decimal string and compare with `BigInt`, never
  as a JavaScript number.
* **Two spellings.** Server-side events (`events.list`, sinks, `compute.events`) write `sequence` as `<txid>:<seq>`.
  The events package's readers, the SSE stream and webhook bodies write `<txid>.<seq>`. They name the same position.

Cursor saves only move forward: the upsert applies only when the new `(txid, seq)` is greater, so a late save by a
holder that lost the lease cannot move a cursor back.

**Retention.** `retentionModule()` from `@better-compute/postgres` deletes outbox rows only behind the lowest cursor
of the `outbox` stream (and older than a minimum age); a stream with no cursor is never pruned. So a sink that stops
delivering holds the outbox back, and a sink you remove from the options leaves its `sink:<id>` row behind: delete
that row, or retention keeps waiting for it. On a schema without `outbox_cursors`, cursors live in memory, a
warning is logged once, and a new lease holder starts every sink at the origin.

## Poison events [#poison-events]

OutboxDispatch isolates an event your sink can never accept, so it no longer blocks the sink:

1. A batch that throws is bisected pass by pass until the failing event is delivered alone.
2. The lone event is retried with back-off. After 5 failures it is tested against its neighbour: the next event is
   delivered on its own.
3. If the neighbour goes through, the lone event is poison. It is written to `outbox_dead_letters` (the event as
   delivered, the error and the attempt count), logged as an error, and the sink's cursor moves past both events.
4. If the neighbour fails too, the sink itself is down: nothing is dead-lettered and the back-off goes on, so an
   outage never loses events. A failing event with no neighbour yet (the last one in the outbox) keeps retrying
   until one exists.

```sql
CREATE TABLE outbox_dead_letters (       -- core/0019_outbox_dead_letters
  sink text NOT NULL,                    -- EventSink.id
  txid xid8 NOT NULL,
  seq bigint NOT NULL,
  event jsonb NOT NULL,                  -- the ComputeEvent as delivered
  error text NOT NULL,
  attempts integer NOT NULL CHECK (attempts > 0),
  dead_at timestamptz NOT NULL DEFAULT compute_now(),
  PRIMARY KEY (sink, txid, seq)
);
```

On a schema without the table the event is only logged, and the cursor still moves past it. Nothing redelivers a
dead-lettered event: read the table and replay what you need yourself. Retention does not prune it.

The dispatcher tells "this event" from "this destination" only by whether the neighbour goes through, right after
the fifth failure. When your sink can tell a permanent failure from a transient one, it can do better itself:
isolate poison inside `deliver` and keep the dead letters where your destination lives.

```ts title="warehouse-sink.ts"
import { defineSink } from 'better-compute';

export const warehouse = defineSink({
  kind: 'events',
  id: 'warehouse',
  async deliver(events) {
    try {
      await store.insertMany(events); // the fast path: one idempotent write
      return;
    } catch (error) {
      if (isTransient(error)) throw error; // retried with back-off, cursor unchanged
    }
    // A permanent failure somewhere in the batch: store what you can, park the rest.
    for (const event of events) {
      try {
        await store.insertMany([event]);
      } catch (error) {
        if (isTransient(error)) throw error;
        await store.deadLetter(event, String(error)); // your own table or queue
      }
    }
  },
});
```

Throw only for failures that a retry can fix (the destination is down, a timeout). An event you record and skip
yourself never costs the sink five retries and a neighbour test.

## Webhooks are not a sink [#webhooks-are-not-a-sink]

Per-tenant webhooks are built differently, as the `eventsModule()` from `better-compute/events`, and they show the
other way to consume the outbox: from a module of your own.

|               | Event sink (`sinks.events`)                                                                                     | `eventsModule()` webhooks                                                                                                               |
| ------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Runs in       | OutboxDispatch, the `sinks` role                                                                                | Reconcilers `events.fanout` (every 1 s), `events.deliver` (every 2 s) and `events.prune` (hourly), the `controllers` role               |
| Cursor        | `outbox_cursors`, `sink:<id>`; holds retention back                                                             | `event_cursors`, `webhooks`; does not hold retention back                                                                               |
| Starts at     | The origin                                                                                                      | The outbox tail when the module first starts: history is not replayed                                                                   |
| Unit of retry | A batch, bisected down to one event; no attempt limit while the neighbour fails too                             | One delivery row per webhook and event: backoff 1 m, 5 m, 30 m, 2 h, 6 h, 12 h, 24 h, then `dead` after 8 attempts                      |
| Poison        | Dead-lettered to `outbox_dead_letters` after 5 lone failures when its neighbour goes through; never redelivered | A failing endpoint dead-letters its own deliveries; `webhooks.redeliver` retries one; `events.prune` deletes dead letters after 90 days |

The fan-out pass locks its cursor row, reads a page with `readOutbox`, inserts one `webhook_deliveries` row per
matching webhook and event, and advances the cursor, all in one transaction. A module that needs per-item retries
can follow the same pattern with `readOutbox`, `outboxTail` and `formatCursor` from `better-compute/events`. Remember
that such a cursor is invisible to retention: if you need every event, save a copy of your position in
`outbox_cursors` too, or use a sink. See [webhooks](/docs/guides/events/webhooks).

For in-process reactions, `compute.events.subscribe('reservation.*', handler)` tails the outbox from the moment you
subscribe, with an in-memory cursor. It is not durable: use a sink when you need every event.

## Audit sinks [#audit-sinks]

`defineSink({ kind: 'audit', id, write })` builds an audit sink: `write(records: AuditRecord[]): Promise<void>`
receives committed rows of `audit_events`, for a SIEM or an archive. List it in `betterCompute({ sinks: { audit } })`
or in a module's or plugin's `extensions`.

```ts title="compute.ts"
import { betterCompute, defineSink } from 'better-compute';

const siem = defineSink({
  kind: 'audit',
  id: 'siem',
  async write(records) {
    // records: { id, tenantId, time (epoch ms), action, resource: { type, id }, admission?, details? }[]
    await siemClient.send(records.map((record) => ({ key: record.id, body: record })));
  },
});

export const compute = betterCompute({
  // …database, secret, baseURL, iam
  sinks: { audit: [siem] },
});
```

When at least one audit sink is listed, `betterCompute()` registers the built-in `audit-sinks` module (right after
the `tunnels` module, before your own). It works like OutboxDispatch, with its own tables:

|           | Audit sinks                                                                                                                      |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Runs in   | The singleton reconciler `audit-sinks`, key `tick` every 5 s, the `controllers` role                                             |
| Reads     | `audit_events` after the sink's `(txid, seq)` cursor, finished transactions only (`txid < pg_snapshot_xmin(…)`), in commit order |
| Batches   | 500 records per `write`; a pass keeps going until a batch comes back short, so it drains a backlog                               |
| Cursor    | `audit_sink_cursors` (`sink_id`, `txid`, `seq`; migration `audit-sinks/0001_cursors`), saved forward only after each batch       |
| Failure   | The batch is retried after 1 s, doubling to at most 5 minutes, per sink; the other sinks go on. No bisecting and no dead letters |
| Guarantee | At least once: a batch written before its cursor was saved is written again. Deduplicate on `record.id`                          |

Ids must be non-empty and unique among the audit sinks (`INVALID_CONFIG` otherwise). A new sink starts at the origin
and receives every audit row still in the table. The `audit` module's hourly checkpoints still seal the rows in
place; a sink is a copy, not a replacement. See [audit](/docs/guides/events/audit).

## Usage [#usage]

`defineSink` also accepts `kind: 'usage'`, but a usage sink is refused: `sinks.usage` with any entry, or a usage sink
listed in a module's `extensions`, makes `betterCompute()` throw `INVALID_CONFIG`. Usage windows go to Better IAM
billing through the IAM bridge, which owns the single `recorded_at` flush. PLAN.md §5.4 sketches it as
`sinks: { usage: [bridge.usageSink()] }`; the code implements it as reconcilers:

1. `capacity.usage` (every 5 minutes) writes closed hourly usage windows of running
   allocations to `usage_windows`, each with the idempotency key `cmp:<allocationId>:<windowStartMs>`.
2. `iam.usage` in the bridge's `iam` module (every hour) sends up to 1,000 closed, unrecorded windows through
   `IamPort.recordUsage`, then stores each window's outcome.

`recordUsage` isolates poison the way an event sink should. Better IAM records a batch of up to 200 in one
transaction, so one bad event would fail the whole batch on every retry. On an error specific to the events
(`NOT_FOUND`, `CONFLICT`, `BILLING_PERIOD_CLOSED`, `INVALID_INPUT`, `INVALID_TRANSITION`, `METER_ARCHIVED`) the
batch is bisected down to the event, which gets one of these results:

| Result                  | Meaning                                                                                                                                     |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `recorded`, `duplicate` | Recorded now, or already recorded under the same key                                                                                        |
| `reattributed`          | Its identity no longer exists: recorded under the offboarded identity's successor, or tenant-attributed with the tag `attribution=orphaned` |
| `dead-lettered`         | Still fails on its own; stored in `usage_dead_letters` with the error code and message                                                      |

Any other error is transient and rejects the whole call; the windows stay unrecorded and the next pass retries them
with the same keys. See [billing and usage](/docs/guides/iam/billing-and-usage).

## Checklist for your own sink [#checklist-for-your-own-sink]

1. Choose a stable `id`; it names the cursor.
2. Make `deliver` idempotent on `event.id`: every event can arrive twice.
3. Throw only for transient failures; record permanent ones yourself, or leave them to `outbox_dead_letters`.
4. Keep `deliver` within seconds: the sink's next batch waits for it (other sinks run in their own loops).
5. Expect a new sink to start with everything the outbox still holds.
6. When you retire a sink, delete its `sink:<id>` row from `outbox_cursors`.
7. Run at least one worker with the `sinks` role (`compute.start()` runs every role by default).


# Steps and profiles (/docs/extending/steps-and-profiles)

> Write workflow steps with defineStep, ship profiles, use the step context's PKI and secret ports, and keep steps idempotent across crashes and wake-ups.



The provision engine runs a profile, a versioned document, as a
workflow of persisted steps for one node or instance. Metal's whole
lifecycle is built this way: enrolling, inspecting, commissioning, deploying, releasing and retiring are each a profile
that metal ships. You extend the engine with your own steps and profiles, and around them with user-data templates,
config units and image signers, without editing `@better-compute/*`.

How workflows run, fail, retry and are aborted is on [provisioning workflows](/docs/guides/bare-metal/workflows); this
page is about writing the pieces.

> **Where this runs today.** 
  The engine, `defineStep`, `defineProfile` and every metal step run in the control plane and are tested on PGlite,
  including the Ubuntu and Talos walkthroughs. Steps that send commands wait for agents: `bcd site` and `bcd ramdisk`
  are written but not released and tested only by Go unit tests, so in the TypeScript tests simulated agents (or
  FakeDaemon site agents) answer the commands.

## Three kinds of step [#three-kinds-of-step]

| Kind          | Declared with                                      | Runs as                                                                               | Example                               |
| ------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------- |
| control-plane | `runsOn: 'control-plane'`, `run`                   | `run(input, ctx)` in the worker that owns the target's shard, outside any transaction | `userdata.render`, `wait.agentJoined` |
| relaying      | `runsOn: 'control-plane'`, `run`, `relays`         | the same, but `run` builds a command and sends it to an agent with `awaitCommand`     | `power.setPower`, `media.insert`      |
| agent         | `runsOn: 'site' \| 'node' \| 'ramdisk'`, `command` | exactly one command of kind `command`, whose payload is the step's input              | `diag.dcgm`, `boot.kexec`             |

Relaying steps exist because the command often needs data only the control plane has: a BMC address and credential
reference, a resolved catalog image, a freshly minted token. `relays: ['site']` lets profiles still write
`{ use: 'power.setPower', on: 'site' }`.

## defineStep [#definestep]

```ts title="steps/render-iso.ts"
import { createHash } from 'node:crypto';
import { manifest } from 'better-compute/core';
import { defineStep, renderTemplate } from 'better-compute/provision';

export const renderAutoinstallIso = defineStep({
  id: 'autoinstall.renderIso',
  runsOn: 'control-plane',
  timeout: '5m',
  retrySafe: false,
  input: manifest({}),
  async run(_input, ctx) {
    const node = ctx.node!;
    const { token } = await ctx.sql.transaction((tx) =>
      ctx.pki!.issueJoinToken(tx, { tenantId: 'root', role: 'provisioned', siteId: node.siteId, ttlSeconds: 7200, maxUses: 1 }),
    );
    const userData = renderTemplate(autoinstallTemplate, { hostname: node.id.replace(/_/g, '-'), joinToken: token });
    const digest = createHash('sha256').update(userData).digest('hex');
    await publishIso(digest, userData); // your ISO build and storage behind the site cache
    return { url: `https://cache.site-a/autoinstall/${digest}.iso` };
  },
});
```

This is the step of [walkthrough 1](/docs/extending/walkthroughs). `defineStep` checks the shape up front and returns
the step frozen with `extension: 'step'`; every refusal is `INVALID_CONFIG` with `details.path`.

<TypeTable
  type="{
  id: { type: 'string', description: 'What profiles use: dotted, lower-case first segment, at most 128 characters, e.g. power.setPower or acme.renderIso.', required: true },
  version: { type: 'string', description: 'Optional version string.' },
  runsOn: { type: &#x22;'control-plane' | 'site' | 'node' | 'ramdisk'&#x22;, description: 'Where it runs.', required: true },
  timeout: { type: 'Duration', description: &#x22;One attempt's budget: '30s', '10m', '1h' or milliseconds; must be positive. A profile step may override it.&#x22;, required: true },
  retries: { type: 'number', description: 'Extra attempts after a failure, 0 to 100. A profile step may override it.', default: '0' },
  retrySafe: { type: 'boolean', description: 'Whether running it twice does no harm. Required; see idempotency below.', required: true },
  input: { type: 'Manifest<I> | StandardSchema', description: &#x22;Validates the profile step's with when the profile is expanded, and again before run.&#x22;, required: true },
  run: { type: '(input, ctx) => Promise<Json | void>', description: 'Control-plane steps only (required there). The result is stored as the step result.' },
  relays: { type: &#x22;('site' | 'node' | 'ramdisk')[]&#x22;, description: 'Control-plane steps only: the places a profile may name with on:.' },
  sends: { type: 'string[]', description: &#x22;Control-plane steps only: every dotted command kind run may enqueue. The workflow engine then refuses any other kind with INVALID_CONFIG, and records the kinds of every step when the workflow starts (workflows.command_kinds), so the owner can scope the workflow's v1.0 destructive-batch before anything is sent. A step without it is not checked, and in a batch whatever it sends falls outside the scope and is refused. Required in practice for any relaying plugin step: see relaying under the quorum below. Every relaying metal step and health.resetGpus declare theirs.&#x22; },
  command: { type: 'string', description: 'Agent steps only (required there): the dotted command kind sent.' },
  await: { type: &#x22;'rejoin'&#x22;, description: 'Agent steps only: complete when the agent comes back with a new boot or role, for self-terminating kinds.' },
  cleanup: { type: 'ProfileStep[]', description: &#x22;Steps that run, last in first out, in the cleanup phase if this step started.&#x22; },
}"
/>

An agent step's `command` must be a command kind some module registers (`ComputeModule.commandKinds`, see
[modules](/docs/extending/modules) and [command kinds](/docs/reference/command-kinds)); the server's command queue
refuses others with `INVALID_CONFIG`. Its payload is the step's validated input as the kind's proto3 JSON. The agent
has to implement the kind, which for a new kind means a change to `bcd`.

## The step context [#the-step-context]

A control-plane step's `run` receives a `StepContext`: the whole `ModuleContext`, plus the workflow's coordinates.

<TypeTable
  type="{
  workflowId: { type: 'string', description: 'The workflow.' },
  stepId: { type: 'string', description: &#x22;The step's id within the workflow: main.3, main.3.on-failure, cleanup.1.&#x22; },
  attempt: { type: 'number', description: '0 on the first attempt, then 1, 2, ... on retries.' },
  idempotencyKey: { type: 'string', description: 'wf:<workflowId>:<stepId>, the same on every run of the step.' },
  batchId: { type: 'string', description: &#x22;(v1.0) The destructive-command batch the workflow runs under, when it has one. awaitCommand puts it on the step's destructive commands only.&#x22; },
  node: { type: '{ id, siteId, poolId? }', description: 'The target node, for node workflows.' },
  signal: { type: 'AbortSignal', description: &#x22;Aborted at the step's deadline and when the worker stops.&#x22; },
  emit: { type: '(progress) => void', description: 'Accepted but discarded for control-plane steps today.' },
  sql: { type: 'SqlExecutor', description: 'Queries, and transaction(fn) for writes. No transaction is open while run executes.' },
  commands: { type: 'CommandQueue', description: 'enqueue(tx, command); prefer awaitCommand, which adds the keys and the waiting.' },
  pki: { type: 'PkiPort', description: 'issueJoinToken(tx, { tenantId, role, poolId?, siteId?, ttlSeconds, maxUses?, expect? }) → { id, token }. Optional.' },
  secrets: { type: 'SecretBroker', description: 'put(tenantId, value) → SecretRef, resolve(ref), delete?(ref). Optional.' },
  'clock, events, work, log, …': { type: 'ModuleContext', description: 'Everything a module gets; see modules.' },
}"
/>

`pki` and `secrets` are optional on `ModuleContext`. A step that needs them should fail with `INVALID_CONFIG` when
they are missing, as `userdata.render` does, rather than crash on `undefined`.

* **`ctx.pki.issueJoinToken`** mints the credential an agent your step brings up will join with. Roles are `node`,
  `site`, `provisioned` and `discovery`. For anything a machine uses to join, mint `maxUses: 1`, a short `ttlSeconds`,
  and `expect` (`mac`, `productUuid`, `serial` or `gpuUuid`) when you know the machine, so a join with other hardware
  claims is refused without spending the token.
* **`ctx.secrets.put`** stores a value and returns a `SecretRef`. Put tokens and rendered documents there and send only
  the reference. An agent can fetch a reference's value only when a live command of this workflow step for that agent
  (key `wf:<workflow>:<step>`, with any `#n` or `:key` suffix) carries the reference in its body, so the reference must
  travel in the step's own command.

Never return a secret from `run`. The result is stored in `workflow_steps.result` and readable through
[`workflows.get`](/docs/reference/api/workflows#get); `userdata.render` returns `{ userDataRef, kind, joinTokenId }`,
not the document.

## Idempotency and wake-ups [#idempotency-and-wake-ups]

A control-plane step is marked `running` in one transaction, runs outside any transaction, and is settled in another
that first checks it is still `running` with the same attempt. A worker that dies in between leaves the step
`running`, and the next owner of the shard runs it again **under the same idempotency key**. The engine does not read
`retrySafe` today, so every `run` must be safe to repeat. The pattern is find before create:

```ts title="steps/acme-reserve-ip.ts"
async run(input, ctx) {
  return ctx.sql.transaction(async (tx) => {
    const { rows } = await tx.query('SELECT ip FROM plg_acme_ips WHERE step_key = $1', [ctx.idempotencyKey]);
    if (rows[0]) return rows[0]; // a re-run after a crash finds the first run's work
    const ip = await nextFreeIp(tx); // your allocator, over your plugin's own tables
    await tx.query('INSERT INTO plg_acme_ips (step_key, ip) VALUES ($1, $2)', [ctx.idempotencyKey, ip]);
    return { ip };
  });
}
```

Key per attempt, not per step, when a retry must start over: `stepCommandKey(ctx, key)` is
`wf:<id>:<step>#<attempt>[:<key>]`.

### Waiting [#waiting]

A step that cannot finish yet throws `stepNotReady(ms)`. The engine settles nothing, and runs the step again after
`ms` (5,000 by default), never later than its deadline; at the deadline the step fails `TIMEOUT`. The built-in waits
are written this way:

```ts
async run(input, ctx) {
  const { rows } = await ctx.sql.query('SELECT generation, observed_generation FROM nodes WHERE id = $1', [ctx.node!.id]);
  if (Number(rows[0].observed_generation) < Number(rows[0].generation)) throw stepNotReady(5_000);
  return { generation: Number(rows[0].generation) };
}
```

### Sending commands from a step [#sending-commands-from-a-step]

`awaitCommand(ctx, { nodeId, kind, payload, key?, timeoutMs? })` is find-before-create for commands. It looks the
command up by `wf:<id>:<step>#<attempt>[:<key>]` on that node, enqueues it only if absent (with `notAfter` now plus
`timeoutMs`, 10 minutes by default), and then:

* throws `stepNotReady(2000)` while the command is `pending`, `dispatched`, `acked` or `running`;
* returns `{ commandId, result, attested }` once it `succeeded`, with the agent's raw result bytes;
* throws `DRIVER_ERROR` with the state and reason once it `failed`, was `rejected`, `expired` or `cancelled`.

Give each command of one step its own `key` (`'boot-device'`, `'power-cycle'`): `metal.ensureRamdisk` sends a
one-time PXE boot, a boot entry per MAC and a power cycle in sequence, and a re-run skips the ones already done
because it finds them by key. To reach a node's site
agent, `siteAgentFor(ctx.sql, nodeId)` returns a connected agent of the node's site, the boot-lease holder first, or
`undefined`, in which case throw `stepNotReady` and wait for one. Metal's `httpViaSite(ctx, request)` is a relaying
helper built the same way: one HTTP request from the site agent to the machine's own address, with SSRF checks; see
[walkthroughs](/docs/extending/walkthroughs).

A command's result is not trusted just because it arrived. A result that decides a transition, like a wipe, should be
node-signed and verified: metal's steps check the stored signature against the node's certificates before they
accept `attested`.

### Relaying under the quorum [#relaying-under-the-quorum]

With the v1.0 [destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) on, a destructive
command runs only in a batch its approvers signed, and the batch's scope is fixed before the workflow sends
anything. The engine records every kind a workflow can send when it starts (`workflows.command_kinds`: each agent
step's `command`, and each control-plane step's `sends`), metal creates the workflow's batch from those kinds
(f66e236), and `awaitCommand` and the engine's agent steps put `ctx.batchId` on the destructive commands only. So a
relaying step must declare what it sends, or the kind is missing from the scope:

```ts title="acme-talos.ts"
const talosReady = defineStep({
  id: 'wait.talosReady',
  runsOn: 'control-plane',
  relays: ['site'],
  sends: ['site.httpProxy'], // httpViaSite enqueues site.httpProxy, a destructive kind
  timeout: '20m',
  retrySafe: true,
  input: manifest({ port: field.int({ min: 1, max: 65535 }).default(50000) }),
  async run(input, ctx) {
    const response = await httpViaSite(ctx, { url: `https://${address}:${input.port}/machine/version`, poll: true });
    // ...
  },
});
```

Without `sends`, the step's `site.httpProxy` carries the workflow's batch but is not in its scope, and with approver
keys set the queue refuses it at enqueue (`INVALID_INPUT`; or `INVALID_STATE`, reason `batch-required`, when the
workflow sends nothing else destructive and so got no batch). With `sends`, enqueueing any undeclared kind is
`INVALID_CONFIG`. `defineStep` refuses `sends` on an agent step, whose `command` already names its kind. See
[who creates batches](/docs/operations/pki#who-creates-batches) and, for batches over many nodes,
[`metal.bulk`](/docs/reference/api/metal).

### What wakes a workflow [#what-wakes-a-workflow]

Nothing depends on a wake-up arriving. The `workflow` reconciler is keyed by the node id (or the workflow id for
workflows without a node), and after each pass it queues itself in the work queue with `notBefore` set to the next
moment it cares about:

* an agent step waiting on its command: in 10 seconds (`workflowPollMs`), or at the step's deadline if sooner;
* a control-plane step that threw `stepNotReady(ms)`: in `ms`, or at the deadline if sooner (a relaying step's
  `awaitCommand` uses 2 seconds);
* a verb (`abort`, `retry`, `resume`) or a new workflow: at once.

When a workflow ends, its `owner` hears about it. `ctx.workflows.start(tx, { …, owner })` names a reconciler that is
queued under the workflow's key when it finishes. Metal starts every workflow with `owner: 'metal'`, so the node moves
on without polling.

## Profiles [#profiles]

```ts title="profiles/ubuntu-autoinstall.ts"
import { defineProfile } from 'better-compute/provision';

export const ubuntuAutoinstall = defineProfile({
  id: 'ubuntu-24.04-autoinstall',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'autoinstall.renderIso' },
    { use: 'media.insert', on: 'site', with: { fromStep: 'autoinstall.renderIso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.agentJoined', with: { role: 'node' } },
    { use: 'media.eject', on: 'site' },
  ],
});
```

`defineProfile` rejects unknown fields, malformed steps, ids and durations, non-JSON values, fields of the other
target, and an image, user data or steps on an `adopt` profile; it returns a deeply frozen copy. The fields are listed
on [profiles and images](/docs/guides/bare-metal/profiles-and-images).

Steps pass data forward through their results. `media.insert` with `fromStep` reads the `url` the latest succeeded
step with that action returned in the same workflow; `deploy.deploy` finds `userdata.render`'s reference the same way
(`renderedUserData(sql, workflowId)`). Read earlier results from `workflow_steps` by `workflow_id` and `action`, never
from memory.

When a workflow starts, each `use:` resolves once and the result is persisted:

1. a registered step with that id (an `on:` must match its `runsOn` or one of its `relays`);
2. otherwise, with `on:` naming an agent, a driver operation `<concern>.<operation>` sent as one `driver.invoke`
   command (see [drivers](/docs/extending/drivers#from-a-profile-step-to-a-driver));
3. otherwise `fallback:`, resolved the same way.

`with` is validated against the step's `input` at the same moment. `profiles.validate` reports every unresolved `use`
and rejected `with` with its path, and `profiles.render` returns the expanded steps; run both before the first deploy.
A version becomes immutable the first time a workflow uses it, and a workflow holds at most 512 steps.

## Shipping steps and profiles [#shipping-steps-and-profiles]

The engine resolves only what `provisionModule` knows. There are two ways to tell it:

```ts title="compute.ts"
import { definePlugin } from 'better-compute/core';
import { metal } from 'better-compute/metal';
import { provisionModule } from 'better-compute/provision';

// 1. Directly: provision registers them as its own.
const provision = provisionModule({ steps: [talosReady], profiles: [talos] });

// 2. As a plugin: the engine borrows its steps and profiles, the plugin registers them itself.
const acmeAutoinstall = definePlugin({ id: 'acme-autoinstall', extensions: [renderAutoinstallIso], profiles: [ubuntuAutoinstall] });
const bareMetal = metal();
const provisionWithPlugin = provisionModule({ modules: [bareMetal, acmeAutoinstall] });
```

With the second form, register the plugin with the server as well (`betterCompute({ plugins: [acmeAutoinstall] })`),
the way metal is listed both in `provisionModule({ modules })` and in `modules`: provision registers only its own
steps and profiles, and the registry refuses an extension registered twice. Shipped profiles are read-only, visible to
every tenant, and listed under the tenant `builtin`; tenants can still store their own with `profiles.create`, and a
lookup prefers the tenant's own profile, then a public stored one, then a shipped one. Step ids share one namespace:
a duplicate is `INVALID_CONFIG`, so prefix yours (`acme.`, `autoinstall.`).

## User-data templates [#user-data-templates]

`userdata.render` renders the pinned profile's `userData: { kind, template }` for the workflow's node with a fresh
single-use `provisioned` token (2 hours, or `tokenTtlSeconds`), stores the document through `ctx.secrets`, and returns
only the reference. Add templates by name:

```ts
provisionModule({
  userData: {
    controlPlaneUrl: 'https://gw.eu-1.example.com',
    bcdUrl: 'https://downloads.example.com/bcd/0.2.0/bcd-linux-amd64',
    bcdSha256: '…',
    templates: { 'acme-talos.yaml': acmeTalosTemplate }, // kind 'talos' ships no template
  },
});
```

A template is a string with `{{name}}` placeholders or a function of the variables: `nodeId`, `hostname` (the node id
with `_` as `-`), `joinToken`, `controlPlaneUrl`, `bcdUrl` and `bcdSha256`. `renderTemplate` refuses a value with a line
break or `{{` (`INVALID_INPUT`) and a placeholder with no value (`INVALID_CONFIG`), so a value can never add lines to
the document. Build structured formats as data and serialize them, as the built-in Ignition template does, rather
than splicing strings. `provisionedTargetResolver()` (for the `provisioned` attestor) maps the token back to the node
through the step result's `joinTokenId`, so the installed OS re-keys the same node.

## Config units [#config-units]

A profile's `hostConfig` is not a step: when the workflow succeeds, its units become the node's NodeSpec `host`
section, which bcd's host converger applies (see [host configuration](/docs/daemon/host-config)). A unit is
`{ name, description, settings }`, where `settings` is a manifest:

```ts
import { field, manifest } from 'better-compute/core';

provisionModule({
  configUnits: [
    {
      name: 'acme-nic-firmware',
      description: 'Pins the NIC firmware branch',
      settings: manifest({ branch: field.string({ pattern: '^[0-9]+\\.[0-9]+$' }) }),
    },
  ],
});
```

Any unit takes `when:`, a label expression that turns it on per node. An unknown unit or invalid settings is
`INVALID_INPUT` when the host section is resolved. The built-ins are `kernel-modules`, `sysctl`, `hugepages`,
`nvidia-driver`, `fabric-manager`, `mig-layout`, `gpu-sharing`, `container-toolkit` and `k8s-join`; a unit you add with
the name of a built-in replaces it. A new unit is data to the control plane only: bcd applies the units it was built
with, and reports any other as `unit <name> is not supported by this daemon`, which fails the `host` section
terminally once the known units have had their turn.

## Images [#images]

Steps that deploy an image resolve it from the catalog in the machine owner's tenant with
`resolveImageRef(tx, tenantId, 'name@version')`, which prefers the tenant's own image, then a public one. Images are
registered by digest and verified against allowlisted signers (Ed25519, or ECDSA P-256 over SHA-256);
`provisionModule({ imageVerifier })` replaces the check. `ImageSource`, a core contract for resolving other schemes,
is declared but nothing uses it yet. See [profiles and images](/docs/guides/bare-metal/profiles-and-images#the-image-catalog).

## Testing [#testing]

* **Shapes.** `tests/provision-define.test.ts` covers `defineStep` and `defineProfile` refusals; assert yours throw
  `INVALID_CONFIG` with the `details.path` you expect.
* **The engine with a stub queue.** `tests/extensions/talos.test.ts` runs the real provision module and metal on
  PGlite with an in-memory `CommandQueue` and `WorkQueue`, answers commands from a simulated machine, and drives the
  reconcilers until nothing is left. Copy its `settle()` loop.
* **A simulated site.** `createMetalSim({ modules, steps, profiles, install })` in `tests/extensions/metal-sim.ts` adds
  a site agent, per-machine BMC state and node-signed results; `ubuntu-autoinstall.test.ts` runs 20 deploy and
  release loops on it.
* **Crashes.** `tests/metal-chaos.test.ts` kills the worker at every point of a deploy and checks that no destructive
  command is sent twice. Do the same for a step with side effects.

  - [Provisioning workflows](/docs/guides/bare-metal/workflows): Phases, states, retries, abort and the workflows API.

  - [Profiles and images](/docs/guides/bare-metal/profiles-and-images): Profile fields, the shipped profiles and the image catalog.

  - [Walkthroughs](/docs/extending/walkthroughs): Plugins that add steps and profiles, end to end.


# Walkthroughs (/docs/extending/walkthroughs)

> The acceptance walkthroughs of PLAN.md 5.6 and the emulated-site example, what each one proves about extending Better Compute, and how to run them.



PLAN.md §5.6 sets the bar for the extension points: each one is proven by a walkthrough that adds something real to
Better Compute &#x2A;*with zero edits to `@better-compute/*`**. A walkthrough is a third-party plugin, written in a test the
way an outside package would write it, plus the test that shows the platform picks it up and runs it. They double as
the best worked examples of the contracts on the other pages of this section.

| Walkthrough                                     | Phase | Extension points                                     | Where                                                              | State                                                                        |
| ----------------------------------------------- | ----- | ---------------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| 1. Ubuntu autoinstall via Redfish virtual media | v0.2  | a control-plane step, a profile, a plugin            | `tests/extensions/ubuntu-autoinstall.test.ts`                      | runs in simulation                                                           |
| 2. VPS provider `acme-cloud`                    | v0.3  | `defineProvider`, conformance cases, a provider pool | `tests/extensions/acme-cloud.test.ts`                              | runs against an in-memory cloud emulator, live and from a recording          |
| 3. Talos on an agentless pool                   | v0.3  | a relaying step, a profile, `site.httpProxy`         | `tests/extensions/talos.test.ts`, `tests/metal-fakedaemon.test.ts` | runs in simulation and on the dev stack                                      |
| 4. Kata runtime in Go                           | v0.5  | a bcx `runtime` plugin in bcd's `[plugins] dir`      | `daemon/internal/agent/pluginruntime_test.go`                      | bcd serves runtimes from bcx plugins (Go unit tests); no Kata plugin written |
| Emulated site                                   | v0.2  | a hardware profile, the whole metal lifecycle        | `examples/emulated-site`                                           | configuration checked; needs Linux, KVM and a `bcd` build to run             |

> **No real hardware yet.** 
  Every metal walkthrough needs `bcd site`, and all but the agentless Talos one need `bcd ramdisk`. Both modes of the Go
  daemon are now written and answer every command these walkthroughs send, BIOS, RAID and firmware updates through
  Redfish included (944221c), but they are not released and only their Go unit tests run them. No walkthrough runs against them
  yet: the tests answer the commands with simulated site agents and machines, or with a FakeDaemon site agent over
  the real gateway. Running one for real also needs what the repository does not provide (iPXE builds, a ramdisk
  image, a KVM host). PLAN.md's pass criterion for walkthrough 1, "a sushy-tools VM reaches `deployed`", has not
  been met. See [what bcd does today](/docs/daemon#what-bcd-does-today).

Run them all from the repository root:

```bash
pnpm vitest run tests/extensions tests/metal-fakedaemon.test.ts tests/examples-smoke.test.ts
```

## Walkthrough 1: Ubuntu autoinstall [#walkthrough-1-ubuntu-autoinstall]

**The plugin.** `acme-autoinstall` installs Ubuntu with its own installer from a per-machine ISO mounted through the
BMC, instead of streaming an image from a ramdisk. It is three definitions:

```ts
const renderAutoinstallIso = defineStep({ id: 'autoinstall.renderIso', runsOn: 'control-plane', /* … */ });
const ubuntuAutoinstall = defineProfile({ id: 'ubuntu-24.04-autoinstall', version: 1, target: 'metal', role: 'host', /* … */ });
const acmeAutoinstall = definePlugin({ id: 'acme-autoinstall', extensions: [renderAutoinstallIso], profiles: [ubuntuAutoinstall] });
```

* `autoinstall.renderIso` mints a single-use `provisioned` join token through `ctx.pki`, renders it into an
  autoinstall `late-commands` entry with `renderTemplate`, stores the ISO under its sha256, and returns
  `{ url }`, an unguessable per-node URL behind the site cache.
* The profile chains built-in metal steps around it: `media.insert` on the site agent with
  `fromStep: 'autoinstall.renderIso'`, `management.setBootDevice` to `cd`, `power.setPower` `cycle`,
  `wait.agentJoined` for role `node`, and `media.eject`.

Both are written out in full on [steps and profiles](/docs/extending/steps-and-profiles#definestep) and
[deploying](/docs/guides/bare-metal/deploying#walkthrough-ubuntu-autoinstall-over-virtual-media).

**The harness.** `createMetalSim({ modules: [acmeAutoinstall], install })` in `tests/extensions/metal-sim.ts` builds a
simulated site: PGlite, the real provision engine and metal module, a site agent, and per-machine BMC state (boot
device, mounted media, power, firmware, BIOS and RAID). Booting from PXE brings up the ramdisk; booting from a CD whose
ISO the `install` hook accepts brings up the installed OS as role `node`. Results come back as each kind's binary result
message, node-signed where the kind is attested, as real agents send them.

**What it proves.**

* `profiles.validate` resolves every step of the plugin's profile: the engine found a third-party step and profile.
* One machine goes enroll, `ready`, `deployed` with role `host`, undeploy, `ready`. The ISO is ejected, and the join
  token appears inside the ISO and nowhere in any command the agents saw.
* The v0.2 acceptance of PLAN.md §16 in simulation: 20 consecutive unattended deploy and release loops, a fresh
  single-use token for each install, and every transition into `ready` preceded by exactly one attested wipe in that
  workflow (checked in SQL over `workflows` and `workflow_steps`).
* A release whose wipe is not attested never reaches `ready`: it ends `failed` from `releasing`, with maintenance on.

```bash
pnpm vitest run tests/extensions/ubuntu-autoinstall.test.ts
```

**What it does not prove.** The ISO build and hosting are stubbed (a map from digest to token), the installed OS
rejoining as the same node is simulated, and no real BMC mounts anything. A token a plugin mints itself joins through
the `provisioned` attestor only if it carries `expect: { productUuid }` or the host's resolver can trace it; see
[attestors](/docs/extending/attestors).

## Walkthrough 3: Talos on an agentless pool [#walkthrough-3-talos-on-an-agentless-pool]

Talos manages itself through its own API and never runs `bcd`. The plugin is one step and one profile:

* `wait.talosReady` is a control-plane step with `relays: ['site']` and `sends: ['site.httpProxy']`, the kind it
  relays. `site.httpProxy` is destructive (0d42de8), so the declaration is what puts it in the workflow's quorum
  batch: without it, a deployment with `pki.approvers` set refuses the step's commands at enqueue (see
  [relaying under the quorum](/docs/extending/steps-and-profiles#relaying-under-the-quorum)). It reads the machine's address from
  `metal_nodes.addresses` (recorded by Redfish inspection), and calls `httpViaSite(ctx, { url, tlsSpkiSha256, poll: true })`
  on `https://<address>:50000/machine/version`. Until Talos answers 200, it throws `stepNotReady`.
* `talos-1.8` mounts the Talos ISO with `media.insert`, sets boot to `cd`, power-cycles, and waits in
  `wait.talosReady`.

The pool opts in to what an agentless machine cannot do: `agent: 'none'`, `policy.allowUnverifiedRelease: true`, and
`policy.provisioningCidrs` for plain-HTTP targets.

`httpViaSite` is the extension point this walkthrough added to metal: a control-plane step reaches a machine only
through its site agent's `site.httpProxy` command, and the control plane refuses anything else before a command
exists. The request takes `url`, `method` (`GET`, `POST`, `PUT`, `DELETE`), `headers`, `secretHeaders` (secret
references only), `body`, `tlsSpkiSha256`, `timeoutMs`, `key` and `poll`. With `poll`, each call returns the newest
settled answer and sends the next request, which is what a readiness loop needs.

**What it proves** (`tests/extensions/talos.test.ts`, on PGlite with a stub command queue):

* Enrollment and commissioning run out of band (the agentless variants of metal's profiles); the recorded addresses
  come from Redfish.
* The deploy polls the Talos API until it answers, always at the machine's own address, with the pinned SPKI, and the
  node is `deployed` with role `host`. Every command went to the site agent, never to the machine.
* The node is never `connected`, so it is never schedulable for `hold` reservations.
* Release reaches `ready` without an attested wipe only because the pool opted in, and says so with
  `node.release.unverified`. Rescue fails the node with `UNSUPPORTED`.
* `site.httpProxy` is SSRF-bounded: other hosts, names, credentials in URLs, plain HTTP off the provisioning network,
  forbidden headers and header line breaks are refused; a request for another node's IP creates no command; a
  redirect or an oversize response is `DRIVER_ERROR`; secret headers travel by reference.

`tests/metal-fakedaemon.test.ts` runs the same Talos plugin on the dev stack: the real control plane with provision
and metal, the real gateway, JoinService, and a site agent that is a FakeDaemon of role `site`, joined with a site
token (`tests/metal-dev-site.test.ts`), in front of simulated BMCs. Every command crosses the gateway and back, and
the real server's batch check runs (f66e236). Both tests declare `sends` on the Talos step.

```bash
pnpm vitest run tests/extensions/talos.test.ts tests/metal-fakedaemon.test.ts
```

## Walkthrough 2: a VPS provider [#walkthrough-2-a-vps-provider]

PLAN.md describes it as `defineProvider(…)`, `providerConformance(acmeCloud, { recordReplay })`, a provider account, and
a `class: 'provider'` pool with `agent: 'none'` whose instances run through the generic provider steps.
`tests/extensions/acme-cloud.test.ts` runs it as a third party would, with zero edits to `@better-compute/*`:

* **The plugin.** `tests/extensions/acme-cloud/provider.ts` is a `defineProvider` over Acme Cloud's HTTP API with an
  injectable `fetch`; `cloud.ts` is an in-memory emulator of that API, and `record-replay.ts` a transport that records
  every exchange and replays it.
* **Conformance, live and offline.** `runConformance(providerConformanceCases(), subject)` passes against the emulator
  through the recording transport, leaving no server behind, and passes again offline from the recording with every
  exchange consumed. A wrong account token is a configuration error, not a provider outage.
* **On a real server.** With `vps({ providers: [acmeCloudProvider({ fetch })] })` on a compute fixture: a provider
  account whose token is sealed in the secret store, a `class: 'provider'` pool (its `agent` is `none`) offered to a
  tenant with a quota, and a flavor mapped onto the provider's offer. The tenant's instance is created once (a
  retried reconciler pass creates nothing new), with the SSH key and no join token in its user data, followed to
  `running`, stopped (the machine powered off) and deleted; a third instance over the tenant's quota is refused, and
  the pool never gets a node.

The conformance API is `providerConformanceCases()` with `runConformance`, not a single `providerConformance`
function, and recording is left to the transport: adapters take a `fetch`. Instances of provider pools are driven by
the `vps.instances` reconciler calling the account's client, not by workflow steps. `tests/vps-provider.test.ts`
(manifest-typed accounts and shape refusals) and the Hetzner Cloud, Hetzner Robot and Proxmox suites cover the same
contract piecewise. [Providers](/docs/extending/providers) shows how to write and test one.

```bash
pnpm vitest run tests/extensions/acme-cloud.test.ts
```

## Walkthrough 4: a Kata runtime over bcx [#walkthrough-4-a-kata-runtime-over-bcx]

A `runtime` driver written in Go, pinned in bcd's `[plugins] dir`, advertised in `Hello.drivers`, and selected by pools
through `vps({ runtimes })`. The TypeScript half of bcx is in `@better-compute/driver-sdk`, with a real-subprocess test
suite. bcd hosts pinned plugins and advertises their drivers, and `bcx.Serve` in `daemon/internal/bcx` is the start
of a Go plugin SDK.

On `bcd node`, a hosted driver whose manifest lists the kind `runtime`, and whose id is not a runtime built into bcd,
becomes an instances runtime (`agent.PluginRuntime`, 494de69). bcd speaks the TypeScript `RuntimeDriver` contract to
it over `Invoke`, with positional JSON arguments:

| Operation | Arguments                                          | Answers                                                                   |
| --------- | -------------------------------------------------- | ------------------------------------------------------------------------- |
| `list`    | none                                               | `[{ id, state, addresses, message? }]`                                    |
| `apply`   | core's `InstanceSpec` as JSON, with `desiredState` | `{ id, state, addresses, message? }`; `state: 'error'` fails the instance |
| `remove`  | the instance id                                    | `null`                                                                    |

`apply` converges one instance, desired state included, so bcd records each spec and applies it when the state
changes. A new instance's first `apply` carries its user data and, for a restore, the snapshot bcd already downloaded
and verified, as a `file://` URL; later ones do not. Snapshots answer `unsupported`, since they are not in the bcx
contract yet. A plugin never shadows a built-in runtime of the same name, and `Hello` lists its manifest once. Like
every plugin it runs [sandboxed](/docs/extending/drivers#the-plugin-sandbox): under its own uid by default, with a
seccomp filter that refuses mounts, namespaces, `bpf` and module loading, so a runtime that needs them (Kata) must be
named in `[plugins] privileged`, which runs it as root without the filter. Go
unit tests (`pluginruntime_test.go`) cover it; no Kata plugin exists yet. The control-plane half is in place: the gateway writes each driver a daemon advertises in
`Hello.drivers` as the node label `bc.dev/driver.<id>=<version>`, and vps places an instance, whatever its runtime (bcd's
built-ins too since 0fca413), only on nodes carrying `bc.dev/driver.<runtime>` (`runtimeNodeLabels`, through
`PlacementRequest.nodeLabels`; see [placement](/docs/guides/capacity/placement#eligibility-filters)). See
[drivers](/docs/extending/drivers#driver-plugins-in-bcd).

## The emulated site [#the-emulated-site]

`examples/emulated-site` is the metal walkthrough without a rack: QEMU/KVM virtual machines stand in for servers,
`sushy-emulator` gives each a Redfish BMC over libvirt, and a `bcd site` agent on the libvirt bridge runs proxy DHCP,
iPXE and the image cache. Its `better-compute.config.ts` uses the shared example deployment (`examples/shared/compose.ts`,
every module) and adds one hardware profile:

```ts title="examples/emulated-site/better-compute.config.ts"
export const hardwareProfiles = {
  'emulated-redfish': {
    description: 'QEMU/KVM machines behind sushy-tools (Redfish), PXE-booted by the site agent',
    drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
  },
};
```

A pool that names it in `policy.hardwareProfile` drives the emulated BMCs through commission, deploy, release and
retire. The run needs Linux with `/dev/kvm`, QEMU, libvirt, sushy-tools and a `bcd` build, plus iPXE builds for the
site agent's TFTP root and a ramdisk kernel and initramfs with `bcd ramdisk` in it, so none of it runs in CI. One
`sushy-emulator` serves both VMs, so each enrollment names its system in the BMC address
(`https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>`); bcd's Redfish client then drives that
system and its own manager (see [Redfish in bcd site](/docs/guides/bare-metal/bmc#redfish-in-bcd-site)).
What CI checks, in `tests/examples-smoke.test.ts`, is that the configuration loads, migrates on PGlite and passes
`better-compute doctor` with no error findings (on an unconfigured machine it reports `iam.not-configured`).

The step-by-step run, from `virt-install` to `metal.undeploy`, is on
[deploying](/docs/guides/bare-metal/deploying#walkthrough-an-emulated-site). Two details there supersede the example's
README: site agents join with a site token from `siteTokens.create`, and after `metal.enroll` the reconciler carries a
machine to `ready` on its own, so `metal.manage`, `metal.inspect` and `metal.provide` are not needed.

## Writing your own [#writing-your-own]

A walkthrough for your extension follows the same recipe:

1. Write the plugin in the test file, importing only public entry points, the way an outside package would.
2. Register it the way a deployment would (`provisionModule({ modules })`, `vps({ providers })`,
   `betterCompute({ attestors })`), not by reaching into internals.
3. Assert the platform's view first: `profiles.validate` returns no issues, `drivers.list` shows the driver, the
   conformance report is empty.
4. Drive the lifecycle end to end on a simulated site or the dev stack, and assert on states, events and the commands
   agents saw, including what must **not** appear (a token in a command body, a destructive command sent twice).
5. Add the failure cases: an unattested result, a crash mid-step (`tests/metal-chaos.test.ts` kills the worker at every
   point of a deploy), a refused request.

  - [Deploying machines](/docs/guides/bare-metal/deploying): The same walkthroughs from the operator's side.

  - [Steps and profiles](/docs/extending/steps-and-profiles): defineStep, profiles, the step context and idempotency.

  - [Providers](/docs/extending/providers): Adapters, idempotent create and conformance.

  - [Drivers](/docs/extending/drivers): Driver kinds, bindings and bcx plugins.


# Fleet configuration (/docs/guides/fleet-config)

> Keep sites, provider accounts, pools, quotas, offers, flavors and sandbox templates in one reviewed FleetConfig document; plan, apply, catch drift.



The shape of a platform changes rarely and matters a lot: which sites exist, which pools they hold, how much of each
pool its owner keeps, which tenants each pool is offered to and with what quota, and which VM sizes are sold. Made
through one-off API calls, those changes have no review, staging and production slowly diverge, and nobody notices
when someone edits a pool by hand during an incident.

FleetConfig treats that shape as code, the way Better IAM's configuration as code
treats an access model. The fleet is one document keyed by name. You export it once, keep it in version control,
review changes as pull requests, plan them, and apply them in one transaction. A reconciler re-plans the applied
document every 15 minutes and raises an event when the fleet drifts from it.

Every fleet method decides `compute:fleet:admin` on `compute-fleet/global`, a scope that exists only in the root
tenant, so FleetConfig is a tool for platform operators. `fleet.apply` also decides every call it makes, as the
caller.

## Turn it on [#turn-it-on]

```ts title="compute.ts"
import { fleetModule } from 'better-compute/fleet';
import { betterCompute } from 'better-compute/server';
import { vps } from 'better-compute/vps';

export const compute = betterCompute({
  // database, secret, iam, baseURL ...
  modules: [vps(), fleetModule()],
});
```

`fleetModule({ driftEvery })` takes one option, how often the drift check runs (default `15m`). It adds the
migrations `fleet/0001_names` (table `fleet_names`) and `fleet/0002_applied` (table `fleet_applied`), and the
`fleet.drift` reconciler. Flavors, provider accounts and sandbox templates are served by
the `vps` module; without it (or, for templates, without its v0.5 migrations), `fleet.export` leaves them out and
reports them in `missing`, and a document's flavors and templates plan as blocked.

## The document [#the-document]

A document has `version: 1` and any of five lists. References between items are names from the same document or
objects fleet already manages. Secrets are references only (`secret://`, `vault://` or `env://`), never values, so
the document is safe to commit.

```json title="fleet.json"
{
  "version": 1,
  "sites": [{ "name": "fra", "kind": "shared", "region": "eu-central" }],
  "providerAccounts": [
    {
      "name": "cloud",
      "provider": "hetzner-cloud",
      "config": { "token": "secret://hetzner", "location": "fsn1" }
    }
  ],
  "pools": [
    {
      "name": "vps-fra",
      "site": "fra",
      "class": "vps-host",
      "gpuModel": "L40S",
      "policy": { "disruptionBudget": 0.1 },
      "ownerQuota": { "gpu": 8 },
      "offers": [{ "tenant": "acme", "quota": { "vcpu": 64, "gpu": 2 }, "priority": 5 }]
    },
    { "name": "burst", "site": "fra", "class": "provider", "providerAccount": "cloud" }
  ],
  "flavors": [{ "pool": "vps-fra", "name": "g1.small", "vcpu": 2, "memoryGiB": 4, "diskGiB": 40 }],
  "sandboxTemplates": [
    {
      "name": "py-small",
      "pool": "vps-fra",
      "flavor": "g1.small",
      "imageId": "ubuntu-24.04",
      "env": { "PIP_INDEX_URL": "secret://root/sec_01k…" },
      "network": "private",
      "warm": { "min": 2, "max": 4 }
    }
  ]
}
```

  **sites:**

    <TypeTable
      type="{
  name: { type: 'string', description: 'Lowercase letters, digits, &#x22;.&#x22;, &#x22;_&#x22; and &#x22;-&#x22;, up to 63 characters.', required: true },
  kind: { type: &#x22;'shared' | 'dedicated' | 'custom'&#x22;, description: 'The site kind.', required: true },
  region: { type: 'string', description: 'Lowercase letters, digits and &#x22;-&#x22;, up to 64 characters.', required: true },
  iamRegion: { type: 'string', description: 'The IAM region, same form as region.' },
}"
    />

    See [sites](/docs/guides/bare-metal/sites).
  
  **providerAccounts:**

    <TypeTable
      type="{
  name: { type: 'string', description: 'Same form as a site name.', required: true },
  provider: { type: 'string', description: 'The provider id, such as hetzner-cloud.', required: true },
  config: { type: 'object', description: 'Provider configuration; secret fields hold references.', default: '{}' },
}"
    />

    See [providers](/docs/guides/vps/providers).
  
  **pools:**

    <TypeTable
      type="{
  name: { type: 'string', description: 'Same form as a site name.', required: true },
  site: { type: 'string', description: 'A site name.', required: true },
  class: { type: 'string', description: 'metal-host, metal-dedicated, vps-host or provider.', required: true },
  gpuModel: { type: 'string', description: 'Letters, digits and &#x22;-&#x22;, up to 32 characters.' },
  sharing: { type: 'string', description: 'exclusive, mig, timeslice or mps.' },
  agent: { type: &#x22;'bcd' | 'none'&#x22;, description: 'Whether machines run the daemon.' },
  policy: { type: 'object', description: 'The pool policy, compared as a whole.', default: '{}' },
  ownerQuota: { type: 'Record<string, number>', description: &#x22;The owner's own maximum per resource, such as { gpu: 8 }.&#x22; },
  providerAccount: { type: 'string', description: 'A provider account name, for provider pools.' },
  offers: { type: 'FleetOffer[]', description: 'Who else may use the pool, one entry per tenant.' },
}"
    />

    See [pools](/docs/guides/capacity/pools).
  
  **offers:**

    <TypeTable
      type="{
  tenant: { type: 'string', description: 'The tenant the pool is offered to. Never the owner: use ownerQuota.', required: true },
  quota: { type: 'Record<string, number>', description: &#x22;That tenant's maximum per resource, such as { gpu: 16 }.&#x22;, default: '{}' },
  includeDescendants: { type: 'boolean', description: &#x22;Offer to the tenant's child tenants too.&#x22;, default: 'false' },
  priority: { type: 'number', description: 'An integer from -1000 to 1000.', default: '0' },
}"
    />
  
  **flavors:**

    <TypeTable
      type="{
  pool: { type: 'string', description: 'A pool name.', required: true },
  name: { type: 'string', description: 'Unique within the pool.', required: true },
  vcpu: { type: 'number', description: '1 to 1024.', required: true },
  memoryGiB: { type: 'number', description: '1 to 65536.', required: true },
  diskGiB: { type: 'number', description: '1 to 1048576.', required: true },
  gpus: { type: 'number', description: '0 to 64.', default: '0' },
  offerId: { type: 'string', description: 'Sell the flavor under one offer only.' },
  placement: { type: &#x22;{ numaAffine?: boolean; pinned?: boolean; hugepages?: '2Mi' | '1Gi' }&#x22;, description: 'Where its guests live on their host; sandbox templates inherit it (2e61612). Unknown keys are refused, false flags are dropped, and the plan compares it by value.' },
}"
    />

    See [flavors](/docs/guides/vps/flavors).
  
  **sandboxTemplates:**

    <TypeTable
      type="{
  name: { type: 'string', description: 'Same form as a site name; the key fleet matches on.', required: true },
  pool: { type: 'string', description: 'A pool name in this document, or the id (pool_…) of a pool offered to the tenant. Fixed once created.', required: true },
  flavor: { type: 'string', description: 'A flavor name of that pool.', required: true },
  imageId: { type: 'string', description: 'The image the template boots.', required: true },
  runtime: { type: 'string', description: &#x22;Must be the pool's runtime. Fixed once created.&#x22;, default: &#x22;the pool's&#x22; },
  userData: { type: 'string', description: 'Cloud-init of at most 32 KiB, or a secret://<tenant>/<id> reference to it.' },
  defaultTtlSeconds: { type: 'number', description: '60 to 2592000.' },
  maxTtlSeconds: { type: 'number', description: '60 to 2592000.' },
  warm: { type: '{ min: number; max?: number }', description: '0 to 64 each.' },
  env: { type: 'Record<string, string>', description: 'Variable names (A-Z, 0-9, _) to secret://<tenant>/<id> references only, written to /etc/sandbox/env in each sandbox.' },
  network: { type: &#x22;'egress' | 'private' | 'none'&#x22;, description: &#x22;What the sandbox's NICs may reach.&#x22;, default: &#x22;'egress'&#x22; },
  volumes: { type: '{ volume: string; mountPath: string; readOnly?: boolean }[]', description: '(v1.1, 94b8b70) Existing sandbox volumes every sandbox mounts, by volume name, in attach order. mountPath is an absolute path of letters, digits and . _ / -. At most 8, the API\'s limit per template (f181f43), so a longer list fails validation with INVALID_INPUT before anything is applied.' },
}"
    />

    Only the fields you list are compared, so TTL bounds and warm sizes the server filled in are not drift. `env` is
    compared and replaced as a whole map. `volumes` is compared by value and in order, since the attach order is the
    guest's disk order, and replaced as a whole list. Fleet never creates or deletes volumes, because they hold data:
    each name must already exist ([`sandboxVolumes.create`](/docs/reference/api/sandbox-volumes#create)), and an export
    names the template's volumes by name again. Each volume is decided for the caller when the apply checks the call,
    as in a direct `sandboxTemplates.create` or `update`, and the write runs on those decisions; before 7911c5f an apply
    that created or changed a template with volumes always failed `CONFLICT` (`volumes-changed`). See
    [sandboxes](/docs/guides/vps/sandboxes#templates) and [volumes](/docs/guides/vps/sandboxes#volumes).
  
Validation rejects unknown fields, a `version` other than 1, lists over 1000 items, repeated names (flavors repeat
by pool and name, offers by tenant) and malformed values, with `INVALID_INPUT` and the offending `details.path`.
Quota keys are resource names (a letter, then up to 63 letters, digits, `.`, `_` or `-`) with integer maxima of 0
or more.

### Names instead of ids [#names-instead-of-ids]

Sites and pools have no name of their own, so fleet keeps the names it gave them in `fleet_names`. Provider accounts
and sandbox templates use their `name`, flavors are named `<pool>/<name>`, offers `<pool> -> <tenant>`, and owner
quotas `<pool>:<resource>`. An exported template names a pool offered to the tenant by its id, since the pool is not
the tenant's.

* **Objects fleet did not create** appear in an export under their id (`pool_01...`). Keep those names: the plan
  matches by name, so a new name plans a new object.
* **Created objects get deterministic ids.** The idempotency key is `fleet:<tenant>:<kind>:<name>`, and the id
  derives from it and the caller, so an apply retried after a failure converges instead of creating duplicates.

## Plan [#plan]

`fleet.plan` (a read) diffs a document against the tenant's current objects and changes nothing. It returns:

* `changes`: one entry per item with `kind`, `name`, `action` (`create`, `update`, `delete` or `unchanged`), the
  differing `fields`, and `blocked` or `note` when they apply;
* `calls`: the exact API calls `fleet.apply` would make, in order;
* `summary`: counts per action, and `blocked` when any change cannot be applied;
* `planHash`: SHA-256 over the canonical call list.

What each kind can do:

| Kind             | Create                               | Update                                                                                                                                 | Delete, only with `prune`                                                                                                               | Cannot change                                          |
| ---------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| Provider account | `providerAccounts.create`            | `providerAccounts.update` (the listed `config` fields)                                                                                 | `providerAccounts.delete`, once no remaining pool uses it                                                                               | `provider`                                             |
| Site             | `sites.create`                       | Never                                                                                                                                  | Never                                                                                                                                   | `kind`, `region`, `iamRegion`                          |
| Pool             | `pools.create`, with its owner quota | `pools.update` (`policy`, `gpuModel`)                                                                                                  | `pools.archive`: running reservations keep their bookings. Removing `policy.substrate` from a pool that stays: its cluster is torn down | `site`, `class`, `sharing`, `agent`, `providerAccount` |
| Owner quota      | `quotas.set`                         | `quotas.set`                                                                                                                           | No                                                                                                                                      |                                                        |
| Offer            | `pools.offer`                        | `pools.offer` (`quota`, `includeDescendants`, `priority`)                                                                              | `pools.unoffer`                                                                                                                         |                                                        |
| Flavor           | `flavors.create`                     | `flavors.update` (`vcpu`, `memoryGiB`, `diskGiB`, `gpus`, `offerId`, `placement`)                                                      | `flavors.delete`                                                                                                                        |                                                        |
| Sandbox template | `sandboxTemplates.create`            | `sandboxTemplates.update` (`flavor`, `imageId`, `userData`, `defaultTtlSeconds`, `maxTtlSeconds`, `network`, `env`, `warm`, `volumes`) | `sandboxTemplates.delete`: its warm instances are deleted, running sandboxes keep their TTL                                             | `pool`, `runtime`                                      |

A change is **blocked** when it touches a field that cannot change, names an unknown site, pool or provider account,
offers a pool to its own owner, or lists flavors or sandbox templates on a deployment without them (a template then
reads `this deployment does not run sandboxes (vps v0.5)`). Fix the document, or create a new
object instead of changing a fixed field. Without `prune`, items the document leaves out are left alone, so you can
adopt FleetConfig one list at a time.

A pool's `policy.substrate` counts as such an item. Taking it out of a pool the document keeps tears the pool's
cluster down, so the plan shows it as a delete of its own (`- pool gpu (policy.substrate)`) and makes it only with
`prune`. Without `prune` the pool keeps its substrate, other policy changes are still made, and the plan notes
`policy.substrate is kept`.

Calls run in dependency order: provider account creates and updates, site creates, pool creates, pool updates and
quotas, offers, flavor creates and updates, sandbox template creates and updates, template deletes, flavor deletes,
unoffers, pool archives, and provider account deletes.

## Apply [#apply]

`fleet.apply` takes the same input plus an optional `planHash`, and runs the plan through the owning modules' own
methods. It is a composite method (`transaction: 'manual'`): it decides each call with `ctx.check` while no
transaction is open, then runs them with `ctx.invoke` in one transaction. A host without composite methods answers
`UNSUPPORTED`; the standard server has them (see [methods](/docs/extending/methods)).

      ### Plan and compare [#plan-and-compare]

    Fleet plans the document. A blocked plan fails with `INVALID_STATE` and the blocked changes in `details.blocked`.
    If you passed a `planHash` and the plan no longer has it, apply fails with `CONFIG_DRIFT` (409). A plan with no
    calls only records the document and returns `applied: 0`.
  
      ### Decide every call [#decide-every-call]

    Each call is decided for the caller, one IAM decision per call, with no transaction open. A single denial fails the
    apply with `ACCESS_DENIED` before anything is written.
  
      ### Run everything in one transaction [#run-everything-in-one-transaction]

    Inside one transaction fleet plans again. If the call list changed while IAM was deciding, nothing is applied and
    the apply fails with `CONFIG_DRIFT`. Otherwise every call runs, created names go into `fleet_names`, and the
    document is recorded as the tenant's applied document.
  
The caller therefore needs `compute:fleet:admin` and the permission of every call the plan makes:

| Calls                                                                           | Action                     |
| ------------------------------------------------------------------------------- | -------------------------- |
| `sites.create`                                                                  | `compute:sites:manage`     |
| `pools.create`                                                                  | `compute:pools:create`     |
| `pools.update`, `pools.archive`                                                 | `compute:pools:manage`     |
| `pools.offer`, `pools.unoffer`                                                  | `compute:pools:share`      |
| `quotas.set`                                                                    | `compute:quotas:manage`    |
| `flavors.create`, `flavors.update`, `flavors.delete`                            | `compute:flavors:manage`   |
| `providerAccounts.create`, `providerAccounts.update`, `providerAccounts.delete` | `compute:providers:manage` |
| `sandboxTemplates.create`, `sandboxTemplates.update`, `sandboxTemplates.delete` | `compute:sandboxes:manage` |

Every call writes its own audit row, and the apply writes one more after it succeeds. The result is
`{ plan, applied }`, the number of calls that ran.

```ts title="Plan, review, apply exactly that plan"
const { config } = await compute.api.fleet.export(credential, { tenantId: 'root' });

const plan = await compute.api.fleet.plan(credential, { tenantId: 'root', config, prune: true });
console.log(plan.summary, plan.changes.filter((change) => change.action !== 'unchanged'));

await compute.api.fleet.apply(credential, {
  tenantId: 'root',
  config,
  prune: true,
  planHash: plan.planHash, // refuse with CONFIG_DRIFT if the fleet moved since the review
});
```

## Drift [#drift]

Every apply, including one with nothing to change, records the document it applied. The `fleet.drift` reconciler
runs its key `all` every `driftEvery`, which queues one `tenant:<id>` key per tenant with an applied document; each
tenant is then re-planned in its own transaction, so one tenant's failure never blocks another.

A non-empty call list, or a blocked plan, means someone changed the fleet outside FleetConfig:

* `fleet.drifted` is published once per distinct drift, with the applied `planHash`, the summary, `blocked`, and up to
  50 changes;
* `fleet.converged` is published when the drift clears on its own;
* `fleet.status` shows the drift as of the last check.

Both events have the subject `fleet/<tenant>` and reach [webhooks](/docs/guides/events/webhooks) like any other.
Applying the document again resets the drift.

```json title="fleet.status result"
{
  "tenantId": "root",
  "applied": {
    "planHash": "3f1c...",
    "appliedBy": "usr_ops",
    "appliedAt": 1790251200000,
    "prune": true,
    "checkedAt": 1790252100000
  },
  "drift": {
    "changes": [{ "kind": "pool", "name": "vps-fra", "action": "update", "fields": ["policy"] }],
    "summary": { "create": 0, "update": 1, "delete": 0, "unchanged": 5 },
    "blocked": false,
    "planHash": "9ab0..."
  }
}
```

A tenant that never applied a document gets only `{ tenantId }`. The console's **Fleet drift** page (platform
administrators) shows the same: who applied last, when it was checked, and the drifted changes by kind.

## From the CLI [#from-the-cli]

The four `fleet` commands act as `BETTER_COMPUTE_TOKEN`, through the configuration or against `--url`. They read
and write JSON; the server validates the document.

```bash
better-compute fleet export --tenant root --out fleet.json   # refuses to overwrite a file
better-compute fleet plan fleet.json --tenant root            # --prune, --calls
better-compute fleet apply fleet.json --tenant root --yes
better-compute fleet status --tenant root
```

`fleet plan` prints one line per change that does something, then the counts:

```text
~ pool vps-fra (policy)
~ ownerQuota vps-fra:gpu (max)
+ offer vps-fra -> globex
+ flavor vps-fra/g1.large
2 to create, 2 to update, 0 to delete, 5 unchanged.
```

* **`fleet apply`** plans and prints the changes. Without `--yes` it stops there and prints the `--plan-hash` to
  apply that exact plan later. With `--yes` it applies what it just planned; with `--plan-hash` it applies a plan
  reviewed earlier, and fails with `CONFIG_DRIFT` if the fleet moved.
* **A blocked plan** exits `PLAN_BLOCKED` from `plan` and `apply`.
* **`--fail-on-drift`** makes `fleet plan` exit `FLEET_DRIFTED` when any call would run, and `fleet status` exit
  `FLEET_DRIFTED` while the recorded drift is set.

These codes exit with status 1 (usage errors exit 2). When nothing would change, `fleet apply` without
`--plan-hash` does not call `fleet.apply`, so it does not re-record the document; pass `--plan-hash` to make an
unchanged document the new drift baseline. See [the CLI reference](/docs/reference/cli#fleet-plan).

### In CI [#in-ci]

Plan on every pull request and fail when production no longer matches the reviewed file; apply on merge:

```yaml title=".github/workflows/fleet.yml (excerpt)"
- name: Production matches the reviewed document
  run: better-compute fleet plan fleet.json --tenant root --fail-on-drift
  env:
    BETTER_COMPUTE_URL: https://compute.example.com
    BETTER_COMPUTE_TOKEN: ${{ secrets.BETTER_COMPUTE_TOKEN }}

- name: Apply the reviewed document
  if: github.ref == 'refs/heads/main'
  run: better-compute fleet apply fleet.json --tenant root --yes
  env:
    BETTER_COMPUTE_URL: https://compute.example.com
    BETTER_COMPUTE_TOKEN: ${{ secrets.BETTER_COMPUTE_TOKEN }}
```

Give the pipeline an API key of a service account in the root tenant that holds `compute:fleet:admin` and the call
permissions above, and nothing else. Between runs, schedule `fleet status --fail-on-drift` or subscribe to
`fleet.drifted` to hear about hand edits the same day.

  - [fleet API reference](/docs/reference/api/fleet): export, plan, apply and status.

  - [CLI: fleet plan](/docs/reference/cli#fleet-plan): Flags of export, plan, status and apply.

  - [Rollouts](/docs/guides/rollouts): The other root-tenant tool: bcd versions in waves.


# Introduction (/docs/guides)

> Better Compute is an embeddable TypeScript control plane for GPU, CPU and sandbox capacity that uses Better IAM for identity, policy and billing.



Better Compute tracks GPU and CPU capacity: what hardware exists, what is free, and who holds it, from a whole NVLink
set of GPUs to a few vCPUs and GiB of memory. The database referees every booking, so a GPU or a vCPU can never be
given to two holders at once. It also makes machines useful: you can adopt a server that is already running, deploy
an image to a bare machine over PXE, or run isolated VMs, as VPS or as [sandboxes](/docs/guides/vps/sandboxes), on
your own hosts or on a provider's. A sandbox is a hosted VPS instance with a template, a time-to-live and an optional
warm pool, served by the `sandboxes`, `sandboxTemplates` and `sandboxSnapshots` API groups of `vps()`.

It is a library, not a hosted service. You construct the control plane with `betterCompute()` in your own process,
on your own PostgreSQL, the same way you construct Better IAM with `betterIam()`:

```ts title="compute.ts"
import { betterCompute } from 'better-compute';
import { postgres } from 'better-compute/postgres';

export const compute = betterCompute({
  database: postgres({ url: process.env.BETTER_COMPUTE_DATABASE_URL! }),
  secret: process.env.BETTER_COMPUTE_SECRET!, // at least 32 characters
  baseURL: 'https://compute.example.com',
  rootTenantId, // the Better IAM root (platform) tenant
  iam, // an IamPort, for example bridge.embedded(iam, { rootTenantId })
  modules: [], // metal(), vps(), provisionModule() and friends
});

await compute.initialize(); // registry checks, migrations, module start()
```

## What it is for [#what-it-is-for]

GPU fleets are usually tracked in spreadsheets, a scheduler's memory, or a cloud console, and each of those forgets
something: who holds which device, which machine is healthy, or what was promised for next Tuesday. Better Compute
keeps one authoritative record and builds everything else on it:

* **Fleet truth.** Sites, pools, nodes,
  devices, health and holders, in PostgreSQL.
* **Exact capacity, now and over a horizon.** Every claim is a booking row guarded by an
  `EXCLUDE` constraint, so double-booking is impossible by construction, not by careful scheduling.
* **Many provisioning paths.** Adopt a running server, PXE or virtual media plus an image stream, a provider
  reinstall, a VPS on libvirt/KVM or Cloud Hypervisor, or a provider VM. Each path is a composition of drivers.
* **A daemon per server and a horizontally scalable control plane,** with an API, a typed client, a CLI and MCP tools.
* **Scale targets per region on one PostgreSQL primary:** 10,000 nodes, 100,000 GPUs, at least 200 placements per
  second and a command p99 under 1.5 s. These are design targets that the v0.4 phase is meant to prove.
* **Modularity.** New provisioning methods and providers are modules, drivers and profiles. They need no edits to
  the `@better-compute/*` packages.
* **Agent-ready.** AI agents use the same typed API or MCP tools, and a person can confirm what an agent asks for.

## What it is not [#what-it-is-not]

The design lists these as non-goals for v1:

* A workload orchestrator. Capacity goes to Kubernetes or Slurm through config drivers; Better Compute hands out
  devices, it does not run jobs.
* A source of users, roles, keys or invoices of its own. Better IAM owns all of those.
* A telemetry store. Metrics go to a time-series database, never through PostgreSQL or the control stream.
* Active-active multi-region, live migration, or volumes and software-defined networking beyond bridges, VLANs and
  IPAM.
* Windows or macOS nodes.
* Equinix Metal, which reached end of life on 2026-06-30.

## Three runtime parts [#three-runtime-parts]

<Mermaid
  chart="flowchart LR
  C[&#x22;SDK, CLI, MCP&#x22;] -->|&#x22;POST /api/compute&#x22;| A[&#x22;api: authn, admission, placement&#x22;]
  A --> P[(PostgreSQL)]
  W[&#x22;worker: reconcilers, sinks&#x22;] --> P
  G[&#x22;gateway: sessions, dispatch&#x22;] --> P
  D[&#x22;bcd node, site, ramdisk&#x22;] -->|&#x22;gRPC over mTLS, outbound 443&#x22;| G
  A -.->|&#x22;one decision per mutation&#x22;| I[Better IAM]"
/>

### bcd, the daemon [#bcd-the-daemon]

bcd is one Go binary with a mode per job:

| Mode      | Runs on                            | Does                                                                     |
| --------- | ---------------------------------- | ------------------------------------------------------------------------ |
| `node`    | every server                       | session, inventory, NVML health, CDI device holds, commands              |
| `site`    | one or two per datacenter          | proxy DHCP, iPXE, image cache, BMC access                                |
| `ramdisk` | a machine during PXE commissioning | inspection, diagnostics, wipe, image streaming                           |
| `sim`     | test hosts                         | many real node agents on fake hardware in one process, against a gateway |

It only ever dials out, over gRPC on mutual TLS to a gateway on port 443. See
[the daemon](/docs/daemon).

### The control plane [#the-control-plane]

`betterCompute(options)` returns `{ api, handler, nodeHandler, gatewayServices, initialize, start, stop, … }`. The
same configuration runs in any of four process modes: `api` (stateless HTTP, admission and JoinService), `gateway`
(daemon sessions and dispatch), `worker` (reconcilers, event sinks, IAM jobs) or `all`. PostgreSQL is the truth, and
all SQL reads one clock, `compute_now()`. See [architecture](/docs/guides/concepts/architecture).

### The API and its clients [#the-api-and-its-clients]

Every method is `POST /api/compute/{group}/{method}`. On the server you call it as
`compute.api.reservations.create(credential, input)`, in the browser through a typed client inferred from your
server instance, from a shell through the `better-compute` CLI, and from AI agents through MCP tools. All of them go
through the same envelope: authentication, the tenant pre-check, one IAM decision, then one transaction. See
[clients](/docs/guides/clients) and the [API reference](/docs/reference/api).

## Better Compute and Better IAM [#better-compute-and-better-iam]

Better Compute has no users, roles or API keys of its own. [Better IAM](/docs/guides/iam) is its only identity
provider, policy decision point and billing ledger, and also its feature-flag store and event source. Better Compute
owns only its machine PKI: the certificates that identify daemons.

* **One decision per mutation.** Each mutating call makes exactly one Better IAM decision, such as
  `compute:reservations:create` on the reservation being requested. Reads use a short decision cache.
* **Fleet paths never call IAM.** Heartbeats, inventory, commands and reconcilers run without a single IAM call, so
  a busy fleet cannot saturate the identity system.
* **Application-owned resources.** Better Compute contributes `compute-*` resource types (`compute-pool`,
  `compute-reservation`, …) to Better IAM's catalog and resolves them itself from an in-memory projection.
* **Billing on reserved time.** Usage windows such as `gpu.<model>.hours` are recorded in Better IAM's billing
  ledger.
* **Embedded by default.** The IAM bridge (`better-compute/iam`) runs Better IAM in the same processes. A remote
  mode, where Better IAM runs elsewhere, is part of the v0.4 phase.

Both are on npm: Better Compute 0.2.0 works with Better IAM `>=0.2.0 <0.4.0` (0.3 today), and you install the two side by side
(`npm install better-compute better-iam`).

## What it covers [#what-it-covers]

  - [Concepts](/docs/guides/concepts): Architecture, the domain model, desired state and commands, and the consistency rules everything rests on.

  - [Nodes](/docs/guides/nodes): Joining with a token, attestation, inventory and health, cordon and drain, and one-shot commands.

  - [Capacity](/docs/guides/capacity): Pools, placement, reservations, quotas, gangs, calendar and spot reservations, and fractional GPUs.

  - [Bare metal](/docs/guides/bare-metal): Sites, workflows, profiles and images, BMC access, deploys, and dedicated-root leases.

  - [VPS](/docs/guides/vps): Instances on your own hosts or a provider's, flavors, provider accounts and networking.

  - [Access](/docs/guides/access): Exec and console tunnels with single-use tickets, and SSH certificates.

  - [Fleet config](/docs/guides/fleet-config): Export, plan and apply sites, pools, offers and quotas as one document, with drift detection.

  - [Rollouts](/docs/guides/rollouts): Signed bcd releases rolled out in waves within each pool's disruption budget.

  - [IAM](/docs/guides/iam): Actions and resources, policies, tenancy, billing and delegated confirmation with Better IAM.

  - [Events](/docs/guides/events): Signed webhooks, the server-sent event stream, and tamper-evident audit checkpoints.

  - [Clients](/docs/guides/clients): The typed TypeScript client, the CLI, MCP tools and the console kit.

## Project status [#project-status]

> **Version 0.2.0.** 
  Every package is at version 0.2.0 and on npm; see [Installation](/docs/guides/installation). This is an early
  release: the Go daemon `bcd` has no released binary yet, and nothing has run on real GPU hardware, so machines
  join as simulated daemons (`better-compute dev --sim N`) or as a `bcd` you build yourself.

The roadmap runs in phases, and a phase closes only when its exit tests pass in CI. Much of the TypeScript control
plane for later phases has already landed; what is still open is mostly the Go daemon and the tests that need it:

| Phase | Theme                   | What has landed                                                                                                                                                                                                                                                                                                                                                        | What is open                                                                                                                                                                                |
| ----- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| P0    | Scaffold                | Workspace, repository rules, CI, Better IAM wired in; contracts frozen (`agent/v1`, `@better-compute/core`, the v0.1 schema and method table)                                                                                                                                                                                                                          | Nothing                                                                                                                                                                                     |
| v0.1  | Adopt and reserve       | Server runtime, JoinService, gateway, machine PKI, inventory, capacity with the inline placer, IAM bridge, CLI `dev`, `serve` and `sim`, the TypeScript FakeDaemon and fleet simulator, bcd's node agent (join, session, journal, CDI holds, the v0.1 kinds, `bcd status`) tested end to end                                                                           | A bcd release, a run on a real GPU host (bcd reads GPUs through `nvidia-smi`, cb987e5, tested against captured output only); the gateway and IAM-lane benchmark gates on reference hardware |
| v0.2  | Metal                   | Workflow engine, `metal()`, images and profiles, site tokens, join requests, write-only secrets, FleetConfig, XID health and remediation, node-signed attested results                                                                                                                                                                                                 | `bcd site` and `bcd ramdisk`; emulation CI on real QEMU machines                                                                                                                            |
| v0.3  | VPS and access          | `vps()` with hosted and provider pools, the Hetzner provider, tunnels (including bcd's node half) and break-glass SSH (including bcd's `access` converger), webhooks and the SSE stream, MCP tools, delegated confirmation; bcd's `libvirt-kvm` and `cloud-hypervisor` runtimes                                                                                        | A run of either runtime on real hardware                                                                                                                                                    |
| v0.4  | Scale and HA            | Pool-lease placer, the WebSocket fallback (gateway and bcd), signed releases and rollouts, bcd self-update, `bcd helper` (an unprivileged node agent beside a root helper), remote IAM mode, Merkle audit checkpoints, metrics                                                                                                                                         | 10,000-node runs, a bcd release                                                                                                                                                             |
| v0.5  | GPU depth and ecosystem | Gang, calendar and spot reservations, MIG and time-slice atoms, dedicated-root metal, TPM attestor, the bcx driver SDK (draft), the Proxmox provider, the console kit; sandboxes (templates with env and network, TTL expiry, warm pools of paused instances, snapshots and restores, exec over SSH certificates, the instance guard, the exit test on emulated hosts) | Real-hardware MIG tests; bcd enforcing a sandbox's network policy                                                                                                                           |
| v1.0  | Freeze                  | The destructive-batch quorum end to end, the API surface check, the N-2 skew matrix, upgrade in place (in simulation)                                                                                                                                                                                                                                                  | API and protocol freeze, external security review, npm provenance from CI                                                                                                                   |

The daemon deserves a note. `daemon/` holds the Go module, and its v0.1 node agent works: `bcd join`, `bcd node`,
`bcd status`, `bcd doctor` and `bcd version`, with the bbolt journal, the `devices` and `access` convergers, five
command kinds (including `gpu.reset`), tunnels, XID and host-pressure reporting, and sealed secret and workload-token
requests. `tests/e2e/bcd.test.ts` runs it against a real control plane and gateway. It is not released (no
binaries have been published), and it reads GPUs through `nvidia-smi` rather than the NVML library (cb987e5). `bcd site` (Redfish BMC operations, proxy DHCP, TFTP, iPXE
and the image cache) and `bcd ramdisk` (wipes, image streaming, kexec, DCGM) are written and covered by Go unit tests
only; a site agent drives IPMI BMCs through the `bcx-ipmi` plugin. The `instances` converger runs VMs on
`libvirt-kvm` on a host with `virsh` and on `cloud-hypervisor` on a host with its binary (both tested against fake
tools only) and on a file-backed runtime in fake-hardware builds, which alone run `bcd sim` (many real node agents in one
process against a gateway). Self-update with A/B slots, the WebSocket fallback and `bcd helper` (privilege
separation, self-update included) are written too. The
TypeScript `FakeDaemon` in `@better-compute/testing` covers the rest, and it is what joins in
`better-compute dev --sim N`, the simulator and most test suites. See [what bcd does today](/docs/daemon#what-bcd-does-today)
and the [roadmap](/docs/reference/roadmap) for each phase's exit tests.

## Where to go next [#where-to-go-next]

  - [Quickstart](/docs/guides/quickstart): Install from npm, start the development server with simulated GPU machines, and reserve GPUs.

  - [Installation](/docs/guides/installation): Requirements, installing from npm, the umbrella package and its subpaths, and building from a clone.

  - [Concepts](/docs/guides/concepts): The architecture, the domain model and the design principles.

  - [API reference](/docs/reference/api): Every API group and method, with its HTTP route, IAM action and phase.


# Installation (/docs/guides/installation)

> Requirements, installing better-compute and Better IAM from npm, the umbrella package's subpaths, packages outside it, optional peers, and building from a clone.



Better Compute is split into `@better-compute/*` packages. Most applications install the umbrella package,
`better-compute`, which depends on all of them and re-exports each one as a subpath. A subpath loads only when you
import it, and the server never imports the injected modules (`provision`, `metal`, `vps`): your application imports
them and passes them in, so what you do not use costs nothing at run time.

> **Version 0.2.0.** 
  `better-compute` and every `@better-compute/*` package are published to npm at 0.2.0, on the `latest` dist-tag,
  and accept Better IAM `>=0.2.0 <0.4.0` (`better-iam` and `@better-iam/*`; 0.3 is current), which is on npm too. The Go daemon `bcd` is
  not an npm package and has no released binary yet; see [the daemon](#the-daemon).

## Requirements [#requirements]

| Requirement                            | Notes                                                                                                                                                                                                    |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node.js 22.12 or newer                 | The `engines` field of every package. CI runs Node.js 22 and 24 on Ubuntu and Windows. A TypeScript configuration file (`better-compute.config.ts`) needs 22.18 or newer, which strips types by default. |
| ESM                                    | Every package is an ES module with TypeScript declarations and `{ types, import }` exports only. Use `"type": "module"` or a bundler.                                                                    |
| PostgreSQL 16 or newer                 | The production store; CI runs PostgreSQL 17. The first migration creates the `btree_gist` extension, which the booking constraint needs. PgBouncer in transaction mode is supported.                     |
| PGlite, for development                | PostgreSQL in WebAssembly, through the optional peer `@electric-sql/pglite`. One connection, with transactions serialized. `btree_gist` works on it.                                                     |
| Native builds, for embedded Better IAM | Better IAM's SQLite adapter and password hashing use `better-sqlite3` and `argon2`, so allow their install scripts (pnpm: `allowBuilds`).                                                                |
| TypeScript, optional                   | `moduleResolution` of `bundler`, `node16` or `nodenext`, so subpath exports resolve.                                                                                                                     |

Compute's own store is PostgreSQL or PGlite only. It needs `EXCLUDE` constraints, `SKIP LOCKED` and range types, so
there is no SQLite adapter for it. Embedded Better IAM can still use Better IAM's SQLite adapter in development.

## Install from npm [#install-from-npm]

      ### Install Better Compute and Better IAM [#install-better-compute-and-better-iam]

    <CodeBlockTabs defaultValue="npm" groupId="package-manager">
      <CodeBlockTabsList>
        <CodeBlockTabsTrigger value="npm">
          npm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="pnpm">
          pnpm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="yarn">
          yarn
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="bun">
          bun
        </CodeBlockTabsTrigger>
      </CodeBlockTabsList>

      <CodeBlockTab value="npm">
        ```bash
        npm install better-compute better-iam
        ```
      </CodeBlockTab>

      <CodeBlockTab value="pnpm">
        ```bash
        pnpm add better-compute better-iam
        ```
      </CodeBlockTab>

      <CodeBlockTab value="yarn">
        ```bash
        yarn add better-compute better-iam
        ```
      </CodeBlockTab>

      <CodeBlockTab value="bun">
        ```bash
        bun add better-compute better-iam
        ```
      </CodeBlockTab>
    </CodeBlockTabs>

    `better-compute` is the umbrella package: it depends on every `@better-compute/*` package in it and gives you the
    `better-compute` command. `better-iam` is Better IAM's umbrella package; it brings `@better-iam/server`,
    `@better-iam/core`, `@better-iam/adapter-sqlite` and the rest of Better IAM, which Better Compute declares as
    optional peers (`^0.2.0`; see [optional peers](#optional-peers)).
  
      ### Add PGlite for development [#add-pglite-for-development]

    PGlite is the development database, and `better-compute dev` uses it when there is no database URL. It is an
    optional peer, so npm does not install it for you:

    <CodeBlockTabs defaultValue="npm" groupId="package-manager">
      <CodeBlockTabsList>
        <CodeBlockTabsTrigger value="npm">
          npm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="pnpm">
          pnpm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="yarn">
          yarn
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="bun">
          bun
        </CodeBlockTabsTrigger>
      </CodeBlockTabsList>

      <CodeBlockTab value="npm">
        ```bash
        npm install @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="pnpm">
        ```bash
        pnpm add @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="yarn">
        ```bash
        yarn add @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="bun">
        ```bash
        bun add @electric-sql/pglite
        ```
      </CodeBlockTab>
    </CodeBlockTabs>

    A deployment that runs only against PostgreSQL does not need it.
  
      ### Check it [#check-it]

    ```bash
    npx better-compute --help
    npx better-compute init   # writes better-compute.config.ts
    ```
  
With pnpm, list the Better IAM packages Better Compute imports as your own dependencies, so the optional peers
resolve under pnpm's strict layout, and allow the native builds (`allowBuilds` for `argon2` and `better-sqlite3` in
`pnpm-workspace.yaml`):

```bash
pnpm add better-compute better-iam @better-iam/server @better-iam/core @better-iam/adapter-sqlite @electric-sql/pglite
```

Then import the subpaths you need:

```ts
import { betterCompute } from 'better-compute';
import { metal } from 'better-compute/metal';
import { postgres } from 'better-compute/postgres';
import { createComputeClient } from 'better-compute/client';
```

Each package is also published on its own as `@better-compute/<name>`, for services that need only part of the
platform, for example a browser bundle that needs only `@better-compute/client`.

## Install from a clone [#install-from-a-clone]

To work on Better Compute itself, or to try changes that are not released, build it from the repository. The
workspace pins pnpm 11.19.0: install with that version once, then run the pinned copy from `node_modules`.

```bash
git clone https://github.com/better-compute/better-compute.git
cd better-compute
npx pnpm@11.19.0 install
node node_modules/pnpm/bin/pnpm.mjs build   # tsc -b: every package's dist/
```

Better IAM resolves from npm like any other dependency. The `better-compute` command is then
`packages/cli/dist/index.js`; the [quickstart](/docs/guides/quickstart) shows how to put it on your path and run the
repository's example configuration. To use the local build in another project, `pack:all` writes one tarball per
package to `artifacts/` (`@better-compute/core@0.1.0` becomes `artifacts/better-compute-core-0.1.0.tgz`) without
publishing anything; install those in place of the registry versions. The repository's packed smoke test
(`pnpm test:packed`) does exactly that in a fresh consumer. [Local development](/docs/operations/local-development)
covers the workspace scripts.

## Subpath imports [#subpath-imports]

The umbrella's subpaths are generated from `packages/better-compute/umbrella.json`; each one is a one-line
re-export of its package, and nested subpaths map to nested exports of the same package.

| Import                                              | Package                                          | What it provides                                                                                                                                                                                                |
| --------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `better-compute`                                    | `@better-compute/server`, `@better-compute/core` | `betterCompute`, `defineConfig`, `configOptions`, `ComputeError`, `defineModule`, `definePlugin`, `defineDriver`, `defineSchedulerPlugin`, `defineAttestor`, `defineSink`, `manifest`, `field`, and their types |
| `better-compute/core`                               | `@better-compute/core`                           | Browser-safe contracts: ids, errors, domain types, state tables, extension and port contracts, the IAM catalog, `defineMethod`                                                                                  |
| `better-compute/protocol`                           | `@better-compute/protocol`                       | Generated daemon and control-plane wire types                                                                                                                                                                   |
| `better-compute/inventory`                          | `@better-compute/inventory`                      | Section hashing, diffs, GPU model mapping, labels and health conditions                                                                                                                                         |
| `better-compute/capacity`                           | `@better-compute/capacity`                       | Atoms, bookings, quotas, rollups and the placement pipeline                                                                                                                                                     |
| `better-compute/provision`                          | `@better-compute/provision`                      | The workflow engine, `defineStep`, `defineProfile`, `provisionModule()`                                                                                                                                         |
| `better-compute/metal`                              | `@better-compute/metal`                          | Bare-metal verbs and drivers, `metal()`                                                                                                                                                                         |
| `better-compute/vps`                                | `@better-compute/vps`                            | Instances, runtimes, flavors, `defineProvider`, `vps()`                                                                                                                                                         |
| `better-compute/iam`, `/iam/plugin`, `/iam/testing` | `@better-compute/iam`                            | The Better IAM bridge, the plugin for IAM hosts, and an embedded Better IAM for tests and `dev --sim`                                                                                                           |
| `better-compute/events`                             | `@better-compute/events`                         | Signed webhooks and the server-sent `/watch` stream                                                                                                                                                             |
| `better-compute/fleet`                              | `@better-compute/fleet`                          | FleetConfig export, plan and apply                                                                                                                                                                              |
| `better-compute/rollout`                            | `@better-compute/rollout`                        | Signed bcd releases and wave rollouts                                                                                                                                                                           |
| `better-compute/audit`                              | `@better-compute/audit`                          | Signed per-tenant Merkle checkpoints over the audit log                                                                                                                                                         |
| `better-compute/observability`                      | `@better-compute/observability`                  | A bounded Prometheus registry and the standard metrics                                                                                                                                                          |
| `better-compute/server`                             | `@better-compute/server`                         | Everything the server exports: `betterCompute()`, the envelope, the registry, JoinService, controllers                                                                                                          |
| `better-compute/gateway`                            | `@better-compute/gateway`                        | Daemon sessions over gRPC on HTTP/2 mTLS, liveness and dispatch                                                                                                                                                 |
| `better-compute/pki`                                | `@better-compute/pki`                            | Local CAs, SPIFFE node certificates, sealed join tokens, attestors                                                                                                                                              |
| `better-compute/postgres`                           | `@better-compute/postgres`                       | `postgres({ url })`, migrations, `compute_now()` and sim-clock helpers                                                                                                                                          |
| `better-compute/pglite`                             | `@better-compute/postgres/pglite`                | `pglite()`, the development and test store                                                                                                                                                                      |
| `better-compute/client`                             | `@better-compute/client`                         | `createComputeClient`, the typed, browser-safe client                                                                                                                                                           |
| `better-compute/cli`                                | `@better-compute/cli`                            | The command line, also the umbrella's `better-compute` bin                                                                                                                                                      |
| `better-compute/mcp`                                | `@better-compute/mcp`                            | MCP tools for AI agents, gated by Better IAM                                                                                                                                                                    |
| `better-compute/testing`                            | `@better-compute/testing`                        | `computeFixture()`, `FakeDaemon`, the fleet simulator, the dev stack                                                                                                                                            |

`defineMethod` comes from `better-compute/core`, `defineStep` and `defineProfile` from `better-compute/provision`,
and `defineProvider` from `better-compute/vps`.

## Built-in and injected modules [#built-in-and-injected-modules]

`inventory` and `capacity` are built in: `betterCompute()` always registers them (after the node controllers), and
their groups (`inventory`, `pools`, `capacity`, `reservations`, `quotas`) exist in every deployment. Everything else
is a module you pass in `modules`:

```ts title="compute.ts"
import { betterCompute } from 'better-compute';
import { eventsModule } from 'better-compute/events';
import { fleetModule } from 'better-compute/fleet';
import { metal } from 'better-compute/metal';
import { postgres } from 'better-compute/postgres';
import { provisionModule } from 'better-compute/provision';
import { vps } from 'better-compute/vps';

const secret = process.env.BETTER_COMPUTE_SECRET!; // at least 32 characters
const bareMetal = metal();

export const compute = betterCompute({
  database: postgres({ url: process.env.BETTER_COMPUTE_DATABASE_URL! }),
  secret,
  baseURL: 'https://compute.example.com',
  rootTenantId,
  iam,
  // provision runs metal's workflows; metal contributes its verbs, steps and profiles
  modules: [provisionModule({ modules: [bareMetal] }), bareMetal, vps(), eventsModule({ secret }), fleetModule()],
});
```

The API types follow the modules: `client.instances` exists only when `vps()` is in `modules`. The repository
enforces the direction with `scripts/check-deps.mjs`: `@better-compute/server` never depends on `provision`,
`metal`, `vps`, `gateway`, `iam`, `postgres`, a provider or a driver. See [modules](/docs/extending/modules).

## Packages outside the umbrella [#packages-outside-the-umbrella]

These are installed directly, not through `better-compute`:

| Package                                 | What it is                                                                                                                                                                                                     |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@better-compute/provider-hetzner`      | Hetzner Cloud servers as VPS instances and offers, and Hetzner Robot bootstrap                                                                                                                                 |
| `@better-compute/provider-gcp`          | Google Compute Engine VMs as VPS instances: discovery, attach, reservations, VPCs and firewalls (new in 0.2.0)                                                                                                 |
| `@better-compute/provider-proxmox`      | Proxmox VE QEMU VMs cloned from templates                                                                                                                                                                      |
| `@better-compute/provider-digitalocean` | DigitalOcean Droplets (GPU Droplets included) as VPS instances and offers, with attach and tenant networks                                                                                                     |
| `@better-compute/provider-kubernetes`   | Not a VPS provider: the manifests that run bcd as a DaemonSet on an existing cluster's nodes (`bcdDaemonSetManifests`, `renderKubernetesManifests`; 2c9f953). See [bcd on Kubernetes](/docs/daemon/kubernetes) |
| `@better-compute/driver-sdk`            | bcx, the out-of-process driver protocol (a v0.5 draft; the protocol is unstable until v0.5 ships)                                                                                                              |
| `@better-compute/console-kit`           | Framework-free view models for consoles over the typed client                                                                                                                                                  |

Nothing in `packages/` imports a `@better-compute/provider-*` package; you pass providers to `vps()` yourself. See
[providers](/docs/guides/vps/providers).

## Optional peers [#optional-peers]

The umbrella declares these as optional peer dependencies. Install the ones you use; `better-iam` brings all of the
`@better-iam/*` ones:

| Peer                                                      | Needed for                                                                                                                                           |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@electric-sql/pglite` `^0.5.8`                           | `better-compute/pglite`, the testing fixture, `dev` without a database URL. Without it, the CLI fails with `MISSING_DEPENDENCY` and an install hint. |
| `@better-iam/server`, `@better-iam/core` `>=0.2.0 <0.4.0` | Embedded Better IAM through the bridge                                                                                                               |
| `@better-iam/adapter-sqlite` `>=0.2.0 <0.4.0`             | Better IAM on SQLite in development, and the embedded IAM of `dev --sim`                                                                             |
| `@better-iam/client` `>=0.2.0 <0.4.0`                     | `better-compute iam setup --meters`, which creates the billing meters in Better IAM                                                                  |
| `@better-iam/cli` `>=0.2.0 <0.4.0`                        | Declared for Better IAM's CLI; today the `better-compute` CLI reads its credential from `BETTER_COMPUTE_TOKEN` only                                  |

`@better-compute/cli` also declares `@better-compute/testing` as an optional peer, for `dev --sim` and `sim`; the
umbrella depends on it, so installing `better-compute` provides it. See [Better IAM](/docs/guides/iam) for how the
two products fit together.

## The daemon [#the-daemon]

`bcd` is not an npm package. It is designed as a Go binary for Linux on amd64 and arm64, released by goreleaser as
`bcd_<version>_linux_{amd64,arm64}.tar.gz` with a `.sha256`, plus the bare binaries for machines, and listed in an
offline-signed release manifest (`better-compute release sign`). `better-compute daemon install-script` already
writes the POSIX installer: it downloads the pinned binary from
`{base-url}{base-path}/downloads/bcd/{version}/linux-{arch}`, checks its SHA-256, writes `/etc/bcd/config.toml`
and a hardened `bcd.service`, and runs `bcd join`. The control plane serves that download route when you configure
`downloads` (see [serving the binaries](/docs/daemon/install#serving-the-binaries)).

No binary is released yet. `daemon/` is a Go module whose v0.1 node agent is built and tested: `bcd join`,
`bcd node` and `bcd status` run end to end against a real gateway in `tests/e2e/bcd.test.ts`. What is missing is a
release (goreleaser is configured but has not run), a run on a real GPU host (GPUs are read through `nvidia-smi`,
cb987e5), and the later modes. You can build it yourself with Go 1.26
or later (`go build ./cmd/bcd` in `daemon/`). The TypeScript `FakeDaemon` in `better-compute/testing` speaks the real
protocol for everything else. See [installing the daemon](/docs/daemon/install).

## All packages [#all-packages]

## Verify the installation [#verify-the-installation]

`doctor` loads your configuration and checks the database, the schema, the clock and the method registry without
changing anything:

```bash
better-compute migrate
better-compute doctor --strict
```

It runs `compute.selfCheck()` (applied migrations, `btree_gist`, `compute_now()` against the database clock, every
method's IAM action, each module's own checks) and adds the CLI's findings, such as `iam.not-configured` when no
Better IAM is wired in. It prints the report and exits non-zero with `DOCTOR_FINDINGS` when any finding is an error
(an unreachable or unmigrated database, a broken PKI bundle); `--strict` does so on any warning too, so you can run
it in CI.
See the [CLI reference](/docs/reference/cli).

## Next steps [#next-steps]

  - [Quickstart](/docs/guides/quickstart): Run the development server with simulated GPU machines and reserve GPUs.

  - [Configuration](/docs/operations/configuration): Every `betterCompute()` option and the configuration file the CLI loads.

  - [Database](/docs/operations/database): PostgreSQL, PGlite, migrations and PgBouncer.


# Kubernetes fleet pools (/docs/guides/kubernetes-fleet-pools)

> Back a GPU pool with a managed Kubernetes cluster on GKE, EKS or DOKS. The same FleetConfig document works on each; swap the provider account and keep the rest.



A Kubernetes fleet pool is a GPU pool whose machines are the nodes of a managed Kubernetes cluster. The pool's
`policy.substrate` names one of the owner's provider accounts and the node shapes it wants (flavors). The
`kubernetes.substrates` reconciler then creates the cluster and its node pools at the provider, or adopts a cluster
that already exists. It runs bcd on the nodes as a DaemonSet, and those nodes join the pool. From that point compute
books their GPUs like any other pool's, with `binding: 'kubernetes'` reservations that bcd's device plugin hands to the
kubelet.

The provider runs the control plane. better-compute never runs one; it orchestrates GKE, EKS or DOKS through
`ProviderImpl.kubernetes`. The pool substrate is provider-neutral: nothing in it is specific to Google, AWS or
DigitalOcean. Only the provider account changes, along with the names that account uses for a region, a machine type
and an accelerator.

## Turn it on [#turn-it-on]

The substrate needs the vps and kubernetes modules, built with the same provider definitions, and an agent that says
what to run on the cluster. Add `fleetModule()` to manage the pool as FleetConfig.

```ts title="compute.ts"
import { betterCompute } from 'better-compute/server';
import { vps } from 'better-compute/vps';
import { fleetModule } from 'better-compute/fleet';
import { kubernetes } from '@better-compute/kubernetes';
import { kubernetesSubstrateAgent } from '@better-compute/provider-kubernetes';
import { awsProvider } from '@better-compute/provider-aws';
import { digitaloceanProvider } from '@better-compute/provider-digitalocean';
import { gcpProvider } from '@better-compute/provider-gcp';

const providers = [gcpProvider(), awsProvider(), digitaloceanProvider()];

export const compute = betterCompute({
  // database, secret, baseURL, iam, ...
  modules: [
    vps({ providers }),
    kubernetes({
      providers,
      // What runs bcd on the cluster: the DaemonSet options, except what the pool decides.
      substrate: { agent: kubernetesSubstrateAgent({ image, gatewayUrl, pins, controlPlaneUrl }) },
    }),
    fleetModule(),
  ],
});
```

Without `substrate`, pools that declare `policy.substrate` stay in `error`, and `selfCheck` reports
`kubernetes.substrate-agent-missing`.

## One document, three providers [#one-document-three-providers]

The pool below asks for an 8 x H100 node pool of two nodes. Only `providerAccounts`, and what that account calls a
region, a machine type and an accelerator, differ between providers.

```json title="fleet.json"
{
  "version": 1,
  "sites": [{ "name": "k8s", "kind": "shared", "region": "us-east-1" }],
  "providerAccounts": [
    {
      "name": "managed-k8s",
      "provider": "aws",
      "config": {
        "region": "us-east-1",
        "zones": ["us-east-1a", "us-east-1b"],
        "eksClusterRoleArn": "arn:aws:iam::111111111111:role/eks-cluster",
        "eksNodeRoleArn": "arn:aws:iam::111111111111:role/eks-node",
        "eksSubnetIds": ["subnet-0000000000000000a", "subnet-0000000000000000b"]
      }
    }
  ],
  "pools": [
    {
      "name": "gpu-fleet",
      "site": "k8s",
      "class": "metal-host",
      "gpuModel": "H100",
      "policy": {
        "kubernetes": { "platformOperated": true },
        "substrate": {
          "kind": "kubernetes",
          "account": "managed-k8s",
          "cluster": { "create": { "location": "us-east-1" } },
          "flavors": [
            { "name": "h100", "machineType": "p5.48xlarge", "nodes": 2, "gpus": { "type": "nvidia-h100", "count": 8 } }
          ]
        }
      }
    }
  ]
}
```

  **GKE:**

    ```json
    { "name": "managed-k8s", "provider": "gcp", "config": { "projectId": "my-project", "zones": ["us-central1-a"] } }
    ```

    Site region and `cluster.create.location`: `us-central1`. Flavor: `"machineType": "a3-highgpu-8g"`,
    `"gpus": { "type": "nvidia-h100-80gb", "count": 8 }`.
  
  **EKS:**

    ```json
    {
      "name": "managed-k8s",
      "provider": "aws",
      "config": {
        "region": "us-east-1",
        "zones": ["us-east-1a", "us-east-1b"],
        "eksClusterRoleArn": "arn:aws:iam::111111111111:role/eks-cluster",
        "eksNodeRoleArn": "arn:aws:iam::111111111111:role/eks-node",
        "eksSubnetIds": ["subnet-0000000000000000a", "subnet-0000000000000000b"]
      }
    }
    ```

    Site region and `cluster.create.location`: `us-east-1` (EKS clusters are regional; a zone is refused). Flavor:
    `"machineType": "p5.48xlarge"`, `"gpus": { "type": "nvidia-h100", "count": 8 }`. The accelerator must match the GPUs
    of the instance type, which EKS reads from EC2.
  
  **DOKS:**

    ```json
    { "name": "managed-k8s", "provider": "digitalocean", "config": { "token": "secret://<tenant>/<id>", "regions": ["tor1"] } }
    ```

    Site region and `cluster.create.location`: `tor1`. Flavor: `"machineType": "gpu-h100x8-640gb"`,
    `"gpus": { "type": "nvidia-h100", "count": 8 }`. The API token is a secret reference, never a value.
  
Apply it as any FleetConfig document (see [Fleet configuration](/docs/guides/fleet-config)):

```sh
better-compute fleet plan fleet.json
better-compute fleet apply fleet.json
```

The account and the cluster's place (location, private nodes, or the adopted cluster's id) are the substrate's
identity: a plan that changes them is blocked, so create a new pool instead. Sizing changes in place.

## What happens [#what-happens]

1. **Cluster.** The substrate records an ordinary `kubernetesClusters` cluster (`substratePoolId` set) with one node
   pool per flavor. Each node pool is labelled `better-compute.dev/substrate-pool=<pool id>`, and a Spot flavor's is
   also labelled `better-compute.dev/capacity=spot`. The clusters reconciler creates them at the provider.
2. **bcd.** Once the cluster is ready, each run mints a 600 s admin credential through the provider
   (`ProviderKubernetes.credentials`), lists the substrate's Nodes and server-side applies bcd: a Namespace, a
   ConfigMap and a DaemonSet that selects exactly the labelled Nodes. The credential is held in memory for that run
   only. It is never stored, sealed, logged or published.
3. **Join.** While Nodes are still to join, a hosted `kubernetes` join token sits in a Secret, with one use per Node
   whose bcd is not connected. bcd's join container reads it, and each node joins the pool. Once every Node has
   joined, the token is revoked and the Secret deleted.
4. **Book.** Reservations with `binding: 'kubernetes'` book the nodes' GPUs. bcd's device plugin advertises only the
   devices a reservation holds. The nodes of a Spot flavor carry the system label `bc.dev/capacity=spot` once they
   are matched to their Node: only reservations made with `spot: true` are placed on them, so nothing sold as
   uninterruptible sits on a machine the provider may reclaim.
5. **Scale.** Change a flavor's `nodes` in the document and apply. The provider removes Nodes. Each pool node is
   matched to its Kubernetes Node by the Node name bcd reports: the pod's `spec.nodeName`, in its inventory's `os`
   section (`kubernetesNodeName`), or the hostname from a bcd that does not report one yet. When a matched node is disconnected and its Node is
   gone, its reservations end: bound ones are released with their quota refunded once, and spot reservations and
   anything on a Spot node end as `preempted`. Then the node is retired.
6. **Teardown.** Remove the pool from the document and apply with `prune`, or archive the pool. Taking only
   `policy.substrate` out of a pool that stays tears the substrate down the same way, and also needs `prune`: a
   plain apply keeps the substrate. The token is revoked
   and the nodes are cordoned. The substrate waits for their bookings to end; nothing is force-released. Then it
   removes bcd from the cluster (only objects carrying its label), deletes a created cluster and its node pools, and
   retires the nodes. Nothing is left at the provider or in compute.

Read the state with `kubernetesSubstrates.get({ tenantId, poolId })`: mode, cluster, state and message, Nodes listed
and joined, and a drain's upgrade. It never returns a token or a credential.

A substrate is `ready` once bcd is applied, whether or not its Nodes have joined yet. `waitingToJoin` says how many
listed Nodes have no joined bcd and since when none has joined. If that lasts 15 minutes, the message says so and
names bcd's DaemonSet: look at its pods (the image pull, the gateway URL and pins given to the substrate agent, the
GPU runtime class). `doctor` reports the same as `kubernetes.substrate-nodes-not-joining`.

An `error` does not stop the nodes that already joined from being looked after. When the provider refuses one
flavor's node pool (quota, a stockout), or a change to the document cannot be made in place, the substrate stays in
`error` with the reason as its message, and each run still lists the Nodes, retires the ones that are gone, keeps a
join token for new Nodes and re-applies bcd. It only stops at a cluster whose control plane never answered or is
being deleted.

## Adopt a cluster you already run [#adopt-a-cluster-you-already-run]

Use `"cluster": { "adopt": "<provider cluster id>" }` instead of `create`. The cluster is read and never created,
changed or deleted. Each flavor adds a node pool named `bc-<last 6 of the pool id>-<flavor>`, so it can never take
over a node pool of your own with the same name. On teardown only those node pools and bcd's labelled objects go;
the cluster and its own node pools stay. Last, the substrate revokes its own access (`revokeCredentials`): the
`bc-substrate` service account in `bc-access`, its cluster-admin binding, and the `bc-access` namespace once no
subject is left in it (only if better-compute made it). GKE, EKS and DOKS all implement this; the cluster-wide
`better-compute:edit` ClusterRoles and the access-guard admission policy stay, since they grant nothing on their own
(an EKS access entry the tenant made stays too). If bcd or that access cannot be removed from an adopted cluster (a
refused credential, a failed delete), the substrate stays `deleting` and retries every 2 minutes. It never leaves a
privileged DaemonSet or a cluster-admin identity behind unannounced. For the same reason `kubernetesClusters.credentials`
refuses an adopted cluster (`INVALID_STATE`): a kubeconfig's service account and binding would outlive the substrate
on a cluster better-compute no longer manages. Reach your own cluster with the provider's credentials.

One cluster backs one pool, whichever account or deployment reaches it. Three checks hold that:

* A second pool that names the same account and cluster id is refused when its substrate is recorded.
* Before any node pool is made, the cluster's provider-global id (the GKE selfLink, the EKS ARN, the DOKS UUID) is
  recorded as what the pool claims. A second pool that reaches the same cluster through another provider account is
  refused there: its cluster goes to `error`, and nothing is created in the cluster.
* The substrate then claims the cluster itself with a ConfigMap, `kube-system/better-compute-substrate`, labelled
  with the pool id. It is created, never applied over an existing one, so another better-compute deployment that
  adopts the same cluster is refused before it adds node pools or applies bcd. Teardown deletes the claim, and a
  substrate that never held it removes nothing and does not revoke the `bc-substrate` identity.

If the provider refuses to delete a created cluster (it is in use, the account lost a permission), the cluster and
its substrate go to `error` with the provider's message and the delete is retried with a growing wait: 5 minutes,
then 10, 20, 40 and so on, at most every 6 hours. Fix the cause and the next retry finishes the teardown.

To move a cluster to another pool, archive the first pool and wait for its substrate to be `deleted`. Delete the
ConfigMap by hand only for a cluster whose deployment is gone for good.

| Provider | Adopt id                    | What the cluster needs                                                                                                                             |
| -------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| GKE      | `<location>/<cluster name>` | Nothing more: the account's Google identity reaches the cluster API.                                                                               |
| EKS      | `<region>/<cluster name>`   | An access entry granting the account's principal `AmazonEKSClusterAdminPolicy`: better-compute grants itself admin only on clusters it created.    |
| DOKS     | the cluster's UUID          | DigitalOcean's own GPU device plugins off (`nvidia_gpu_device_plugin`, `amd_gpu_device_plugin`), or a GPU node pool is refused (`devicePluginOn`). |

## Provider differences [#provider-differences]

Everything above behaves the same on each provider. These are the places where providers differ:

|                       | GKE                                                                                                                         | EKS                                                                                | DOKS                                                                                 |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Cluster location      | A region or a zone                                                                                                          | A region only                                                                      | A region                                                                             |
| A flavor's `nodes`    | Per zone of the node pool: the flavor's `zones`, else the cluster's default node zones (often three for a regional cluster) | Per Availability Zone of the account's subnets: two subnets in two zones double it | The total (DOKS node pools have no zones)                                            |
| `nodes: 0`            | Yes                                                                                                                         | Yes                                                                                | Only with `autoscaling: { minNodes: 0, … }`; a fixed-size pool has at least one node |
| `zones`, `diskGiB`    | Yes                                                                                                                         | Yes (no disk type)                                                                 | No: node pools are regional and take their size's disk                               |
| Spot flavors          | Yes                                                                                                                         | Yes (Spot capacity)                                                                | A DigitalOcean preview: the account needs `allowSpot` and `previews`                 |
| Kubernetes Node names | The VM hostname                                                                                                             | Instance DNS names; bcd reports the Node name it runs on                           | `<node pool>-<suffix>`                                                               |

To get the same node count everywhere, pin each flavor to one zone on GKE and EKS. Otherwise, size each flavor with
its account's zone count in mind.

## Check your own provider [#check-your-own-provider]

A driver should report the cluster's provider-global id as `outputs.globalId` in the state `ensureCluster` and
`getCluster` return: the same string whichever account reads the cluster. Without it, adoption is only checked per
provider account and by the in-cluster claim.

`@better-compute/testing` exports `kubernetesProviderConformanceCases()`: what the clusters and substrate reconcilers
need of any `ProviderImpl.kubernetes`, as cases shaped like vps' `providerConformanceCases()`. GKE, EKS and DOKS pass
them against their emulators. The subject gives your provider's vocabulary, and the cases do the rest. They create a
cluster idempotently and check its ready outputs (`endpoint`, `version`, `location`, `nodeVersion`). They check that
the mode never changes in place, and that an upgrade happens in place. They add a labelled GPU node pool and a Spot
one, resize them in place, and check that a node pool never changes shape. They mint credentials for the cluster
endpoint, delete twice, and read and extend an adoptable cluster.

```ts
import { runConformance } from '@better-compute/core/conformance';
import { kubernetesProviderConformanceCases } from '@better-compute/testing';

const report = await runConformance(kubernetesProviderConformanceCases(), () => ({
  provider: myProvider({ fetch: emulator.fetch }),
  config: { region: 'us-east-1' },
  location: 'us-east-1',
  nodePool: { machineType: 'm7i.large', nodeCount: 1 },
  gpuNodePool: { machineType: 'g6.xlarge', nodeCount: 1, accelerators: { type: 'nvidia-l4', count: 1 } },
  accepts: (credentials) => emulator.issued.includes(credentials.token),
}));
```

Set `zeroNodes: false` if a fixed-size node pool needs at least one node; `nodeCount: 0` must then be refused as
`INVALID_INPUT`. Set `spot: false` if the provider has no Spot node pools.


# Quickstart (/docs/guides/quickstart)

> Install Better Compute from npm, run the development server with simulated GPU machines, reserve GPUs, watch capacity change and release them.



This walkthrough runs a complete control plane on your laptop: PGlite as the database, an embedded Better IAM, a
gateway, and eight simulated HGX H100 machines that join the way the real daemon does. You then reserve four GPUs,
see where they were placed, watch capacity change, and release them. Nothing above the daemon is mocked: the
machines get sealed join tokens, call the Join RPC, open a Session stream, pass `diag.nvml` acceptance and are
adopted by the real runtime.

> **Requirements.** 
  Node.js 22.12 or newer; 22.18 or newer to load a TypeScript configuration such as `better-compute init` writes,
  because Node.js strips types by default from 22.18. No database server is needed.

      ### Install [#install]

    In a new, empty project, install Better Compute, Better IAM (for the embedded IAM that `--sim` starts) and PGlite
    (the in-memory database):

    <CodeBlockTabs defaultValue="npm" groupId="package-manager">
      <CodeBlockTabsList>
        <CodeBlockTabsTrigger value="npm">
          npm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="pnpm">
          pnpm
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="yarn">
          yarn
        </CodeBlockTabsTrigger>

        <CodeBlockTabsTrigger value="bun">
          bun
        </CodeBlockTabsTrigger>
      </CodeBlockTabsList>

      <CodeBlockTab value="npm">
        ```bash
        npm install better-compute better-iam @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="pnpm">
        ```bash
        pnpm add better-compute better-iam @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="yarn">
        ```bash
        yarn add better-compute better-iam @electric-sql/pglite
        ```
      </CodeBlockTab>

      <CodeBlockTab value="bun">
        ```bash
        bun add better-compute better-iam @electric-sql/pglite
        ```
      </CodeBlockTab>
    </CodeBlockTabs>

    With pnpm, also list `@better-iam/server`, `@better-iam/core` and `@better-iam/adapter-sqlite`; see
    [Installation](/docs/guides/installation#install-from-npm). Check the command:

    ```bash
    npx better-compute version   # 0.2.0
    ```

    This page writes the command as `better-compute`: in the project directory, run it as `npx better-compute` (or
    `pnpm exec better-compute`), or define an alias such as `alias better-compute="npx better-compute"`.
    `better-compute help` lists every command, and `better-compute help dev` shows one. The
    [CLI reference](/docs/reference/cli) has them all.
  
      ### Start the development server with simulated machines [#start-the-development-server-with-simulated-machines]

    ```bash
    npx better-compute dev --memory --sim 8
    ```

    * `dev` runs the API and every worker role on `127.0.0.1:3000`, and always migrates first.
    * `--memory` uses a throwaway in-memory PGlite. Leave it out to keep the data between runs, in
      `.better-compute/dev-db`.
    * `--sim 8` adds a gateway on loopback (cleartext HTTP/2, sessions authenticated by a header, for local development
      only) and joins eight simulated machines through it. `--sim-variant l40s` gives 4×L40S machines instead;
      `hgx-h100-gpu-missing` gives machines that fail acceptance and are never adopted; `cpu` gives machines without GPUs.
      The simulated machines come from `@better-compute/testing`, which the umbrella package depends on.

    With no configuration file, `dev` runs the server's own groups and the built-in `inventory` and `capacity` modules,
    with a development secret generated once in `.better-compute/dev-secret`. No IAM is wired in, so `--sim` starts an
    embedded, in-memory Better IAM and prints a root administrator's token for it. Standard error shows:

    ```text
    Development database: pglite:memory (secret in …/.better-compute/dev-secret)
    better-compute listening on http://127.0.0.1:3000/api/compute
    Workers running: controllers, scheduler, sinks, iam
    Dev gateway (h2c, header auth, loopback only): http://127.0.0.1:52011
    Embedded IAM root tenant …; root token (dev only): biam_ses_…
    Joined 8 simulated hgx-h100 node(s) into pool_…
    8 simulated node(s) adopted
    ```

    Behind those lines, `dev` created a shared site in region `dev-1`, a `metal-host`
    pool for `H100` GPUs whose acceptance policy runs `diag.nvml` and expects 8 GPUs, and one
    join token with `maxUses` 8. Each simulated machine joined with it, received a node
    certificate, opened its session, passed acceptance, and became `adopted` at desired-state generation 1, which is when
    the placer starts to consider it. Leave this terminal running.

          #### With the repository's example configuration

        The repository's `examples/dev` is a configuration with every module registered (provision and metal, vps, events,
        fleet) and a host export that adds server-sent events at `GET /watch`, which the [watch events](#watch-events) step
        uses. It is not part of the npm packages, so run it from a clone:

        ```bash
        git clone https://github.com/better-compute/better-compute.git
        cd better-compute
        npx pnpm@11.19.0 install
        node node_modules/pnpm/bin/pnpm.mjs build   # tsc -b: every package's dist/
        alias better-compute="node '$PWD/packages/cli/dist/index.js'"
        cd examples/dev
        better-compute dev --memory --sim 8
        ```

        The workspace pins pnpm 11.19.0 (pnpm 12 refuses it). In PowerShell, define the command with
        `$bc = "$PWD\packages\cli\dist\index.js"; function better-compute { node $bc @args }` instead of the alias. The
        first line of the output then names the configuration:
        `Development database: pglite:memory for …/examples/dev/better-compute.config.ts`.
            
      ### Look around [#look-around]

    Open a second terminal in the same directory (in a clone, define the alias again), and act as the root administrator
    that `dev` printed. With
    `BETTER_COMPUTE_URL` set, the CLI calls the running server over HTTP instead of loading a configuration:

    ```bash
    export BETTER_COMPUTE_URL=http://127.0.0.1:3000
    export BETTER_COMPUTE_TOKEN='biam_ses_…'   # the root token dev printed
    export BETTER_COMPUTE_TENANT='…'           # the root tenant dev printed

    better-compute api system.whoami
    better-compute nodes list
    better-compute pools list --query "items[].id"
    source <(better-compute completion bash)   # optional: tab completion for every command
    ```

    * Every API group has its commands (`nodes list`, `pools list`, `reserve`, ...; `better-compute help` lists them),
      and they read the tenant from `--tenant` or `BETTER_COMPUTE_TENANT`. See
      [command groups](/docs/guides/clients/cli#command-groups).
    * `api GROUP.METHOD key=value …` calls any method; `key:=json` passes JSON values. `api --list` prints every route
      with the IAM action it is decided on.
    * `nodes list` shows eight nodes with `provisionState` `adopted`.

    Copy the pool id and look at its capacity:

    ```bash
    export POOL=pool_…   # the id pools list printed
    better-compute capacity --pool $POOL
    ```

    The rollup reports, per resource, `total`, `allocatable`, `allocated`, `reserved` and `available`, plus the GPUs
    that are `provisioning`, `unhealthy`, in `maintenance` or `unreachable`. Eight adopted H100 boards give 64 allocatable
    GPUs.
  
      ### Reserve GPUs [#reserve-gpus]

    ```bash
    better-compute reserve --pool $POOL --gpus 4 --for 1h
    ```

    `reserve` calls [`reservations.create`](/docs/reference/api/reservations#create). The request goes through the full
    admission sequence: the tenant pre-check, an idempotency lookup, a draft reservation, exactly one Better IAM decision
    on `compute:reservations:create`, then placement in a second transaction. The CLI prints the idempotency key it
    generated on standard error; repeat the command with `--key` and the same value to retry safely without booking twice.

    The pool's owner gets an effectively unlimited GPU quota when the pool is created, so this reservation fits. To
    try a cap, set one with `better-compute quotas set --pool $POOL --resource gpu --max 4`
    and reserve again: the second request fails with `QUOTA_EXCEEDED`. See
    [quotas and usage](/docs/guides/capacity/quotas-and-usage).

    With the default inline placer, the answer is already `placed`: the placer picked one node, locked it and its GPUs,
    and inserted four bookings. The reservation then moves on without you:

    <Mermaid
      chart="stateDiagram-v2
  direction LR
  placed --> binding: start time reached
  binding --> active: node reports AllocationBound
  active --> releasing: release, end or lapsed lease
  releasing --> released"
    />

    Read it back, with the id `reserve` printed:

    ```bash
    better-compute reservations get rsv_… --query state     # active
    better-compute reservations get rsv_… --query binding
    ```

    ```json
    {
      "nodeId": "node_…",
      "cdiDevices": ["bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…"],
      "envFile": "/run/bcd/allocations/alloc_….env"
    }
    ```

    The binding is a `hold`: the node's desired state gained a device assignment, and the (simulated) daemon wrote a CDI
    spec for exactly the four booked GPUs and reported them. A workload manager on the node, such as Slurm, Kubernetes or
    containerd with CDI, consumes those device names. Run `better-compute capacity --pool $POOL` again: `allocated` is 4
    and `available` is 60.
  
      ### Watch events [#watch-events]

    Events leave through a transactional outbox. The `/watch` route of the repository's `examples/dev` configuration
    (see [with the repository's example configuration](#start-the-development-server-with-simulated-machines)) streams
    your tenant's events as server-sent events, and `events watch` prints them one JSON line each. A `dev` without a
    configuration has no `/watch` route, so skip this step there:

    ```bash
    better-compute events watch --types "node.*,reservation.*"
    ```

    It is plain server-sent events, so `curl -N -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN"
    "$BETTER_COMPUTE_URL/watch?types=node.*"` shows the same. In another terminal, cordon a node and watch
    `node.cordoned` arrive:

    ```bash
    better-compute nodes cordon node_… --reason demo
    ```

    A cordoned node keeps its current reservations; the placer just stops choosing it. `nodes uncordon` reverses it.
  
      ### Release [#release]

    ```bash
    better-compute release rsv_…
    better-compute capacity --pool $POOL
    ```

    `release` calls [`reservations.release`](/docs/reference/api/reservations#release). The bookings are truncated to
    the current database time, the device assignment leaves the node's desired state, and `available` returns to 64.
    Billing covers reserved time, from `active` until the release. A reservation that has not started yet is withdrawn
    with `release --cancel` instead.
  
## The same calls over HTTP and TypeScript [#the-same-calls-over-http-and-typescript]

Every CLI command above is a call to the HTTP API, `POST /api/compute/{group}/{method}`. Requests need a JSON body,
`content-type: application/json` and `X-Better-Compute: 1`; answers are `{ "data": … }` or
`{ "error": { "code", "message" } }`:

```bash
curl -s http://127.0.0.1:3000/api/compute/pools/list \
  -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" \
  -H "content-type: application/json" -H "x-better-compute: 1" \
  -d "{\"tenantId\": \"$BETTER_COMPUTE_TENANT\"}"
```

In TypeScript, the client is typed by your server instance through a type-only import, so every group, method,
input and result is checked without code generation:

```ts title="reserve.ts"
import { createComputeClient } from 'better-compute/client';
import type { compute } from './compute.server'; // type-only: your betterCompute() instance

const client = createComputeClient<typeof compute>({
  baseURL: 'http://127.0.0.1:3000',
  token: () => process.env.BETTER_COMPUTE_TOKEN,
});

const reservation = await client.reservations.create({
  tenantId,
  poolId,
  devices: [{ class: 'gpu', count: 4 }],
  duration: 3_600_000, // milliseconds in the TypeScript type; over HTTP "1h" works too
  binding: 'hold',
  idempotencyKey: 'quickstart-1', // retrying with the same key returns the same reservation
});
```

See [the TypeScript client](/docs/guides/clients/typescript-client).

## Your own configuration [#your-own-configuration]

`better-compute init` writes a starter `better-compute.config.ts` in the current directory, and never overwrites an
existing one. This is what it writes, with its comments shortened:

```ts title="better-compute.config.ts"
import { unconfiguredIam, type CommandSpec } from 'better-compute/cli';
import { pglite } from 'better-compute/pglite';
import { postgres } from 'better-compute/postgres';
import { defineConfig } from 'better-compute/server';

export default defineConfig(({ env = process.env, cwd = process.cwd() } = {}) => ({
  database: env.BETTER_COMPUTE_DATABASE_URL
    ? postgres({ url: env.BETTER_COMPUTE_DATABASE_URL })
    : pglite({ dataDir: `${cwd}/.better-compute-pglite` }),
  secret: env.BETTER_COMPUTE_SECRET ?? '',
  baseURL: env.BETTER_COMPUTE_BASE_URL ?? 'http://localhost:3000',
  rootTenantId: 'root',
  iam: unconfiguredIam(),
}));

export const commands: CommandSpec<any>[] = [];
```

It imports the umbrella package's subpaths, so run it in a project where `better-compute` resolves, with the
optional peers `@electric-sql/pglite` (for PGlite) and `@better-iam/server`, `@better-iam/core` and
`@better-iam/adapter-sqlite` (for the embedded Better IAM of `--sim`, all three in `better-iam`) installed: the
project from the first step is one. See [Installation](/docs/guides/installation). The template reads the secret
from the environment, and `betterCompute()` refuses one shorter than 32 characters with `INVALID_CONFIG`:

```bash
better-compute init
export BETTER_COMPUTE_SECRET=$(node -e "console.log(require('node:crypto').randomBytes(32).toString('base64url'))")
better-compute dev --sim 4
```

`dev` finds the configuration by walking up from the working directory. `unconfiguredIam()` authenticates nobody,
so `--sim` again starts the embedded Better IAM; once you wire a real one in, `dev --sim` acts as
`BETTER_COMPUTE_TOKEN`, which must then be a root administrator's token.

## Next steps [#next-steps]

  - [Concepts](/docs/guides/concepts): How the control plane, the gateway, the daemon and Better IAM fit together.

  - [Joining nodes](/docs/guides/nodes/joining): Join tokens, the install script and what happens when a real machine joins.

  - [Reservations](/docs/guides/capacity/reservations): Durations, leases, extensions, future starts and the reservation state machine.

  - [Local development](/docs/operations/local-development): The dev server, the deterministic simulator and the example deployments.


# Rollouts (/docs/guides/rollouts)

> Register offline-signed bcd releases and roll a daemon version out in waves within each pool's disruption budget, with a failure gate, pause and abort.



Upgrading the agent on every machine is one of the riskiest things a platform does. A bad build that crashes on one
kernel, or loses its session under load, can take a fleet offline faster than anyone can react. A
rollout moves one signed `bcd` version through the fleet in waves (1%, 10%, 50%, 100% by
default), never disrupts more of a pool than its budget allows, pauses by itself when too many nodes of a wave fail,
waits between waves, and can send nodes whose upgrade has not confirmed back. A release that nodes confirmed is
undone by [rolling forward](#undo-a-bad-release-by-rolling-forward) to a newer one.

Nodes learn their target version through their NodeSpec (`daemon`), which is desired
state rather than a command: a node that was offline during the rollout converges on the right version when it
returns.

> **Both halves written; bcd is not released.** 
  The control plane is shipped in `@better-compute/rollout`: releases, rollouts, the controller, `NodeSpec.daemon`
  and the `daemon.upgrade` command kind. bcd implements the node side (de9f380): it takes only a release manifest
  signed by the release keys pinned in its `[update] release_pins` (and refuses every update without pins), checks
  the sequence against its release floor, the expiry and the binary's SHA-256, stages it in the other of two slots,
  swaps with `upgrade.json` and a 120-second deadline, confirms on the new binary's first `Welcome`, and swaps back
  when none comes. Once confirmed, it refuses every older binary, the one in its other slot included (c139b9c). `tests/e2e/bcd.test.ts` runs a real `daemon.upgrade` to a second signed build. No bcd release has
  been cut, so no fleet has self-updated yet. Install nodes with `--release-pins` so they can; see
  [self-update](/docs/daemon/install#self-update). The FakeDaemon's opt-in `selfUpdate` plays the node side in the
  simulator.

Nodes that run `bcd node` unprivileged beside [`bcd helper`](/docs/daemon#privilege-separation) (installed with
`daemon install-script --unprivileged`) take part in rollouts like any other: the node downloads the release, and the
root helper verifies the manifest under its own release pins, swaps root's slots and confirms on the node's
`Welcome` (2d67b25; see [self-update through the helper](/docs/daemon#self-update-through-the-helper)). Their helper
must be a build with the `update.*` operations; an older one refuses them.

<Mermaid
  chart="flowchart LR
  K[&#x22;release sign, offline&#x22;] --> M[&#x22;manifest.json&#x22;]
  M -->|&#x22;releases.register&#x22;| R[&#x22;daemon_releases&#x22;]
  R -->|&#x22;rollouts.create&#x22;| P[&#x22;rollout and node plan&#x22;]
  P -->|&#x22;rollout reconciler, every 30 s&#x22;| S[&#x22;NodeSpec.daemon&#x22;]
  S --> D[&#x22;bcd: download, verify, swap, rejoin&#x22;]
  D -->|&#x22;Hello daemon_version&#x22;| P"
/>

## Sign a release [#sign-a-release]

A release is a manifest of one version's files, signed offline by keys that the control
plane knows only by their `sha256/...` pins. Nothing on a server can sign one.

      ### Create the signing keys, once [#create-the-signing-keys-once]

    Each release manager creates a key on their own machine and hands out only its pin:

    ```bash
    better-compute release keygen --out ~/keys/release-alice.pem   # prints the pin
    ```

    The public half is written to `<out>.pub.pem`, for whoever builds manifests that you cosign.
  
      ### Build [#build]

    goreleaser in `daemon/` produces `bcd_<version>_linux_amd64` and `_arm64` as archives and as bare binaries. The
    bare binary is what a node installs.
  
      ### Sign, and cosign up to the threshold [#sign-and-cosign-up-to-the-threshold]

    ```bash
    better-compute release sign dist --product bcd --version 1.4.0 --sequence 14 \
      --key ~/keys/release-alice.pem --cosigners ~/keys/release-bob.pub.pem
    better-compute release cosign dist/manifest.json --key ~/keys/release-bob.pem
    better-compute release verify dist/manifest.json --pins sha256/...,sha256/... --threshold 2 --dir dist
    ```

    `release sign` hashes the files at the top of the directory (skipping `.sha256` files, `manifest.json` and
    signature files), takes each platform from goreleaser names (`_linux_amd64`), and writes `manifest.json`. `--sequence` is the rollback counter and must grow with every release; the manifest is
    valid for 365 days unless you pass `--validity-days`.
  
The manifest body (`type: bettercompute.release`, `spec: 1`) lists `product`, `version`, `sequence`, `issuedAt`,
`expiresAt`, the signing `keys` (`kid`, `alg: ES256`, `spki`) and the `artifacts` (`name`, `platform`, `sha256`,
`size`), sorted by name. The published form is `{ manifest, signatures }`: the canonical body in base64 and one
`{ kid, sig }` per signer.

## Turn the module on [#turn-the-module-on]

```ts title="compute.ts"
import { rolloutModule } from 'better-compute/rollout';

export const compute = betterCompute({
  // database, secret, iam, baseURL ...
  modules: [
    rolloutModule({
      release: { pins: ['sha256/...', 'sha256/...'], threshold: 2 },
      downloadsUrl: 'https://downloads.example.com',
    }),
  ],
});
```

<TypeTable
  type="{
  'release.pins': {
    type: 'string[]',
    description: 'The sha256/... SPKI pins of the release keys. Without any, rolloutModule throws INVALID_CONFIG.',
    required: true,
  },
  'release.threshold': {
    type: 'number',
    description: 'How many distinct pinned keys must have signed a manifest.',
    default: '1',
  },
  downloadsUrl: {
    type: 'string',
    description: 'An https URL; nodes download {downloadsUrl}/bcd/{version}/linux-{arch}.',
    required: true,
  },
  every: { type: 'Duration', description: 'How often active rollouts advance.', default: &#x22;'30s'&#x22; },
}"
/>

The module only builds download URLs; it does not serve files. Put each release's bare binaries at
`{downloadsUrl}/bcd/{version}/linux-amd64` and `linux-arm64`. The api can serve them itself: with
`betterCompute({ downloads: { dir } })` (or `{ blob }`), set `downloadsUrl` to `{baseURL}{basePath}/downloads`; see
[installing bcd](/docs/daemon/install#serving-the-binaries).

The module adds the migration `rollout/0001_rollouts` (tables `daemon_releases`, `rollouts`, `rollout_nodes`), the
`rollout` reconciler and the `daemon.upgrade` command kind. Every method decides `compute:fleet:admin` on
`compute-fleet/global`, which exists only in the root tenant, so rollouts are for platform operators. See
[the releases reference](/docs/reference/api/releases) and [the rollouts reference](/docs/reference/api/rollouts).

## Register a release [#register-a-release]

`releases.register { tenantId, manifest }` verifies the signed manifest and records it:

* a threshold of distinct pinned keys signed it, its body is canonical, its product is `bcd`, and it has not
  expired. A failure is `INVALID_INPUT` with `details.reason`: `malformed`, `threshold`, `not-canonical`, `product`
  or `expired`;
* its `sequence` is above every registered release's. An older one is `INVALID_INPUT` with reason `rollback`, so the
  release channel never moves backwards;
* it lists at most one bare binary per platform, and at least one for Linux. A bare binary is an artifact with a
  platform that is not an archive, checksum, signature or metadata file. Several candidates for one platform, or no
  Linux binary, is `INVALID_INPUT`.

Registering the same manifest again returns the stored release; the same version with another sequence is
`CONFLICT`. `releases.list` returns every release, highest sequence first.

```bash
better-compute releases register dist/manifest.json --tenant root   # sends the file as it is
better-compute releases list --tenant root
```

## Start a rollout [#start-a-rollout]

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The root tenant.', required: true },
  version: { type: 'string', description: 'A registered release (semver, a leading v is ignored).', required: true },
  poolIds: { type: 'string[]', description: '1 to 1000 pools. Default: every pool the tenant owns.' },
  waves: {
    type: 'number[]',
    description: 'Cumulative percentages: 1 to 10 increasing integers ending at 100.',
    default: '[1, 10, 50, 100]',
  },
  maxFailureRatio: {
    type: 'number',
    description: &#x22;Pause when a wave's failed / (succeeded + failed) exceeds this, 0 to 1.&#x22;,
    default: '0.05',
  },
  soakSeconds: { type: 'number', description: 'Wait after a wave finishes, 0 to 604800.', default: '600' },
  upgradeTimeoutSeconds: {
    type: 'number',
    description: 'A node that has not reported the version by then failed, 60 to 86400.',
    default: '900',
  },
  idempotencyKey: { type: 'string', description: 'A retried create with the same key returns the same rollout.' },
}"
/>

```bash
better-compute rollouts create --tenant root --version 1.4.0 --waves 1,10,50,100 --soak 10m --key bcd-1.4.0
```

The CLI takes `--soak` and `--upgrade-timeout` as durations (`10m`, `15m`) and checks `--waves` and
`--max-failure-ratio` before it calls; without `--key` it sends a new idempotency key. See
[the rollouts commands](/docs/reference/cli#rollouts-create).

`rollouts.create` snapshots every node with desired state (generation above 0) in the pools in scope, shuffled by
a hash of the rollout id so the canaries spread across pools, and assigns each to a wave. Every wave holds at least
one more node than the one before while nodes remain: 20 nodes with `[10, 50, 100]` make waves of 2, 8 and 10.
Nodes that already report the version start as succeeded. It publishes `rollout.started`.

| Situation                                                              | Error                                |
| ---------------------------------------------------------------------- | ------------------------------------ |
| The version is not registered, or a pool is unknown                    | `NOT_FOUND`                          |
| No node in scope runs bcd yet                                          | `INVALID_STATE`                      |
| A node is already in a running or paused rollout (one writer per node) | `CONFLICT`, with `details.rolloutId` |
| The idempotency key was used for another version                       | `CONFLICT`                           |
| Waves, ratios or times out of range                                    | `INVALID_INPUT`                      |

## How the controller advances [#how-the-controller-advances]

The `rollout` reconciler wakes every 30 seconds and queues each running or paused rollout, and each aborted one
that is still rolling nodes back; `resume` and `abort` queue theirs at once. One pass locks the rollout and does
this:

      ### Settle nodes in flight [#settle-nodes-in-flight]

    A node succeeds (or finishes rolling back) when it rejoins and reports the target version in its `Hello`
    (`node_sessions.daemon_version`). One that has not after `upgradeTimeoutSeconds` fails (`timeout` or
    `rollback-timeout`), and its NodeSpec goes back to the version it ran before. This step runs even while the rollout
    is paused.
  
      ### Apply the failure gate [#apply-the-failure-gate]

    If `failed / (succeeded + failed)` in the current wave exceeds `maxFailureRatio`, the rollout pauses with a reason
    such as `failure gate: 4 of 6 node(s) in wave 2 failed` and publishes `rollout.paused`.
  
      ### Soak, then advance [#soak-then-advance]

    When no node of the wave is pending or upgrading, the controller waits `soakSeconds` from the last node's finish.
    Then the next wave starts (`rollout.wave`, with the wave and its percentage), or, after the last wave, the rollout
    completes (`rollout.completed`).
  
      ### Start nodes within the budget [#start-nodes-within-the-budget]

    Pending nodes of the current wave start in order. A node that no longer exists is skipped (`node-gone`), a node that
    is not connected is skipped (`offline`), and a node already on the version succeeds. Any other starts only if its
    pool's disruption budget has room; otherwise it waits for the next pass.
  
<Mermaid
  chart="stateDiagram-v2
  state &#x22;rolling-back&#x22; as rollingBack
  state &#x22;rolled-back&#x22; as rolledBack
  [*] --> pending
  pending --> upgrading: started within budget
  pending --> succeeded: already on the version
  pending --> skipped: offline, gone or aborted
  upgrading --> succeeded: reports the version
  upgrading --> failed: timeout or abort
  upgrading --> rollingBack: abort with rollback
  rollingBack --> rolledBack: reports the old version
  rollingBack --> failed: rollback timeout"
/>

A rollout itself is `running`, `paused`, `completed` or `aborted`.

### The disruption budget [#the-disruption-budget]

A pool's `policy.disruptionBudget` (a fraction from 0 to 1, default 0.05) is shared with
[health remediation](/docs/guides/nodes/inventory-and-health): a node counts as disrupted while it has a pending, running or
paused workflow, or an upgrade or rollback in flight. At most `max(1, floor(nodes × budget))` nodes of a pool are
disrupted at once, so a pool of fewer than 40 nodes upgrades one node at a time under the default, and a pool of 10
with a budget of 0.2 upgrades two.

## What a node is told [#what-a-node-is-told]

When a node starts, the controller marks it `upgrading`, records the version it reported as `fromVersion`, and
invalidates its desired state. Its NodeSpec then carries:

```json title="NodeSpec daemon section"
{
  "version": "1.4.0",
  "url": "https://downloads.example.com/bcd/1.4.0/linux-amd64",
  "sha256": "9f2c...e41a",
  "signature": "eyJtYW5pZmVzdCI6..."
}
```

* **The version** is the node's row in the latest rollout that touched it: the rollout's version while upgrading or
  after success, its previous version after a failure or a rollback. A node no rollout touched gets the version of
  the latest completed rollout that covered its pool, so skipped offline nodes and nodes that join later converge on
  it. A node with neither gets no `daemon` section, and bcd is left alone.
* **The platform** comes from the node's `cpu.arch` label (`x86_64` and `amd64` are `linux-amd64`; `aarch64` and
  `arm64` are `linux-arm64`). Without the label, or without a binary for the platform, there is no section.
* **The digest** is the bare binary's SHA-256 from the verified manifest.
* **The signature** is the release's signed manifest as registered (`{ manifest, signatures }`, the `manifest.json`
  of `release sign`), its UTF-8 JSON as the section's bytes (`releaseSignature`, ecf7cd0). bcd verifies it against its
  own `[update] release_pins`, so a compromised control plane cannot push a binary of its own. `daemon.upgrade`
  carries the same field.

### daemon.upgrade for restricted sessions [#daemonupgrade-for-restricted-sessions]

A daemon older than the server's minimum protocol gets `Welcome.upgrade_required`, and its session then accepts only
`daemon.upgrade`, so it would never apply a NodeSpec. When the controller starts such a node (its
`node_sessions.proto_max` is below the minimum), it also enqueues a `daemon.upgrade` command with the same target,
idempotency key `rollout:<rolloutId>:<nodeId>`, expiring after the upgrade timeout. The kind runs on the `control`
lane, is retry-safe and self-terminating, times out after 15 minutes, and takes the `DaemonSection` as its payload.
The gateway dispatches only that kind to an upgrade-only session (and replays an in-flight one at `Hello`); the node's
other commands stay `pending` until it rejoins at a supported version. See [command kinds](/docs/reference/command-kinds)
and [sessions](/docs/daemon/sessions).

## Pause, resume, abort [#pause-resume-abort]

| Method                                  | Effect                                                                          |
| --------------------------------------- | ------------------------------------------------------------------------------- |
| `rollouts.pause { reason? }`            | `paused` (reason default `paused by an operator`); nodes in flight still settle |
| `rollouts.resume`                       | `running` again, with a pass at once                                            |
| `rollouts.abort { reason?, rollback? }` | `aborted`; pending nodes are skipped (`aborted`)                                |

Reasons are at most 500 characters. Only a running or paused rollout changes; anything else is `INVALID_STATE`.

Without `rollback`, an abort marks nodes still upgrading as failed, and their NodeSpec points back at the version
they reported before (or drops the `daemon` section when that version was unknown); nodes that already succeeded
keep the new version. With `rollback: true`, every node still `upgrading` whose previous version is a registered
release moves to `rolling-back`, gets that version as its target, and finishes as `rolled-back` when it reports it
(8347a3d). Nodes that `succeeded` keep the release either way.

```bash
better-compute rollouts pause rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --reason 'XID spike in wave 2'
better-compute rollouts abort rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --rollback
```

### Undo a bad release by rolling forward [#undo-a-bad-release-by-rolling-forward]

A node that succeeded has confirmed the release: its release floor rose to that release's sequence, and bcd now
refuses every older binary, including the one still in its other slot (c139b9c). Sending it back would leave it
`rolling-back` until the timeout, so abort does not try. Only an upgrade that has not confirmed goes back, and bcd's
own watchdog swaps such an update back anyway when the new binary gets no `Welcome` before its deadline.

To undo a release that nodes already confirmed, roll forward:

1. Build the fix, or rebuild the last good code, as a new version.
2. Sign it with a higher `--sequence` than the bad release (`releases.register` refuses a lower one with reason
   `rollback`).
3. Register it and start a new rollout of that version. Abort the bad rollout first if it is still running, since
   a node can be in only one running or paused rollout.

```bash
better-compute release sign dist --product bcd --version 1.4.1 --sequence 15 \
  --key ~/keys/release-alice.pem --cosigners ~/keys/release-bob.pub.pem
better-compute release cosign dist/manifest.json --key ~/keys/release-bob.pem
better-compute releases register dist/manifest.json --tenant root
better-compute rollouts abort rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --reason 'bad release 1.4.0'
better-compute rollouts create --tenant root --version 1.4.1 --key bcd-1.4.1
```

bcd refuses an older release on its own too: the release floor is enforced on the node, so a control plane cannot
push a node back to an older binary even by accident.

## Watch progress [#watch-progress]

`rollouts.list` returns the tenant's latest 100 rollouts. `rollouts.get` adds `nodeList`: every node's pool, wave,
state, `fromVersion`, start and finish times, and error.

```bash
better-compute rollouts list --tenant root
better-compute rollouts get rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --query nodes
```

```json title="rollouts.get (excerpt)"
{
  "id": "rollout_01jzq3v0m6y8x7w5t4s3r2p1n0",
  "version": "1.4.0",
  "state": "running",
  "waves": [1, 10, 50, 100],
  "wave": 1,
  "maxFailureRatio": 0.05,
  "soakSeconds": 600,
  "upgradeTimeoutSeconds": 900,
  "nodes": {
    "pending": 180,
    "upgrading": 8,
    "succeeded": 12,
    "failed": 0,
    "skipped": 0,
    "rolling-back": 0,
    "rolled-back": 0
  }
}
```

`wave` counts from zero. The console's **Rollouts** page (platform administrators) shows each wave's progress,
lists failures, and has pause, resume and abort buttons.

| Event               | When                                          |
| ------------------- | --------------------------------------------- |
| `rollout.started`   | Created, with the node count and waves        |
| `rollout.wave`      | A wave started, with its index and percentage |
| `rollout.paused`    | Paused by hand or by the failure gate         |
| `rollout.resumed`   | Resumed                                       |
| `rollout.aborted`   | Aborted                                       |
| `rollout.completed` | The last wave finished and soaked             |

All have the subject `rollout/<id>` and reach [webhooks](/docs/guides/events/webhooks) and
[watch](/docs/guides/events/watch).

## Defaults [#defaults]

| Setting                  | Default                                                   |
| ------------------------ | --------------------------------------------------------- |
| Waves                    | 1%, 10%, 50%, 100%                                        |
| Failure gate             | Pause when more than 5% of a wave's finished nodes failed |
| Soak                     | 600 s after each wave, the last one included              |
| Upgrade timeout          | 900 s per node                                            |
| Disruption budget        | 5% of a pool, at least one node                           |
| Controller interval      | 30 s                                                      |
| `daemon.upgrade` timeout | 15 min                                                    |
| Manifest validity        | 365 days                                                  |


# Tutorial: a GPU cluster end to end (/docs/guides/tutorial-gpu-cluster)

> A step-by-step walkthrough with the real CLI commands - create a site and a pool, mint a join token, join a GPU server with bcd, watch it get adopted, reserve GPUs, use the CDI device names on the node, release them and follow the events.



This tutorial builds a small GPU cluster with the commands that exist today: a site and a pool, a join token, a GPU
server joined with `bcd`, its adoption, a reservation of four GPUs, the CDI device names on
the node, the release, and the events each step publishes. Every command below is in the `better-compute help`
manifest or in `bcd`'s own flags. The pages linked from each step explain the parts in depth; this page puts them in
order.

The [quickstart](/docs/guides/quickstart) does the second half (reserve, bind, release) with simulated machines on a
laptop. This tutorial adds the first half, the part a real server goes through.

> **What runs where today.** 
  The control plane and the CLI are released on npm. `bcd`, the Go daemon, is built and tested but not released, so you
  build it from `daemon/` yourself (step 4). The whole path in this tutorial (join, adoption after `diag.nvml`, a GPU
  hold bound with a node-signed `AllocationBound`, release) runs in `tests/e2e/bcd.test.ts` against a real gateway, with
  bcd on fake hardware. `tests/e2e/bcd-host-real.test.ts` runs a host build of bcd on one real machine with an RTX 5090
  in WSL 2 (8e9602c, 898a23c): it joins, is adopted and exports the GPU's metrics. The GPU model table knows that card since
  223022b, but that run's bcd is an ordinary user without `[helper]`, so it binds no GPUs: it claims no binding
  capability and a reservation never reaches `binding` there (see [step 4](#4-join-the-server-with-bcd)). No test has yet bound a
  reservation on a data-center GPU server.

## What you need [#what-you-need]

| Piece                                                        | Why                                                                             | See                                                            |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| A control plane with a gateway                               | `bcd` only dials out, to a gateway over mTLS on port 443 (8443 in the examples) | [deployment](/docs/operations/deployment), `examples/postgres` |
| The machine PKI                                              | JoinService's keys and the gateway's serving CA, whose pins the node carries    | [PKI](/docs/operations/pki)                                    |
| Better IAM wired in, and a root administrator's token        | Every command below is decided by IAM as the token                              | [Better IAM](/docs/guides/iam)                                 |
| A Linux server with NVIDIA GPUs, the driver and `nvidia-smi` | bcd reads GPUs through `nvidia-smi`                                             | [host configuration](/docs/daemon/host-config#nvidia-driver)   |
| Go 1.26 or later                                             | To build bcd                                                                    | [installing bcd](/docs/daemon/install#release-artifacts)       |

`examples/postgres/README.md` is the reference for the control plane: `pki init`, `pki issue-gateway`, `migrate`,
and `serve` with the `api`, worker and `gateway` roles from one configuration, which also mounts `GET /watch` for
step 9. Its IAM section shows the bridge to wire; until it is wired no credential authenticates.

> **Following along on a laptop.** 
  `better-compute dev` cannot take a real bcd: its gateway (`dev --sim`) speaks cleartext HTTP/2 on loopback and admits
  only the simulated machines it joined itself (`packages/cli/src/dev-sim.ts`). On a laptop, run
  `cd examples/dev && better-compute dev --memory --sim 2`, which prints a root token and tenant for step 1 and serves
  `/watch`. Steps 1 to 3 then work as written. In step 4 the two simulated machines stand in for your server: from step
  5 on, use the pool `dev` printed (`Joined 2 simulated hgx-h100 node(s) into pool_…`) as `$POOL`. To join a real bcd
  on a laptop, use the dev stack in code with its mTLS listener, as the end-to-end tests do (see
  [a real bcd](/docs/operations/local-development#a-real-bcd)).

      <div className="fd-steps">
      <div className="fd-step">
        ### Act as the platform operator [#act-as-the-platform-operator]

        Point the CLI at the API and give it the token of a root administrator. With `BETTER_COMPUTE_URL` set, every command
        is an HTTP call to that server (plain `http` is allowed only for loopback addresses); `BETTER_COMPUTE_TENANT` is the
        default `--tenant` of every token command.

        ```bash
        export BETTER_COMPUTE_URL=https://compute.example.com
        export BETTER_COMPUTE_TOKEN='biam_…'      # a root administrator's token
        export BETTER_COMPUTE_TENANT=root         # the root tenant's id

        better-compute api system.whoami
        ```

        `better-compute help` lists every command, and `better-compute help <command>` shows one; see
        [the CLI](/docs/guides/clients/cli) for credentials, output formats and `--query`.
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Create a site and a pool [#create-a-site-and-a-pool]

        A site is where machines live; a pool is a set of machines of one GPU
        model with one policy. A platform site is `shared` (offered to tenants) or `dedicated`, and only the root tenant
        creates those ([sites](/docs/guides/bare-metal/sites#sites)).

        ```bash
        export SITE=$(better-compute sites create --kind shared --region eu-1 --key site-eu-1 --query id)
        ```

        The pool's policy decides what a joining machine must pass before it takes work. Write it to a file:

        ```json title="pool-policy.json"
        {
          "acceptance": [{ "kind": "diag.nvml" }],
          "expect": { "gpuCount": 8 }
        }
        ```

        ```bash
        export POOL=$(better-compute pools create --site $SITE --class metal-host --gpu-model H100 \
          --policy-file pool-policy.json --key pool-eu-1-h100 --query id)
        better-compute pools get $POOL
        ```

        * `--class metal-host` makes joining machines `host` nodes, whose GPUs are reserved as holds
          ([classes](/docs/guides/capacity/pools#classes)). The agent defaults to `bcd`, which join tokens require.
        * `--gpu-model` is the GPU family inventory derives from the card's name, such as `H100` for `H100-SXM5-80GB`. A pool
          holds one model ([one GPU model](/docs/guides/capacity/pools#one-gpu-model)).
        * `diag.nvml` with `expect.gpuCount` adopts a machine only when it shows 8 healthy GPUs (see
          [acceptance checks](/docs/guides/nodes/inventory-and-health#acceptance-checks)). Set the count to what your server
          has.
        * `--key` makes the call retry-safe: the same key returns the same site or pool.

        The pool's owner, here the root tenant, gets an effectively unlimited `gpu` quota on it. To let a customer tenant
        reserve from it, offer it with a quota. The resource is `gpu` (singular): a quota row for any other name grants
        nothing.

        ```bash
        better-compute pools offer $POOL --to acme --quota gpu=16
        ```

        See [offerings](/docs/guides/capacity/pools#offerings).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Mint a join token [#mint-a-join-token]

        A join token lets a limited number of machines join one pool for a limited time. Bind it
        to the machine you mean when you know one of its identifiers, and write it to a file instead of the terminal:

        ```bash
        better-compute join-tokens create --pool $POOL --ttl 2h --max-uses 1 \
          --expect-serial SN-7701 --key join-sn-7701 --out ./join-token
        ```

        `--out` writes a new file with mode 0600 and never overwrites one. The token is shown once; the control plane keeps
        only a hash of its secret. `--ttl` takes `1m` to `24h`. A machine whose firmware reports no product UUID or serial (WSL
        2 and many VMs) has no trusted hardware key, and a token without an `expect` binding is refused for it; bind such a
        token with `--expect-mac` instead, as `tests/e2e/bcd-host-real.test.ts` does. See
        [binding a token to one machine](/docs/guides/nodes/joining#binding-a-token-to-one-machine).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Join the server with bcd [#join-the-server-with-bcd]

        Build bcd for the server. The default build reads GPUs through `nvidia-smi` and needs no cgo:

        ```bash
        cd daemon
        CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/bcd ./cmd/bcd
        ```

        There are two ways to install it.

        **With the install script.** Serve the binary from the API (`downloads`, which the examples set from
        `BETTER_COMPUTE_DOWNLOADS_DIR`; see [serving the binaries](/docs/daemon/install#serving-the-binaries)) under
        goreleaser's file name `bcd_<version>_linux_amd64`, then generate a script that pins its digest and the PKI's keys:

        ```bash
        better-compute daemon install-script --pool $POOL --site $SITE \
          --gateway-url https://gw.eu-1.example.com:8443 --base-url https://compute.example.com \
          --bcd-version 0.1.0 --sha256 amd64=$(sha256sum bin/bcd | cut -d' ' -f1) --out install-bcd.sh
        ```

        Copy `install-bcd.sh` and `join-token` to the server and run the script as root, with the token on standard input:

        ```bash
        sudo sh install-bcd.sh < join-token
        ```

        It checks the host, downloads and verifies bcd, writes `/etc/bcd/config.toml` with the pins, runs `bcd doctor`, pipes
        the token into `bcd join` and enables `bcd.service`. See [what the script does](/docs/daemon/install#what-the-script-does).

        **By hand, from the well-known document.** When the control plane publishes its gateway
        (`BETTER_COMPUTE_GATEWAY_URL` in the examples), a machine with no `config.toml` can join with the control plane's URL
        and three pins. Print the pins on the control plane, from the sealed bundle:

        ```bash
        better-compute pki show --query servingCaSpkiPin
        better-compute pki show --query joinSigningKeyPin
        better-compute pki show --query joinEncryptionKeyPin
        ```

        Then, on the server, as root, with bcd copied to `/usr/local/bin/bcd`:

        ```bash
        bcd join --server https://compute.example.com/api/compute \
          --serving-ca-pin 'sha256/…' --join-signing-pin 'sha256/…' --join-encryption-pin 'sha256/…' \
          --token-file ./join-token
        bcd node --config /etc/bcd/config.toml
        ```

        `bcd join` reads `<server>/.well-known/better-compute`, refuses it when a published pin differs from yours, joins,
        and only then writes `/etc/bcd/config.toml` and `join-encryption.jwk`. `--token-file` takes only a regular file its
        owner alone can read (0600 or 0400), and deletes it after the join. The token is never an argument: bcd reads it from
        `BCD_JOIN_TOKEN`, standard input or `--token-file`, and refuses positional arguments. On success it prints
        `joined as node_… (site site_…, role node); certificate valid until …`, and a second run prints
        `already joined as node_…` without spending a token. Run `bcd node` under systemd in production; the install script's
        unit is in [the systemd unit](/docs/daemon/install#the-systemd-unit). See
        [joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document).

        > **Run bcd node as root, or beside bcd helper.** 
          A GPU hold writes `/etc/cdi/bcd-<allocation>.json` and creates a cgroup slice, which only root can do. Since da60bbd
          a host build of `bcd node` running as an ordinary user without a `[helper]` section advertises no `cdi`,
          `devices.limits` or `binding.kubernetes` capability, so capacity places no GPU reservation on it, and a hold that
          arrives anyway fails at once as `node-unprivileged`; `bcd doctor` warns about it. Run it as root (the install
          script's default) or unprivileged beside a root `bcd helper` (`--unprivileged`; see
          [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)).
        

        What happens inside the join (the sealed token, the signed `JoinResponse`, the node certificate) is in
        [the join flow](/docs/guides/nodes/joining#the-join-flow).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Watch the node get adopted [#watch-the-node-get-adopted]

        The node is created in `adopting`. Once its session is up, the NodeLifecycle controller sends the pool's acceptance
        checks; when `diag.nvml` passes, the node becomes `adopted`, gets desired-state generation 1, and the placer starts
        to consider it.

        ```bash
        better-compute nodes list --pool $POOL
        export NODE=node_…   # the id nodes list printed (or bcd join)
        better-compute nodes get $NODE --query provisionState
        better-compute nodes get $NODE --query conditions
        better-compute nodes events $NODE
        ```

        `nodes events` shows `node.joined`, `node.connected`, `node.acceptance.started` and `node.adopted` in that order. On
        the server, `bcd status` reads the running agent over its unix socket (`/run/bcd/bcd.sock` by default):

        ```bash
        sudo bcd status            # session, spec generations, sections, GPUs
        sudo bcd status --json
        ```

        The text form shows the node, its session (`connected (epoch …)`), the applied and observed spec generations, one
        line per section, and one line per GPU with its utilization, temperature, power and memory, and `held by <allocation>` once a hold has it. See [bcd status](/docs/daemon#bcd-status) and
        [telemetry and status](/docs/daemon/telemetry).

        When the node stays `adopting`:

        | What you see                                        | Cause                                                                                                                                            | Fix                                                                                                    |
        | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
        | `AcceptanceFailed` true, reason `diag.nvml:<check>` | A GPU failed a check (`nvml.init`, `ecc.uncorrectable`, `row-remap.pending`, `pcie.link`, `xid.recent`, …) or the count is not `expect.gpuCount` | Fix the machine, then re-run acceptance (below)                                                        |
        | `AcceptanceFailed` true, reason `Expired`           | The check did not finish within its 10 minutes                                                                                                   | Check the session (`bcd status`), then re-run                                                          |
        | `GpuModelUnknown` true                              | The card's name is not in inventory's GPU model table; the node is NoSchedule                                                                    | See [GPU naming](/docs/guides/nodes/inventory-and-health#gpu-naming)                                   |
        | No node at all                                      | The join was refused: the agent only ever sees `Join refused`                                                                                    | Read the pool owner's `security.join.refused` events ([refusals](/docs/guides/nodes/joining#refusals)) |

        A failed acceptance is sticky. Fix the cause and start a new attempt with `nodes.adopt`, which has no command of its
        own; call it through `api`:

        ```bash
        better-compute api nodes.adopt tenantId=$BETTER_COMPUTE_TENANT nodeId=$NODE reason='GPU 3 reseated'
        ```
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Check capacity [#check-capacity]

        ```bash
        better-compute capacity --pool $POOL
        better-compute capacity --preview --pool $POOL --gpus 4 --for 2h
        ```

        The rollup reports, per resource, `total`, `allocatable`, `allocated`, `reserved` and `available`, plus the GPUs that
        are `provisioning`, `unhealthy`, in `maintenance` or `unreachable`. One adopted 8-GPU server gives 8 allocatable GPUs.
        A preview books nothing; it answers whether the request would fit now, with the host CPUs and memory it would book.
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Reserve GPUs [#reserve-gpus]

        ```bash
        better-compute reserve --pool $POOL --gpus 4 --for 2h
        ```

        `reserve` calls [`reservations.create`](/docs/reference/api/reservations#create) with a new idempotency key, which it
        prints on standard error; run it again with `--key <that key>` to retry without booking twice. It waits up to
        `--wait` seconds (default 20) for the placement. With the default placer the answer is already `placed`: the placer
        picked the node, locked its GPUs and booked four of them, together with the pool's per-GPU share of the host's CPUs
        and memory (see [host resources](/docs/guides/capacity/reservations#host-resources)). Other flags narrow the choice:
        `--model`, `--product`, `--selector "vramGiB >= 80"`, `--nvlink` for one NVLink domain, `--start +2h` for a later
        start. See [reservations](/docs/guides/capacity/reservations).

        The reservation then binds on its own:

        <Mermaid
          chart="stateDiagram-v2
  direction LR
  placed --> binding: start reached
  binding --> active: the node reports AllocationBound
  active --> releasing: release
  releasing --> released"
        />

        At the start, the node's desired state gains a device assignment for the allocation. bcd writes the hold's CDI spec
        and env file, then journals an `AllocationBound` signed with its node key; once the control plane verified it, the
        reservation is `active`.

        ```bash
        export RSV=rsv_…   # the id reserve printed
        better-compute reservations get $RSV --query state
        better-compute reservations get $RSV --query binding
        ```

        ```json
        {
          "nodeId": "node_…",
          "cdiDevices": ["bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…"],
          "envFile": "/run/bcd/allocations/alloc_….env",
          "cgroupParent": "bcd-alloc-alloc_….slice"
        }
        ```

        `tests/e2e/spine.test.ts` checks this end to end (3c3a53f): the CDI device names `reservations.get` returns are
        exactly the GPU UUIDs the database booked, and the node's CDI spec lists exactly those GPUs. A hold that fails on the
        node (`overlap`, `device-missing`, `resources-exhausted`) is placed again, at most twice; no report within 5 minutes
        counts as a failure ([bindings](/docs/guides/capacity/reservations#bindings)).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Use the GPUs on the node [#use-the-gpus-on-the-node]

        The binding is a hold: the GPUs are set aside for the reservation, and a workload manager on the node uses them by
        their CDI names. On the server:

        ```bash
        ls /etc/cdi/                                  # bcd-alloc_….json beside nvidia.json
        cat /etc/cdi/bcd-alloc_….json
        cat /run/bcd/allocations/alloc_….env
        ```

        | Where                                                    | What it holds                                                                                                                                          |
        | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
        | `/etc/cdi/bcd-<allocation>.json`                         | A CDI spec of kind `bcd.dev/gpu` with one device per held GPU, named by its UUID, and the annotations `bcd.dev/device-id` and `bcd.dev/reservation-id` |
        | `/run/bcd/allocations/<allocation>.env`                  | `NVIDIA_VISIBLE_DEVICES` (the held GPUs' UUIDs), `BCD_ALLOCATION_ID` and, with host limits, `BCD_CGROUP_PARENT`                                        |
        | `bcd.slice/bcd-alloc.slice/bcd-alloc-<allocation>.slice` | The cgroup slice with the booked CPUs, memory and huge pages, when the hold has limits                                                                 |

        Since 6ecb860 a hold's spec also carries the device nodes, mounts and hooks of exactly the held GPUs, copied from
        NVIDIA's own CDI spec, `/etc/cdi/nvidia.json`. So the name `bcd.dev/gpu=<uuid>` alone gives a container its GPUs and
        nothing else. bcd writes that vendor spec with the [`container-toolkit` unit](/docs/daemon/host-config#container-toolkit)
        when a profile names it; on a server you adopted, install the NVIDIA container toolkit and generate the same spec once,
        as the unit does:

        ```bash
        nvidia-ctk cdi generate --format=json --device-name-strategy=uuid > /etc/cdi/nvidia.json
        ```

        Without a vendor spec a hold carries only `NVIDIA_VISIBLE_DEVICES`. Any CDI-aware runtime then takes the names as
        devices. For example, with Podman, placing the container in the hold's slice:

        ```bash
        set -a; . /run/bcd/allocations/alloc_….env; set +a
        podman run --rm --device bcd.dev/gpu=GPU-… --device bcd.dev/gpu=GPU-… \
          --cgroup-parent "$BCD_CGROUP_PARENT" nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L
        ```

        bcd does not move processes into the slice and never drops a hold on its own, not even while disconnected; the
        workload manager (containerd, Slurm, a wrapper) places the workload. For Kubernetes, reserve with `--kubernetes` on a
        pool with `policy.kubernetes`, and bcd's device plugin hands the held GPUs to the kubelet as `bcd.dev/gpu` (see
        [Kubernetes workers](/docs/guides/capacity/kubernetes)). The node side is described in
        [device assignments](/docs/daemon/delivery#device-assignments).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Follow the events [#follow-the-events]

        Every step above published events through the transactional outbox. With a host that mounts the events module's
        `GET /watch` (the examples do), follow them live, one JSON line each:

        ```bash
        better-compute events watch --types 'node.*,reservation.*'
        ```

        Run it in a second terminal before step 7 to see the reservation move: `reservation.placed`, `reservation.binding`,
        `reservation.active`, and after step 10 `reservation.released`. Each line carries a `sequence`; `--since <sequence>`
        resumes after it. Without a `/watch` route, page through the same events with `better-compute events list` or
        `better-compute nodes events $NODE`. See [watch](/docs/guides/events/watch) and the
        [event reference](/docs/reference/events).
      </div>
    </div>
  
      <div className="fd-steps">
      <div className="fd-step">
        ### Release [#release]

        ```bash
        better-compute release $RSV
        better-compute capacity --pool $POOL
        ```

        `release` calls [`reservations.release`](/docs/reference/api/reservations#release): the bookings end at the current
        database time, the quota is refunded, and the device assignment leaves the node's desired state. bcd removes the CDI
        spec and the env file, reports `AllocationReleased` signed with its node key, and stops the hold's slice once it is
        empty, so a release never kills a running workload: stop it first. `available` goes back up. A reservation that has
        not started is withdrawn with `release --cancel`; one that should run longer is extended with
        `better-compute extend $RSV --by 2h` ([start, lease, extend and release](/docs/guides/capacity/reservations#start-lease-extend-and-release)).
      </div>
    </div>
  
## Commands used [#commands-used]

| Step | Command                                                                                                                                                  | API method                                               |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| 1    | [`api system.whoami`](/docs/reference/cli#api)                                                                                                           | `system.whoami`                                          |
| 2    | [`sites create`](/docs/reference/cli#sites-create), [`pools create`](/docs/reference/cli#pools-create), [`pools offer`](/docs/reference/cli#pools-offer) | `sites.create`, `pools.create`, `pools.offer`            |
| 3    | [`join-tokens create`](/docs/reference/cli#join-tokens-create)                                                                                           | `joinTokens.create`                                      |
| 4    | [`daemon install-script`](/docs/reference/cli#daemon-install-script), `pki show`; `bcd join`, `bcd node`                                                 | none (local), then JoinService's `Join` RPC              |
| 5    | [`nodes get`](/docs/reference/cli#nodes-get), `nodes list`, `nodes events`, `api nodes.adopt`; `bcd status`                                              | `nodes.get`, `nodes.list`, `nodes.events`, `nodes.adopt` |
| 6    | [`capacity`](/docs/reference/cli#capacity)                                                                                                               | `capacity.*`                                             |
| 7    | [`reserve`](/docs/reference/cli#reserve), `reservations get`                                                                                             | `reservations.create`, `reservations.get`                |
| 9    | [`events watch`](/docs/reference/cli#events-watch), `events list`                                                                                        | `GET /watch`, `events.list`                              |
| 10   | [`release`](/docs/reference/cli#release), `extend`                                                                                                       | `reservations.release`, `reservations.extend`            |

## Cleaning up [#cleaning-up]

To take the server out again, cordon it so nothing new places on it, let its reservations end, then delete the
node, which revokes its certificate and frees its hardware key for a later join:

```bash
better-compute nodes cordon $NODE --reason 'tutorial done'
better-compute nodes delete $NODE --yes
```

Then stop and disable `bcd.service` on the server and remove `/etc/bcd` and `/var/lib/bcd`; bcd has no uninstall step
([removing bcd](/docs/daemon/install#removing-bcd)). `better-compute pools archive $POOL` retires the pool. See
[removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host).

## Next steps [#next-steps]

  - [Bare metal](/docs/guides/bare-metal): Enroll servers by their BMCs and let metal wipe and image them instead of adopting them by hand.

  - [Fractional GPUs](/docs/guides/capacity/fractional-gpus): MIG instances and time-sliced or MPS slots in place of whole GPUs.

  - [Host configuration](/docs/daemon/host-config): Have bcd install the driver, the container toolkit and the GPU settings for every machine of the pool.

  - [Quotas and usage](/docs/guides/capacity/quotas-and-usage): Cap what each tenant may book and bill what it used.


# Operating on AWS (/docs/operations/aws)

> CloudWatch metrics through EMF, bcd telemetry through an ADOT collector, Amazon Managed Service for Prometheus, and control-plane spans to X-Ray.



A deployment that runs on or next to AWS can send its telemetry to AWS's own services without an agent of its own.
Each path below is independent: pick the ones you need. The platform services themselves (Secrets Manager, KMS,
S3, EventBridge and the rest) are in the `@better-compute/aws` README.

## Control-plane metrics in CloudWatch (EMF) [#control-plane-metrics-in-cloudwatch-emf]

`emfMetricsExporter` reads the metrics registry and writes one CloudWatch Embedded Metric Format document per line
to standard output. On ECS, the `awslogs` log driver ships those lines to CloudWatch Logs, which extracts the
metrics. No `PutMetricData` call and no IAM permission for metrics is needed.

```ts
import { createMetrics } from '@better-compute/observability';
import { emfMetricsExporter } from '@better-compute/aws';

const registry = createMetrics();
const emf = emfMetricsExporter({ registry, namespace: 'BetterCompute' });
emf.start(); // every 60 s; emf.stop() on shutdown
```

* Counters, and the `_sum` and `_count` of histograms, are sent as the change since the last flush. The first flush
  only records a baseline. Gauges are sent as their value. Histogram buckets are not sent.
* Only the labels named in `dimensions` (default `module`, `method`, `code`, `pool`, `kind`) become CloudWatch
  dimensions. Series that differ only in other labels are summed, which keeps the metric count bounded.
* Metric names drop the registry's `better_compute_` prefix. A document holds at most 100 metrics; more go into
  further documents.
* Tenant labels never reach CloudWatch: the registry refuses them when a metric is defined.

## bcd telemetry through an ADOT collector [#bcd-telemetry-through-an-adot-collector]

bcd exports host and instance telemetry over OTLP when the node's `TelemetrySpec` has an `otlpEndpoint`. Point it at
an AWS Distro for OpenTelemetry collector (a sidecar, a DaemonSet or a gateway). Put the collector's auth headers in
a secret and set `headersRef` to its `secret://` ref: bcd fetches it through the secret broker, so the headers are
never stored in the node spec. The collector then forwards to CloudWatch, X-Ray or Amazon Managed Service for
Prometheus with its own IAM role.

## Amazon Managed Service for Prometheus [#amazon-managed-service-for-prometheus]

Run an ADOT collector with a Prometheus receiver that scrapes the control plane's `GET /metrics` with the bearer token
`metricsHandler` was given, and a `prometheusremotewrite` exporter with SigV4 auth to the workspace's remote-write
URL. Amazon Managed Grafana reads the workspace. The token is a secret: give it to the collector from Secrets
Manager, not from its configuration file.

## Control-plane spans to X-Ray [#control-plane-spans-to-x-ray]

Pass an `onSpan` observer that turns each `ComputeSpan` into an OpenTelemetry span and exports it to the ADOT
collector over OTLP. The collector's `awsxray` exporter sends them to X-Ray. Spans carry `tenantId` and
`requestId`; drop or hash `tenantId` in the collector if tenant ids must not leave the deployment.


# Backup and recovery (/docs/operations/backup-and-recovery)

> What state lives where (PostgreSQL, the PKI bundle and deployment secret, operator-held keys, bcd's data directory on each node), what to back up, the order to restore in, what a restore cannot bring back, and how nodes rejoin after a loss.



Better Compute keeps almost everything in one PostgreSQL database, a few things in files that only operators hold,
and a small amount of state on every node. This page lists each piece, what losing it costs, and how to get back.

> **Nothing here is automated or tested.** 
  The repository has no backup or restore tooling, and no test restores a database under a running fleet. What follows
  describes the mechanisms that exist and what the code does with restored state; everything marked
  **Recommendation** is advice, not something the code enforces. The consequences of a restore listed under
  [what a restore cannot bring back](#what-a-restore-cannot-bring-back) are read from the code paths, not observed.

## What lives where [#what-lives-where]

### On the control plane [#on-the-control-plane]

| State                                                                                                                                                                                        | Where                                                       | If it is lost                                                                             |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Every aggregate: tenants' reservations, allocations and bookings, pools, nodes, commands, workflows, instances, volumes' records, usage, audit events and checkpoints, releases and rollouts | PostgreSQL                                                  | Restore from backup. Nothing else holds it                                                |
| Runtime state: `leases`, `workers`, `work_queue`, `outbox` and its cursors, `node_sessions`, `gateways`                                                                                      | PostgreSQL                                                  | Rebuilt by running processes; worth nothing on its own                                    |
| Node certificates (`node_certs`), join tokens, join requests, node bindings                                                                                                                  | PostgreSQL                                                  | Restore; see [certificates issued after the backup](#what-a-restore-cannot-bring-back)    |
| Secrets in the default store (`sealed_secrets`, `secret_refs`)                                                                                                                               | PostgreSQL, sealed under the deployment secret              | Unreadable without the secret, even from a good backup                                    |
| The published trust bundle's version and history (`trust_bundle_state`, `trust_bundle_history`)                                                                                              | PostgreSQL                                                  | Restore; see [the trust bundle](#the-trust-bundle-version)                                |
| The PKI bundle: root, node, serving and internal CAs, JoinService keys, command-signing root, exec ticket key, SSH CAs, audit signing key                                                    | One JSON file, `pki.json` by default (`BETTER_COMPUTE_PKI`) | Cannot be recreated: a new `pki init` means new CAs and new pins, and every node rejoins  |
| The deployment secret                                                                                                                                                                        | `BETTER_COMPUTE_SECRET`, in every process's environment     | Nothing sealed under it opens: the PKI bundle, sealed secrets, webhook secrets            |
| The offline root key                                                                                                                                                                         | The file `pki init --offline-root` wrote, if you used it    | `pki renew-ca` needs it when the bundle holds no root                                     |
| Gateway serving and internal certificates                                                                                                                                                    | Files from `pki issue-gateway`                              | Reissue from the bundle                                                                   |
| Better IAM's data                                                                                                                                                                            | Better IAM's own store                                      | Compute keeps only a projection (`authz_projection`) and an inbox; see [IAM](#better-iam) |
| The configuration                                                                                                                                                                            | `better-compute.config.ts` and its environment              | Yours, in source control                                                                  |
| bcd binaries served to nodes                                                                                                                                                                 | `downloads.dir` or `downloads.blob`                         | Copy them again from the release                                                          |

### Held by people, outside the deployment [#held-by-people-outside-the-deployment]

| Key                           | Created by                                                   | If it is lost                                                                                                                                                                                                                                        |
| ----------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Release signing keys          | `better-compute release keygen`                              | Releases can no longer be signed under the pinned keys. New keys mean new pins in `rolloutModule({ release: { pins } })` and in every node's `[update] release_pins` (below)                                                                         |
| Approver keys                 | `better-compute batch keygen`                                | With fewer than the threshold left, no batch can be approved and no quorum change can be signed; daemons that hold the set then need re-provisioning or re-pinning (see [changing the approver set](/docs/operations/pki#changing-the-approver-set)) |
| Workload identity issuer keys | `createWorkloadIssuer`, stored by you through `exportKeys()` | New keys: verifiers must re-read the JWKS                                                                                                                                                                                                            |

### On each node [#on-each-node]

Under bcd's `data_dir` (`/var/lib/bcd` by default) and `/etc/bcd`; the full layout is in
[files on the host](/docs/daemon/install#files-on-the-host).

| File                                                                   | Holds                                                                                | If it is lost                                                                                                                         |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `/etc/bcd/config.toml`, `join-encryption.jwk`                          | Gateway URL, site, the three pins, `[update]` release pins and floor, other sections | Re-run the install script                                                                                                             |
| `node.key`                                                             | The node key, or only the record of a key in the TPM                                 | The node's identity: it must join again                                                                                               |
| `identity.json`                                                        | Certificate chain, trust bundle (with the approver quorum), the server time mark     | The same                                                                                                                              |
| `journal.db`                                                           | The command journal (bbolt, every write fsynced), unacked events and results         | The control plane fails in-flight non-retry-safe commands `journal-lost`; see [journal loss](/docs/daemon/failure-modes#journal-loss) |
| `volumes/<id>.raw`                                                     | Tenants' persistent volumes, as sparse raw files on this host only                   | Gone: the code keeps no replica and no backup of volumes                                                                              |
| Instance disks under `data_dir/libvirt` or `data_dir/cloud-hypervisor` | qcow2 overlays of running instances                                                  | Gone with the host                                                                                                                    |
| `bin/`, `update.json`, `slots.json` (or root's `[update] slots_dir`)   | Self-update slots and the release floor                                              | Re-run the install script                                                                                                             |
| `batch-leaves.json`, `unbatched-destructive.json`                      | The quorum's used leaves and unbatched count                                         | Rebuilt empty; see the warning below                                                                                                  |
| `/var/lib/bcd-hostconfig`                                              | Host units' records and the helper's accepted signed spec                            | Re-applied from the next spec                                                                                                         |

## Backing up [#backing-up]

### PostgreSQL [#postgresql]

Use PostgreSQL's own tools. Nothing in Better Compute adds to or replaces them.

**Recommendation.**

* Continuous archiving with point-in-time recovery (base backups plus WAL), so you can choose the restore point and
  lose as little as possible. A nightly `pg_dump` alone loses up to a day, and [a day matters](#certificates-issued-after-the-backup)
  here because node certificates turn over every 14 hours or so.
* Back up the whole database, every schema and table: compute's base schema and every module's tables share it, and
  `compute_migrations` must stay consistent with them.
* Keep the backups away from the database host, and test a restore on a scratch server regularly. `better-compute
  doctor` against the restored copy is a quick sanity check: it reports a database that was never migrated, missing
  migrations of this release, a missing `btree_gist`, and a `compute_now()` that does not read the database clock.
* Back up Better IAM's store at the same moment, so tenants and grants match the compute data that refers to them.

Two things in the database need care beyond that:

* **Detached history partitions.** The retention module and the audit archiving step can detach monthly partitions
  of `audit_events`, `inventory_snapshots` and `usage_windows`; a detached partition is a standalone table you export
  and drop yourself. Keep those exports with your backups. See [archiving](/docs/guides/events/audit#archiving) and
  [history partitions](/docs/operations/retention#history-partitions).
* **An audit copy outside the database.** An audit sink copies every committed `audit_events` row out in commit
  order (see [audit sinks](/docs/guides/events/audit#copy-it-out-with-an-audit-sink)). It is the record that survives
  a database restore.

### The PKI bundle and the secret [#the-pki-bundle-and-the-secret]

Back up the bundle and `BETTER_COMPUTE_SECRET` together, and keep the offline root key apart from both. The bundle
file is sealed: its private keys are encrypted under a key derived from the secret, and the seal also covers every
certificate and pin, so a copy without the secret is useless and a copy changed without the secret no longer loads
(`INVALID_CONFIG`: "restore it from a backup"). See [PKI](/docs/operations/pki#pki-init) and
[the deployment secret](/docs/operations/configuration#the-deployment-secret).

Take a new copy whenever the bundle changes: `pki rotate-command-root`, `pki renew-ca`, `pki add-sandbox-ca` and
`pki reseal` all write it back. A bundle restored from before a command-signing root rotation lacks the new root,
and daemons that already trust only the new one would refuse every command signed under the old.

### Nodes [#nodes]

**Recommendation.** Do not back up bcd's state files to restore them later. Recover a node by joining it again
instead (below). Restoring an old copy is worse than losing the file:

* an old `journal.db` keeps its `journal_id`, so the control plane believes the journal is intact and replays
  in-flight commands into a journal that no longer knows they ran: the `journal-lost` rule, which exists to stop a
  second execution, would not fire;
* old `batch-leaves.json` and `unbatched-destructive.json` forget leaves and slots already used, so an approved batch
  leaf could run again;
* an old `update.json` lowers the release floor that keeps the node from going back to an older binary.

Tenants' volumes and instance disks are the exception: they are tenant data, and the code does not replicate them.
Whether and how to back them up is a decision for your service; nothing in bcd snapshots a volume.

## Restore order [#restore-order]

      ### Stop everything that writes [#stop-everything-that-writes]

    Stop every api, worker and gateway process of the deployment. Daemons can stay up: they back off, keep enforcing
    their last spec and keep results and events in their journals (see
    [PostgreSQL unavailable](/docs/daemon/failure-modes#postgresql-unavailable)).
  
      ### Restore the secret, the bundle and the configuration [#restore-the-secret-the-bundle-and-the-configuration]

    Every process needs the same `BETTER_COMPUTE_SECRET` and the PKI bundle the nodes pin. Without them, nothing below
    helps: see [losing the PKI](#the-pki-bundle-or-the-secret-is-lost).
  
      ### Restore PostgreSQL [#restore-postgresql]

    Restore into a server that meets the [requirements](/docs/operations/hardware-requirements#postgresql): PostgreSQL 16
    or newer with `btree_gist` available. Restore Better IAM's store to the same point.
  
      ### Migrate with the release that took the backup, or a newer one [#migrate-with-the-release-that-took-the-backup-or-a-newer-one]

    ```bash
    better-compute migrate
    ```

    `migrate` refuses a database that a newer release migrated, naming the migrations it does not know, so run the same
    or a newer release than the one that wrote the backup. A newer release applies its own migrations on top, as in an
    [upgrade](/docs/operations/upgrades#step-1-migrate).
  
      ### Cordon, then start the control plane [#cordon-then-start-the-control-plane]

    **Recommendation:** cordon every node before the workers start placing work, so nothing new lands on devices whose
    state the restored database may not reflect:

    ```bash
    better-compute nodes cordon <node> --tenant <tenant> --reason 'database restored'
    ```

    Then start workers, api and gateways. Daemons reconnect on their own.
  
      ### Reconcile [#reconcile]

    Work through [what a restore cannot bring back](#what-a-restore-cannot-bring-back): re-revoke, revoke tokens, rejoin
    nodes whose certificates the database does not know, raise the trust bundle version if needed, and check each node's
    generation before uncordoning it. `better-compute doctor` reports `fleet.cert-expiry` for connected nodes with no
    certificate valid for another hour.
  
## What a restore cannot bring back [#what-a-restore-cannot-bring-back]

A restore rewinds the database, but not the nodes, the operators' keys or the outside world. Everything below follows
from how the code treats state it does not recognise.

### Work done after the backup [#work-done-after-the-backup]

Reservations, commands, workflows, instances and audit events written after the backup are gone from the control
plane. A daemon still holding unacked results replays them at its next `Hello`; events it had already had
acknowledged were deleted from its journal and are lost. Anyone holding an audit checkpoint signed after the backup
can tell the trail was rewound (see [consistency proofs](/docs/guides/events/audit#consistency-proofs)).

### Desired state ahead of the database [#desired-state-ahead-of-the-database]

bcd applies a desired state only when its generation is above the one it last applied
(`daemon/internal/agent/spec.go`), and a gateway sends a spec only when the node's applied generation is below the
database's. A node that applied generations written after the backup therefore ignores the restored specs and keeps
enforcing its newer one, its holds, instances, volumes and access included, until the control plane's generation for
it passes the node's. The specs that follow are built from the restored state, and the instance converger treats its
section as the whole desired set: an instance created after the backup is removed, and a volume the section no longer
lists and no running VM has open is destroyed, punched and unlinked (see [persistent volumes](/docs/daemon/delivery#persistent-volumes)).
Nothing in the code detects or warns about this gap.

**Recommendation:** keep the node cordoned and compare its `observed_generation` and what it runs (`bcd status`) with
the restored records before letting new specs reach it.

### Certificates issued after the backup [#certificates-issued-after-the-backup]

Node certificates live 24 hours and are renewed at about 60% of that, so most nodes' current certificate is younger
than a backup taken more than about 14 hours earlier. Such a certificate is missing from the restored `node_certs`:

| Path                          | With a certificate the database does not know                                                    |
| ----------------------------- | ------------------------------------------------------------------------------------------------ |
| Session (`Hello`)             | Accepted: an unrecorded serial is treated as unrestricted, unless the node row itself is missing |
| `Renew`                       | Refused: the presented leaf must be byte for byte a `node_certs` row                             |
| Secret requests               | Refused: the broker joins the certificate's serial to `node_certs`                               |
| `x509pop` re-join             | Refused for the same reason                                                                      |
| Attested bindings and results | Can be rejected: they are verified against the node's certificate in `node_certs`                |

The node keeps working until its certificate expires, then loses its session. Rejoin it before that (below). Nodes
that joined for the first time after the backup have no row at all and are refused at `Hello`; they join again as
new machines, or reclaim their old id if the restored database still holds their hardware key.

### Revocations and consumed tokens [#revocations-and-consumed-tokens]

Revocations made after the backup are undone: a certificate you revoked is valid again in the restored
`node_certs`. Re-run them (`nodes.revokeIdentity`; see [revoking a node's identity](/docs/guides/nodes/operating-nodes#revoking-a-nodes-identity)).
Join tokens used after the backup have their uses back, and tokens revoked after it are live again: revoke every
token that should not be usable (`better-compute join-tokens list`, then `join-tokens revoke`).

### The trust bundle version [#the-trust-bundle-version]

A daemon takes a trust bundle only when its version is above the one it holds. The restored `trust_bundle_state`
may be below what daemons already hold (for example after a command-signing root rotation that happened after the
backup), and then the bundle the restored deployment publishes is ignored. `pki.trustVersion` is a floor: set it at or
above the highest `node_sessions.trust_version` the reconnected daemons report, and the next published bundle reaches
them. `better-compute pki trust-version` shows how many connected daemons are behind. See
[signed intent](/docs/operations/pki#signed-intent).

### Releases [#releases]

A release registered after the backup is missing from `daemon_releases`. Nodes that confirmed it hold a release floor
at its sequence and refuse every older binary, so a rollout of an older registered release fails on them. Register
the newer manifest again; `releases.register` accepts it because its sequence is above every registered one.

## Rejoining a node [#rejoining-a-node]

The same steps cover a node whose disk was lost, a node whose certificate the restored database does not know, and a
node whose certificate expired while it was offline.

1. **A certificate expired less than 7 days ago**, with the `x509pop` attestor configured and the certificate in
   `node_certs`, re-joins with its old key. See [renewal](/docs/guides/nodes/joining#renewal).
2. **Otherwise**, move the old identity aside and join again, as root:

   ```bash
   bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt     # with a token
   bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm       # or with the TPM
   ```

   `--reset` renames `identity.json`, `node.key`, a pending `node.key.next` and `journal.db` to
   `<name>.reset-<UTC time>` and deletes nothing. See [joining again with --reset](/docs/daemon/install#joining-again-with---reset).
3. **Same node or a new one.** A machine whose trusted hardware key the database already holds reclaims its node id
   with a token that has `allowReclaim` (identity generation up by one, older certificates revoked); a `tpm` join
   with the same endorsement key reclaims the node bound to it. Without either, delete the old node first and the
   machine joins as a new one. See [placeholder hardware and reclaims](/docs/guides/nodes/joining#placeholder-hardware-and-reclaims).
4. **A lost disk** also lost `config.toml` and the binary: re-run the install script with a token, which installs,
   joins and starts bcd in one go.

The new journal makes the control plane fail the node's in-flight non-retry-safe commands with `journal-lost`, and
their owners re-decide; see [resume rules](/docs/daemon/sessions#resume-rules).

## Disaster scenarios [#disaster-scenarios]

| Scenario                                                       | What happens                                                               | Recovery                                                                                                                                                |
| -------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| An api, worker or gateway host is lost                         | Leases move to the survivors; nodes redial                                 | Nothing to restore; see [high availability](/docs/operations/high-availability)                                                                         |
| The PostgreSQL primary fails, a synchronous standby takes over | The fleet stalls for the failover and resumes                              | Nothing to restore                                                                                                                                      |
| Failover to an asynchronous standby that lost commits          | A small restore, for the lost transactions                                 | [What a restore cannot bring back](#what-a-restore-cannot-bring-back), limited to that window                                                           |
| The database is lost                                           | —                                                                          | [Restore order](#restore-order)                                                                                                                         |
| A migration went wrong                                         | `migrate` stops at the failing migration, which rolled back                | Fix and re-run; to go back to the previous release, restore the pre-upgrade backup with it (see [rolling back](/docs/operations/upgrades#rolling-back)) |
| A node's disk is lost                                          | The node goes `disconnected`, then `unreachable`; nothing is released      | [Rejoin it](#rejoining-a-node); its volumes and instance disks are gone                                                                                 |
| A node's journal is corrupted                                  | bcd moves it aside (`journal.db.corrupt-<unix time>`) and starts a new one | Nothing; `journal-lost` settles in-flight commands                                                                                                      |
| A node's TPM is cleared                                        | The node key no longer loads; renewals fail                                | `bcd join --reset --yes --attestor tpm`; see [TPM cleared](/docs/daemon/failure-modes#tpm-cleared-or-unavailable)                                       |
| A release signing key is lost                                  | —                                                                          | New keys and pins; see [below](#a-release-signing-key-is-lost)                                                                                          |

### The PKI bundle or the secret is lost [#the-pki-bundle-or-the-secret-is-lost]

There is no recovery that keeps the fleet. A bundle that is lost, or no longer opens because the secret changed,
cannot be recreated: `pki init` makes new CAs and new JoinService keys, so every gateway certificate, every node
certificate and every pin changes.

1. `better-compute pki init` with the same trust domain, and `pki issue-gateway` for every gateway.
2. New install scripts (`daemon install-script` reads the new pins) and join tokens.
3. On every node, as root: re-run the new script, then `bcd join --reset --yes` with a token. With the database
   intact, machines with a trusted hardware key reclaim their node ids through tokens with `allowReclaim`.
4. Every secret in the default sealed store is unreadable if the secret, not only the bundle, was lost: set them
   again.

### A release signing key is lost [#a-release-signing-key-is-lost]

Create new keys with `release keygen`, put their pins in `rolloutModule({ release: { pins } })`, and give every node
the new pins in `/etc/bcd/config.toml` under `[update] release_pins`. Re-running the install script does not do it:
when `config.toml` already has an `[update]` section, the script keeps it and prints that `--release-pins` was not
applied. Until a node has the new pins, it refuses every release signed by the new keys. Sign the next release with a
sequence above the last one, since nodes keep their release floor.

## Better IAM [#better-iam]

Compute stores no IAM tenants, principals or policies of its own. It keeps the authorization projection
(`authz_projection`, with its change feed `authz_changes`) that IAM's events maintain, and `iam_inbox`, the events it
still has to apply. Restoring compute and IAM to different points leaves the projection describing one and the
resources the other; &#x2A;*recommendation:** back both up at the same moment and restore both to it.

  - [Database](/docs/operations/database): Requirements, migrations and the runtime tables.

  - [PKI](/docs/operations/pki): The bundle, rotations and losing the bundle.

  - [Failure modes](/docs/daemon/failure-modes): Journal loss, TPM, certificate expiry.

  - [Upgrades](/docs/operations/upgrades): Why a migration can only be undone by a restore.


# Configuration (/docs/operations/configuration)

> The better-compute.config file, every betterCompute() option with its default, the host export that serve runs, environment variables and the deployment secret.



A deployment is configured in one TypeScript or JavaScript module that every process loads. Its default export
holds the options of `betterCompute()`; its named `host` export holds what `better-compute serve` runs around the
control plane. Values that differ between environments come from environment variables.

## The configuration file [#the-configuration-file]

The CLI looks for the configuration in this order:

1. `--config PATH`;
2. `BETTER_COMPUTE_CONFIG`;
3. the nearest `better-compute.config.mjs`, `.js`, `.ts` or `.mts`, in the working directory and then each parent.

With no file, `BETTER_COMPUTE_DATABASE_URL` and `BETTER_COMPUTE_SECRET` describe a minimal deployment on their own
(`configFromEnv`): the store from the URL, `BETTER_COMPUTE_BASE_URL` (default `http://localhost:3000`), root tenant
`root`, and no IAM, so only public methods answer. A `.ts` configuration needs Node.js 22.18 or newer, which strips
types by default. `better-compute init` writes a starter file and never overwrites one.

The module is executable code and is trusted as such. It can export four things:

| Export     | What it is                                                                                                                                                                                                                                                                                                                                                     |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `default`  | `defineConfig(options)` or `defineConfig(factory)`: the options of `betterCompute()`                                                                                                                                                                                                                                                                           |
| `host`     | `defineHost(config)` or `defineHost(factory)`: routes, metrics, tunnels, IAM handlers and the gateway role for `serve`                                                                                                                                                                                                                                         |
| `commands` | Project commands made with `defineCommand`, run as `better-compute <name>`                                                                                                                                                                                                                                                                                     |
| `cli`      | `{ defaults: { [command]: { flag: value } } }`, keyed by command name or `'*'` for every command: flag defaults, used when neither the flag nor its environment variable is set. A key other than `defaults`, a scope that names no command, or a flag its command does not have (under `'*'`: that no command has) is `INVALID_CONFIG`, with the nearest name |

Prefer the factory form. Importing the module then opens nothing; the CLI calls the factory with
`{ command, env, cwd }`, and your own server calls it through `configOptions(config)`:

```ts title="better-compute.config.ts"
import { postgres } from 'better-compute/postgres';
import { pglite } from 'better-compute/pglite';
import { defineConfig } from 'better-compute/server';

export default defineConfig(({ env = process.env, cwd = process.cwd() } = {}) => ({
  database: env.BETTER_COMPUTE_DATABASE_URL
    ? postgres({ url: env.BETTER_COMPUTE_DATABASE_URL })
    : pglite({ dataDir: `${cwd}/.better-compute-pglite` }),
  secret: env.BETTER_COMPUTE_SECRET ?? '',
  baseURL: env.BETTER_COMPUTE_BASE_URL ?? 'http://localhost:3000',
  rootTenantId,
  iam: bridge.embedded(iam, { rootTenantId }),
  modules: [...bridge.modules /* , metal(), vps(), … */],
}));
```

```ts title="server.ts"
import config from './better-compute.config';
import { betterCompute, configOptions } from 'better-compute/server';

const compute = betterCompute(await configOptions(config));
```

## betterCompute() options [#bettercompute-options]

`betterCompute()` is synchronous and side-effect free: it validates the options and registers the method table.
Anything wrong throws `INVALID_CONFIG` before a connection is opened.

### Required [#required]

| Option     | Type          | Notes                                                                                                                                                                 |
| ---------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database` | `SqlExecutor` | `postgres({ url })` or `pglite()`. Must implement `query`, `transaction` and `migrate`. See [database](/docs/operations/database).                                    |
| `secret`   | `string`      | At least 32 characters, the same in every process. See [the deployment secret](#the-deployment-secret).                                                               |
| `baseURL`  | `string`      | The API's public origin, such as `https://compute.example.com`. HTTP or HTTPS, no credentials, query or fragment. Cookie requests must come from exactly this origin. |
| `iam`      | `IamPort`     | `bridge.embedded(iam, …)` from the IAM bridge. Must implement `authenticate`, `require` and `authorize`. See [Better IAM](/docs/guides/iam).                          |

### Optional [#optional]

| Option                              | Default                            | What it does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ----------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rootTenantId`                      | `iam.rootTenantId`                 | The IAM root tenant. Required when the port does not name it; when both are given they must agree. Modules read it from `context.iam.rootTenantId`, which names it either way.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `basePath`                          | `/api/compute`                     | Where the handler is mounted. An absolute path without a trailing slash.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `typePrefix`                        | `compute-`                         | IAM resource type prefix, the same as the IAM bridge's.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `modules`                           | `[]`                               | Injected modules (`provisionModule()`, `metal()`, `vps()`, `eventsModule()`, …), registered after the built-ins.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `plugins`                           | `[]`                               | Third-party modules mounted under `plugins/{id}`, with actions `compute:<id>:*`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `drivers`, `providers`, `attestors` | `[]`                               | Extensions: bare-metal drivers, VPS providers, join attestors beyond `join-token`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `scheduler`                         | none                               | Placement plugins, scorer `weights` and `topN` (default 5) for the built-in capacity module.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `capacity`                          | see below                          | Options of the built-in capacity module.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `pki`                               | none                               | CAs, JoinService keys and command signing. See [pki](#pki).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `secrets`                           | the local sealed store             | `SecretStore`s by URI scheme. Without any, secrets are sealed in `sealed_secrets` under `secret`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `tunnels`                           | none                               | Exec tickets and break-glass SSH (v0.3). Without it neither `tunnels.open` nor `nodes.issueSshCert` exists. See [tunnels](#tunnels).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `notifier`                          | none                               | Best-effort wake-ups after commit, such as `gatewayNotifier({ sql, tls })` from `better-compute/gateway`. Never a source of truth.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `sinks`                             | none                               | `{ events?, audit? }`. Each sink needs an `id` unique among its kind: its cursor is stored under it. Audit sinks add the built-in `audit-sinks` module. `usage` with any entry is `INVALID_CONFIG`: usage goes to IAM billing through the bridge. Sinks a module lists in `extensions` join these. See [sinks](/docs/extending/sinks).                                                                                                                                                                                                                                                                                                                                                        |
| `clock`                             | anchored to `compute_now()`        | Business time. Tests pass the sim clock.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `acceptIamCookie`                   | `false`                            | Accept the IAM session cookie on same-origin requests (a console on one origin).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `log`                               | JSON warnings and errors on stderr | A `Logger` with `debug`, `info`, `warn` and `error`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `iamJobs`                           | `[]`                               | Better IAM's jobs for the `iam` worker role: `{ name, everyMs, run }`, `everyMs` at least 100. Usually filled by the IAM bridge.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `worker`                            | see below                          | Worker runtime tuning.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `ports`                             | built-in                           | Replace `ctx.capacity` or `ctx.inventory`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `downloads`                         | none                               | `{ dir?, blob? }` (at least one): where `GET {basePath}/downloads/bcd/{version}/{os}-{arch}` finds `bcd` binaries, and `GET {basePath}/downloads/bcx-ipmi/{version}/{os}-{arch}` the `bcx-ipmi` plugin archive. The examples set `dir` from `BETTER_COMPUTE_DOWNLOADS_DIR` (5792b2b; see [examples](/docs/operations/deployment#examples)). See [installing bcd](/docs/daemon/install#serving-the-binaries).                                                                                                                                                                                                                                                                                  |
| `supportBundles`                    | none                               | `{ blob, maxBytes? }`: a `BlobStore` with `get` and `presign`, and the largest upload (a positive integer, default 256 MiB). `commands.run` `logs.collect` then hands the daemon a presigned PUT for `support-bundles/{nodeId}/{key}.bundle`. Without it `logs.collect` has no upload destination. The CLI host does not set it. See [support bundles](/docs/guides/nodes/commands#support-bundles).                                                                                                                                                                                                                                                                                          |
| `gateway`                           | none                               | `{ url?, serverName?, publicUrls?, maxStreams? }`. With `url` (https, no credentials, query or fragment; `INVALID_CONFIG` otherwise) the server publishes where daemons dial in `/.well-known/better-compute` as `gateway: { url, serverName?, servingCaPin? }`, the pin computed from `pki.servingCa`, so `bcd join --server <control plane>` on a machine without `config.toml` finds the gateway (a host build still checks the document against the pins its install brings). `serverName` is the TLS name when it differs from the URL's host (a DNS name). `publicUrls` and `maxStreams` are not read. See [installing bcd](/docs/daemon/install#joining-from-the-well-known-document). |

### capacity [#capacity]

| Option             | Default                                                                | What it does                                                                                                                                                                                                                                                                                       |
| ------------------ | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `placer`           | `inline`                                                               | `inline` places a reservation in the admission transaction. `pool-lease` (v0.4) queues it; the worker owning the pool places up to `maxBatch` at a time and `reservations.create` answers `pending` with a `Queued` condition, or waits for the batch when the request sets `wait: { timeoutMs }`. |
| `maxBatch`         | 64                                                                     | Reservations per pool-lease batch, at most 64.                                                                                                                                                                                                                                                     |
| `maxAttempts`      | 3                                                                      | Placement attempts per admission.                                                                                                                                                                                                                                                                  |
| `rollupEvery`      | `5m`                                                                   | How often the rollup job rebuilds every pool.                                                                                                                                                                                                                                                      |
| `usageEvery`       | `5m`                                                                   | How often closed hourly usage windows are written.                                                                                                                                                                                                                                                 |
| `spotNoticeMs`     | 2 minutes                                                              | Notice a preempted spot reservation gets.                                                                                                                                                                                                                                                          |
| `attestedBindings` | `pkiAllocationBindingCheck()` with `pki.commandSigning`, otherwise off | An `AllocationBindingCheck`: a reservation's allocation counts as bound only once the node's own signed `AllocationBound` verifies. See [attested bindings](/docs/guides/capacity/reservations#attested-bindings).                                                                                 |

### worker [#worker]

| Option        | Default      | Range                    | What it does                                                                |
| ------------- | ------------ | ------------------------ | --------------------------------------------------------------------------- |
| `id`          | `wkr_<ulid>` | `[A-Za-z0-9._:-]`, 1-128 | This process's worker id and lease owner                                    |
| `batchSize`   | 32           | 1-1,000                  | Keys a reconciler claims per pass                                           |
| `concurrency` | 4            | 1-64                     | Keys one reconciler runs at once                                            |
| `pollMs`      | 1,000        | 10-60,000                | Wait when there was nothing to do                                           |
| `claimTtlMs`  | 60,000       | 1,000-3,600,000          | How long a claimed key stays claimed                                        |
| `resyncMs`    | 300,000      | 1,000-86,400,000         | Owned keys are re-enqueued every `resyncMs` plus up to `resyncMs` of jitter |

These are fixed: shard and singleton leases live 15 s and renew every 5 s, a worker is live while its heartbeat is
younger than 30 s, `SYNC_AGAIN` brings a key back after 1 s, failures back off from 1 s to 5 minutes, and the
CommandSweeper runs every 5 s.

### pki [#pki]

| Option                    | What it does                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pki.nodeCa`              | The node CA. The gateway role takes its trust anchor and revocation set from it.                                                                                                                                                                                                                                                                                                                                                        |
| `pki.servingCa`           | The gateway serving CA. With `gateway.url`, its SPKI pin is published in `/.well-known/better-compute` as `gateway.servingCaPin` (read once; a failure is logged, the pin left out and retried a minute later). The gateway's own TLS still comes from `host.gateway`, issued with `pki issue-gateway`.                                                                                                                                 |
| `pki.internalCa`          | Accepted, but nothing reads it yet.                                                                                                                                                                                                                                                                                                                                                                                                     |
| `pki.join`                | JoinService's keys and node CA: `pkiJoinCrypto({ trustDomain, nodeCa, signingKey, encryptionKey })`. Without it, `Join` and `Renew` answer `INVALID_CONFIG`. Its public keys appear in `/.well-known/better-compute` as `join`.                                                                                                                                                                                                         |
| `pki.commandSigning`      | The command-signing root (v0.2). Every command is then signed with a 24 h worker key it certifies, every NodeSpec generation is signed as it is written (v0.4), daemons get its root in their trust bundle, and capacity's `attestedBindings` defaults to the pki check.                                                                                                                                                                |
| `pki.commandSigningRoots` | Roots still trusted during a rotation. Needs `commandSigning`.                                                                                                                                                                                                                                                                                                                                                                          |
| `pki.trustVersion`        | The trust bundle's lowest version (default 1). The published version follows the content: a changed bundle (roots, approvers, tiers) takes the next one by itself (`trust_bundle_state`). Setting it above the recorded version republishes a bundle as newer. Approver-set signatures do not depend on it: they sign `pki.approvers.epoch` (c6214dc); see [changing the approver set](/docs/operations/pki#changing-the-approver-set). |

The PKI bundle supplies all of these:

```ts
import { loadPkiBundle } from 'better-compute/cli';
import { pkiJoinCrypto } from 'better-compute/server';

const bundle = await loadPkiBundle('.better-compute/pki.json', env); // unsealed with BETTER_COMPUTE_SECRET
const pki = {
  nodeCa: bundle.nodeCa,
  join: pkiJoinCrypto({
    trustDomain: bundle.trustDomain,
    nodeCa: bundle.nodeCa,
    signingKey: bundle.joinSigning,
    encryptionKey: bundle.joinEncryption.privateKey,
  }),
  commandSigning: bundle.commandSigning,
};
```

See [PKI](/docs/operations/pki).

### tunnels [#tunnels]

| Option           | Default                                      | What it does                                                                                                                                                                                                                             |
| ---------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `issuer`         | required                                     | The exec ticket issuer: `execTicketIssuer(bundle, { issuer })` or `createExecTicketIssuer`, with `issuer` set to `baseURL`'s origin (the api accepts only tickets from it). Its JWKS goes into every node's NodeSpec and to the gateway. |
| `ssh`            | none                                         | The compute SSH CA (`bundle.sshCa`) for `nodes.issueSshCert`. Without it that method answers `UNSUPPORTED`.                                                                                                                              |
| `sshRequiresMfa` | `true`                                       | Break-glass certificates need an MFA session.                                                                                                                                                                                            |
| `sshAccounts`    | `['root']`                                   | Unix accounts a certificate may name; the principal is `<account>@<nodeId>`.                                                                                                                                                             |
| `limits`         | 10 tunnels a minute, 20 certificates an hour | Per identity and per api replica. Beyond them: `RATE_LIMITED` with `retryAfterMs`.                                                                                                                                                       |

The `api` role mounts the tunnel upgrade only when `tunnels.issuer` is set, and it needs a re-authorizer: an
`IamPort` with `authorizeSession` (the IAM bridge has one) or `host.tunnels.authorize`. See
[tunnels](/docs/guides/access/tunnels).

## The host export [#the-host-export]

`host` is what `serve` runs around the control plane. A factory receives `{ compute, options, env, cwd }`, so
routes can close over the same instances the modules use.

```ts title="better-compute.config.ts"
import { readFile } from 'node:fs/promises';
import { defineHost } from 'better-compute/cli';
import { handleWatch } from 'better-compute/events';
import { createMetrics } from 'better-compute/observability';

const metrics = createMetrics();

export const host = defineHost(async ({ compute, env }) => ({
  http: { routes: [{ path: '/watch', method: 'GET', handler: (request) => handleWatch(request, { sql: compute.context.sql, authorize }) }] },
  metrics: { registry: metrics, bearerToken: env.BETTER_COMPUTE_METRICS_TOKEN },
  gateway: {
    region: 'eu-1',
    listen: { port: 8443 },
    tls: {
      key: await readFile('.better-compute/gateway/gateway.key', 'utf8'),
      cert: await readFile('.better-compute/gateway/gateway.crt', 'utf8'),
      ca: bundle.nodeCa.export().certificatesPem,
    },
    internal: { url: 'https://gw-1.internal:8444', tls: internalServerTls },
  },
}));
```

| Field                                               | What it configures                                                                                                                                                                                                                                                                                                                                                                                         |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `http.routes`                                       | Fetch-style routes beside the API: `{ path, method?, prefix?, handler(request) }`                                                                                                                                                                                                                                                                                                                          |
| `http.upgrade`                                      | Upgrade handlers, tried after the tunnel handler                                                                                                                                                                                                                                                                                                                                                           |
| `http.wellKnown`                                    | Extra public fields of `/.well-known/better-compute`                                                                                                                                                                                                                                                                                                                                                       |
| `metrics.registry`                                  | The process's `createMetrics()` registry; `serve` adds the gateway role's series to it                                                                                                                                                                                                                                                                                                                     |
| `metrics.bearerToken`, `metrics.path`               | With a token of at least 16 characters, `GET /metrics` (or `path`)                                                                                                                                                                                                                                                                                                                                         |
| `tunnels.authorize`                                 | The tunnel re-authorizer (default `tunnelAuthorizer(iam)`)                                                                                                                                                                                                                                                                                                                                                 |
| `tunnels.control`                                   | Client TLS (`tls`, `timeoutMs`) for the gateways' internal control APIs                                                                                                                                                                                                                                                                                                                                    |
| `tunnels.reauthorizeMs`, `maxDurationMs`, `onClose` | Tunnel re-authorization period, lifetime and close hook                                                                                                                                                                                                                                                                                                                                                    |
| `iam.bridge`                                        | The IAM bridge whose inbox the workers drain: a revoked delegation closes the agent's tunnels on every gateway, and revocations drop IAM caches                                                                                                                                                                                                                                                            |
| `iam.streams`, `iam.handlers`, `iam.gatewayStaleMs` | This process's `createWatchRegistry()` registries, closed on revocation too; your own reactions in place of the defaults; gateways asked (heartbeat within 120 s by default). See [tunnels](/docs/guides/access/tunnels#closing-on-iam-revocation)                                                                                                                                                         |
| `streams`                                           | This process's own stream registries (such as the `createWatchRegistry()` your `GET /watch` route passes to `handleWatch`) that IAM revocations close at once; added to `iam.streams`, and unlike those also used with the embedded IAM of `dev --sim`. Anything without `closeWhere` is `INVALID_CONFIG`                                                                                                  |
| `iamReactions`, `iam.reactions`                     | Modules' reactions to IAM events (PLAN.md §12.8) beside capacity's: objects with `agent(tx, context, filter)` and/or `identity(tx, context, end)`, run by the default agent and identity handlers in the inbox transaction, such as vps's `{ agent: withdrawAgentSandboxes, identity: reassignSandboxes }`. `iamReactions` is also used with `dev --sim`'s embedded IAM. Anything else is `INVALID_CONFIG` |
| `iam.webhook`                                       | `{ secret, path? }`: remote IAM's signed event webhooks into `iam_inbox` at `POST /iam/events` (or `path`), for deployments that cannot read the IAM database. See [IAM events](/docs/guides/iam#iam-events-compute-reacts-to)                                                                                                                                                                             |
| `mcp`                                               | `{ path?, handler }`: the gated MCP endpoint (`computeMcpEndpoint(...).handler`), default path `/mcp`. See [MCP](/docs/guides/clients/mcp)                                                                                                                                                                                                                                                                 |
| `gateway.region`                                    | Required. The region, as in `sites.region`                                                                                                                                                                                                                                                                                                                                                                 |
| `gateway.listen`                                    | Daemon listener, default `0.0.0.0:8443`                                                                                                                                                                                                                                                                                                                                                                    |
| `gateway.tls`                                       | Required. `key` and `cert` from the serving CA, `ca` the node CA roots; any of them may come from `tlsFiles` instead                                                                                                                                                                                                                                                                                       |
| `gateway.tlsFiles`                                  | `{ key, cert, ca?, intervalMs? }`: file paths read at start and re-read every `intervalMs` (60 s); a renewed certificate is hot-reloaded for new handshakes. See [PKI](/docs/operations/pki#pki-issue-gateway)                                                                                                                                                                                             |
| `gateway.identity`                                  | `{ roots, trustDomain, revocations }`; default: `pki.nodeCa`, `pki.join.trustDomain` and the node CA's revocation set                                                                                                                                                                                                                                                                                      |
| `gateway.internal`                                  | `{ url, listen?, tls, tlsFiles? }`: the internal control API (default port 8444) over internal-CA mutual TLS; `url` is advertised in the `gateways` row. `tls` or `tlsFiles` must include `ca` (the internal CA), or `serve` refuses it: without it Node would accept any publicly issued client certificate                                                                                               |
| `gateway.ws`                                        | `{ publicHost, listen? }`: the v0.4 WebSocket fallback (default port 443)                                                                                                                                                                                                                                                                                                                                  |
| `gateway.options`                                   | Anything else `createGateway()` takes (`maxStreams`, rates, extra `reports`)                                                                                                                                                                                                                                                                                                                               |

`serve` checks the shape and refuses a bad one with `INVALID_CONFIG`; `doctor` checks the wiring. A key the table
does not list is refused too, at every level, with the nearest known one (`host.gatway is not a setting of host.
Did you mean host.gateway?`): a mistyped key is never dropped, since `serve` would then start without what it was
meant to configure. Objects handed on as they are (`gateway.tls`, `gateway.options`, `http.wellKnown`) take
whatever their consumer takes.

## Modules and their options [#modules-and-their-options]

Modules are values you construct and pass in `modules`, each with options of its own. The examples register:

```ts
const bareMetal = metal({ hardwareProfiles });
modules: [
  provisionModule({ modules: [bareMetal] }), // runs metal's workflows
  bareMetal,
  vps(),
  retentionModule(), // added by the CLI when missing
  eventsModule({ secret }), // seals webhook signing secrets under the deployment secret
  fleetModule(),
  rolloutModule({ release: { pins }, downloadsUrl }), // needs the release keys' pins
  auditModule({ signer: bundle.auditSigning }), // needs the bundle's audit key
];
```

Each module's own migrations run with `better-compute migrate` (see [database](/docs/operations/database)). Its
options are documented with the module; see [modules](/docs/reference/modules).

## Environment variables [#environment-variables]

The CLI reads these:

| Variable                                                           | Used by                                             | Meaning                                                                                                                                                                                                                     |
| ------------------------------------------------------------------ | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BETTER_COMPUTE_CONFIG`                                            | every command                                       | The configuration module, like `--config`                                                                                                                                                                                   |
| `BETTER_COMPUTE_DATABASE_URL`                                      | environment-only mode, `dev`, the template          | `postgres://…`, `pglite:DIR` or `pglite:memory`                                                                                                                                                                             |
| `BETTER_COMPUTE_SECRET`                                            | environment-only mode, `pki`, `audit`, the template | The deployment secret                                                                                                                                                                                                       |
| `BETTER_COMPUTE_BASE_URL`                                          | environment-only mode, the template                 | The public origin (default `http://localhost:3000`)                                                                                                                                                                         |
| `BETTER_COMPUTE_GATEWAY_URL`, `BETTER_COMPUTE_GATEWAY_SERVER_NAME` | environment-only mode, the examples                 | `gateway.url` and `gateway.serverName`, published in `/.well-known/better-compute` for `bcd join --server`. Environment-only mode has no PKI bundle, so no serving-CA pin is published; the examples add it from the bundle |
| `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED`                         | every bundle loader                                 | `1` loads a bundle sealed in the former format (or unsealed while a secret is set) until it is resealed; the commands that write the bundle refuse it. See [pki reseal](/docs/operations/pki#pki-reseal)                    |
| `BETTER_COMPUTE_ROLES`, `PORT`, `HOST`                             | `serve`                                             | `--roles` (default `all`), `--port` (default 3000), `--host` (default `0.0.0.0`)                                                                                                                                            |
| `BETTER_COMPUTE_PKI`                                               | `pki`, `doctor`, `audit`                            | The PKI bundle (default `.better-compute/pki.json`)                                                                                                                                                                         |
| `BETTER_COMPUTE_TRUST_DOMAIN`                                      | `pki init`                                          | `--trust-domain`                                                                                                                                                                                                            |
| `BETTER_COMPUTE_TOKEN`                                             | token commands                                      | IAM session token, API key or session JWT to act as                                                                                                                                                                         |
| `BETTER_COMPUTE_URL`                                               | token commands                                      | Call a running server instead of loading the configuration                                                                                                                                                                  |
| `BETTER_COMPUTE_TENANT`                                            | `reserve`, `capacity`, `audit`                      | The tenant                                                                                                                                                                                                                  |
| `BETTER_COMPUTE_AUDIT_PIN`                                         | `audit verify`                                      | The audit key pin                                                                                                                                                                                                           |
| `BETTER_COMPUTE_DEBUG`                                             | every command                                       | Print the stack trace of a failure                                                                                                                                                                                          |

Inside the configuration, variables are yours to name: the file reads `env` and decides. The CLI only guarantees
that `env` is the process environment.

## The deployment secret [#the-deployment-secret]

`secret` is at least 32 random characters, identical in every process of the deployment. Generate it once:

```bash
openssl rand -base64 48
```

It protects:

* **Secrets at rest.** Without `secrets`, `sealed_secrets` holds values sealed with AES-256-GCM under a key derived
  from it (migration `pki/0001_sealed_secrets`).
* **The PKI bundle.** `pki init` seals the private keys under a key derived from `BETTER_COMPUTE_SECRET`
  (HKDF-SHA256, AES-256-GCM); every process unseals the bundle with it.
* **Webhook signing secrets**, when you pass it to `eventsModule({ secret })` as the examples do.

> **Changing the secret.** 
  With the defaults, a new secret cannot open anything sealed under the old one: sealed secrets, the PKI bundle and
  webhook secrets. To rotate the sealed-secret key, configure the store yourself with both keys,
  `secrets: [localSealedSecretStore({ sql, keys: { current: 'k2', keys: { k1: old, k2: next } } })]`, and run its
  `reseal()` until it reports no rows left. There is no CLI command for this yet, and re-sealing the PKI bundle
  under a new secret means exporting it again with `exportPkiBundle`.



# Database (/docs/operations/database)

> PostgreSQL requirements, connecting and pooling through PgBouncer, how migrations run, PGlite for development, the runtime tables and the lock order.



PostgreSQL is the only source of truth. Leases, the work queue, the outbox and every aggregate are tables, so
processes never coordinate through anything else, and a crashed process leaves nothing behind that the database
does not already know. Better Compute's store is PostgreSQL or PGlite only: it needs `EXCLUDE` constraints,
`SKIP LOCKED`, range types and `xid8`, so there is no SQLite adapter.

## Requirements [#requirements]

| Requirement              | Why                                                                                                                                                                                                                                       |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL 16 or newer   | `migrate()` reads `server_version_num` and refuses anything below 160000 with `INVALID_CONFIG`. CI runs PostgreSQL 17.                                                                                                                    |
| `btree_gist`             | The first migration runs `CREATE EXTENSION IF NOT EXISTS btree_gist`. `device_bookings` needs it for text equality inside `EXCLUDE USING gist (device_id WITH =, during WITH &&)`, the constraint that makes a double booking impossible. |
| `CREATE` on the database | Migrations create tables, indexes and the `compute_now()` function.                                                                                                                                                                       |

`btree_gist` is a trusted extension, so a database owner with `CREATE` can install it. On a managed service that
does not allow that, have a superuser run `CREATE EXTENSION btree_gist` once before the first migration. The
official `postgres` image ships it; `examples/postgres/docker-compose.yml` runs `postgres:17`.

The booking constraint is the exclusion constraint behind every capacity
guarantee: two overlapping bookings of one device cannot both commit, whatever the application does. See
[capacity](/docs/guides/capacity).

## Connecting [#connecting]

```ts
import { postgres } from 'better-compute/postgres';

const database = postgres({ url: process.env.BETTER_COMPUTE_DATABASE_URL!, max: 20, statementTimeoutMs: 15_000 });
```

| Option               | Default | What it does                                                                                                                                                    |
| -------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                | —       | A `postgres://` or `postgresql://` connection string. Give this or `pool`.                                                                                      |
| `pool`               | —       | A `pg.Pool` you own: TLS, IAM authentication hooks, a PgBouncer endpoint. `close()` leaves it open.                                                             |
| `max`                | 10      | Pool size when `url` is given, from 1 to 100. A connection attempt gives up after 10 seconds.                                                                   |
| `statementTimeoutMs` | none    | `SET LOCAL statement_timeout` at the start of every transaction; standalone statements then run in a short transaction of their own. Migrations run without it. |

An idle connection that errors (a restarted server) is dropped and the next checkout reconnects; it never crashes
the process.

### Pooling and PgBouncer [#pooling-and-pgbouncer]

Each process holds its own pool, so the total is roughly `max` times the number of processes. PLAN.md lists
connections as one of the first limits at scale and puts PgBouncer in front of PostgreSQL. `postgres()` is safe
behind PgBouncer in **transaction** mode:

* it keeps no session state: the statement timeout is `SET LOCAL`, and no startup parameters are set;
* it never uses `LISTEN`/`NOTIFY`;
* it uses only unnamed prepared statements;
* it takes no session-level advisory locks: concurrent migrators serialize on the transaction-level
  `pg_advisory_xact_lock(1650683248, 1)`, released at `COMMIT`.

Leases are table rows with a TTL and an epoch rather than advisory locks for the same reason: session-level
`pg_advisory_lock` breaks under transaction pooling.

### Transactions [#transactions]

`transaction(fn, { isolation?, readOnly? })` runs `fn` on one connection between `BEGIN` and `COMMIT` at
`read committed` by default; guarded updates and row locks carry the invariants. Inside `fn`'s async call chain the
executor joins the caller's transaction, and a nested `transaction()` becomes a savepoint. A statement error that
`fn` catches still aborts the transaction in PostgreSQL; the executor detects the `COMMIT` that became a `ROLLBACK`
and rejects with `INTERNAL_ERROR`. `sqlState(error)` reads the SQLSTATE, such as `23P01` for an exclusion
violation, from either driver.

## Migrations [#migrations]

`better-compute migrate` runs `compute.initialize()`. It first checks every method's IAM action against the merged
catalog and the normative method table, and fails with `INVALID_CONFIG` before touching the store if one is
missing or wrong. Then it applies migrations, anchors the clock, and starts each module. It is idempotent, and
`serve` and `dev` run it on start unless `--no-migrate`.

```bash
better-compute migrate     # once per deploy, before new processes start
better-compute serve --no-migrate
```

### The ledger [#the-ledger]

Two tables record what ran: `compute_migrations(name, applied_at)` and `compute_schema_version`, whose `version` is
the number of base migrations applied. Every migration is named `<owner>/<nnnn>_<slug>` and:

* runs in its own transaction and is recorded in the ledger in that same transaction, so a failed one leaves
  nothing behind;
* is skipped when the ledger already has it;
* is never edited, renamed or removed once released. Numbers only grow within an owner.

`migrate` takes the advisory lock in every transaction, so two processes migrating at once serialize and the
second skips what the first applied. It refuses to run, with `INVALID_CONFIG` naming the entries, when the ledger
holds something this build cannot account for:

* a later number from an owner it knows: a newer release migrated this database;
* an unknown name at or below an owner's last number: a migration was renamed or removed;
* a base schema version above its own.

Migrations of owners this build has no migrations for are ignored, so dropping a module from the configuration does
not block startup. Its tables stay.

### Order [#order]

Migrations run in registration order:

1. the base schema, owner `core`, which is reserved for it;
2. the built-in modules: node controllers, site boot, `inventory`, `capacity` (`capacity/0001_device_links` to
   `capacity/0005_extend_holds`, the one-extend-at-a-time hold of f36248d), `tenancy` (`tenancy/0001_tenant_states`),
   `confirmations` (`confirmations/0001_bindings`, 06d7f04), `quorum` (`quorum/0001_batches` to `quorum/0008_batch_agent`, which records the agent that proposed a batch,
   f3c5c4e), `pki/0001_sealed_secrets` when the default secret store is in use, and the tunnels module when
   `tunnels` is set;
3. your `modules`, in the order you list them;
4. your `plugins`.

### The base schema [#the-base-schema]

| Migration                            | Adds                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `core/0001_clock`                    | `btree_gist`, `compute_sim_clock`, `compute_now()`                                                                                                                                                                                                                                                                                                                                                                                   |
| `core/0002_base`                     | The frozen v0.1 schema: sites, pools, nodes, devices, `device_bookings`, `node_resources`, `quota_usage`, reservations, commands, `work_queue`, `leases`, `workers`, `outbox`, `authz_projection`, `authz_changes`, `iam_inbox`, `join_tokens`, `gateways`, allocations, usage windows and dead letters, `node_certs`, `inventory_snapshots`, `audit_events`, `pool_capacity_rollup`                                                 |
| `core/0003_v01`                      | v0.1 columns, `node_inventory`, `node_inventory_sections`                                                                                                                                                                                                                                                                                                                                                                            |
| `core/0004_command_cancel`           | `cancel_requested_at`: cancelling a command after dispatch                                                                                                                                                                                                                                                                                                                                                                           |
| `core/0005_audit_commit_order`       | `audit_events.txid` and `seq`, so audit readers page in commit order                                                                                                                                                                                                                                                                                                                                                                 |
| `core/0006_pool_archive`             | Archived pools                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `core/0007_device_health`            | `device_health` quarantines                                                                                                                                                                                                                                                                                                                                                                                                          |
| `core/0008_session_versions`         | The daemon version and protocol a node announced in Hello                                                                                                                                                                                                                                                                                                                                                                            |
| `core/0009_runtime`                  | `outbox_cursors`, `work_queue.attempts`, `commands.owner`, tenant indexes                                                                                                                                                                                                                                                                                                                                                            |
| `core/0010_v02_server`               | `join_requests`, `node_certs.allowed_kinds`, `nodes.deleted_at`, `secret_refs`                                                                                                                                                                                                                                                                                                                                                       |
| `core/0011_v02_security`             | `sites.deleted_at`, `attestation_uses`                                                                                                                                                                                                                                                                                                                                                                                               |
| `core/0012_v02_evidence`             | `node_bindings`, node signatures of attested results                                                                                                                                                                                                                                                                                                                                                                                 |
| `core/0013_image_cache`              | `node_image_cache`                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `core/0014_ssh_certificates`         | `ssh_certificates`, for break-glass revocation and audit                                                                                                                                                                                                                                                                                                                                                                             |
| `core/0015_inventory_images`         | The `images` inventory section                                                                                                                                                                                                                                                                                                                                                                                                       |
| `core/0016_duplicate_identity`       | `node_sessions.duplicate_hellos` and `duplicate_since`, for the duplicate-identity quarantine                                                                                                                                                                                                                                                                                                                                        |
| `core/0017_allocation_evidence`      | `allocations.bound_report`, `bound_sig_kid`, `bound_sig`: node-attested `AllocationBound`                                                                                                                                                                                                                                                                                                                                            |
| `core/0018_partitioned_history`      | `audit_events`, `inventory_snapshots` and `usage_windows` partitioned by month, and `compute_history_partition()`                                                                                                                                                                                                                                                                                                                    |
| `core/0019_outbox_dead_letters`      | `outbox_dead_letters`: events a sink always refuses, set aside                                                                                                                                                                                                                                                                                                                                                                       |
| `core/0020_command_created_at`       | `commands.created_at`, for command latency, and the `commands_settled` index                                                                                                                                                                                                                                                                                                                                                         |
| `core/0021_signed_specs`             | `nodes.spec_body` and `spec_signature`: the signed bytes of each NodeSpec generation                                                                                                                                                                                                                                                                                                                                                 |
| `core/0022_ssh_certificate_sessions` | `ssh_certificates.session_id` and `reauthorized_at`, so each break-glass certificate is re-decided for the session that issued it                                                                                                                                                                                                                                                                                                    |
| `core/0023_session_trust_version`    | `node_sessions.trust_version` (`bigint`, default 0): the trust bundle version each daemon reports in `Hello` and `StatusReport`, which `pki rotate-command-root --stage switch` checks over connected sessions                                                                                                                                                                                                                       |
| `core/0024_session_capabilities`     | `node_sessions.capabilities` (`text[]`, default empty): the `Hello.capabilities` of the current session; `tunnels.open` needs `tunnel.instance-dial` there for instance `tcp` and `ssh` tunnels                                                                                                                                                                                                                                      |
| `core/0025_secret_scope`             | `secret_refs.scope` (`text`, `tenant` or `fleet`, default `tenant`): the scope `secrets.put` decided on, so fleet secrets stored in the root tenant can be told from the tenant's own. Rows from before it read `tenant`; a tenant-scoped put or delete of a `fleet` secret is `INVALID_STATE`, and a fleet put may claim a `tenant` row                                                                                             |
| `core/0026_session_approver_set`     | `node_sessions.approver_set_epoch` (`numeric(20, 0)`, at least 0) and `approver_set_sha256` (`bytea`, no length check, so a session write never fails over a daemon's report): the approver set the daemon reports enforcing in `Hello` and `StatusReport`, beside `trust_version`. Both nullable and NULL until a daemon reports them (the gateway writes NULL for an empty digest); doctor's `pki.quorum-lag` reads them (9e1b242) |
| `core/0027_join_token_hosted`        | `join_tokens.hosted` (`text`, `CHECK (hosted IN ('kubernetes'))`, NULL for every other token): a token that a [bcd DaemonSet](/docs/daemon/kubernetes) joins a cluster's nodes with (1f0f683). `joinTokens.create({ hosted })` writes it, and JoinService lands such a node `tenant-controlled` unless the pool's `policy.kubernetes.platformOperated` is true on a pool the platform owns (a27156a, 259d82c)                        |
| `core/0028_volume_allocations`       | `allocations.owner_kind` also allows `volume`: an allocation that a [sandbox volume](/docs/guides/vps/sandboxes#volumes) owns, booking its `diskGiB` on one host, scalar resources only, never devices, until the volume is gone (2a09202)                                                                                                                                                                                           |

### Module migrations [#module-migrations]

| Owner           | Migrations                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `capacity`      | `0001_device_links`, `0002_lease_health`, `0003_device_counts` (`capacity_device_counts`, kept by a trigger on `devices`), `0004_mig_layouts` (`capacity_mig_layouts`, the MIG layout gate)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `tenancy`       | `0001_tenant_states` (`compute_tenant_states`: suspended and deleted IAM tenants, and each teardown's `teardown_done_at`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `confirmations` | `0001_bindings` (`confirmation_bindings`: per delegated confirmation, the tenant, delegation, action, resource, the SHA-256 of the text the person was shown and its expiry, so an approval admits only that request; always hosted; 06d7f04, see [delegated confirmation](/docs/guides/iam/delegated-confirmation#one-approval-one-request))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `pki`           | `0001_sealed_secrets`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `quorum`        | `0001_batches` (`command_batches`, `command_batch_scope`, `command_batch_members`, `command_batch_approvals`: the v1.0 destructive-batch quorum; the module is always hosted), `0002_payload_digest` (a scope leaf's `payload_sha256`, empty for any payload), `0003_trust_bundle_state` (the published trust bundle's version and digest, so the version follows the content), `0004_trust_bundle_approver_set` (the published quorum, to check approver-set signatures; since c6214dc it also records the set's epoch), `0005_batch_request` (`command_batches.request_sha256`, so a `commandBatches.create` retry returns the same batch), `0006_trust_bundle_history` (every published trust bundle digest with its version, so a stale replica never republishes an old bundle as newer), `0007_leaf_payload` (`command_batch_scope.payload_type_url` and `payload`, a leaf's payload for approvers to decode; 2ca0ba4), `0008_batch_agent` (`command_batches.created_by_agent` and `delegation_id`: the agent that proposed a batch and its delegation, so its reason reads as the agent's; f3c5c4e); see [PKI](/docs/operations/pki#the-destructive-batch-quorum-v10)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `iam`           | `0001_successors`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `provision`     | `0001_workflows`, `0002_images`, `0003_workflow_owner`, `0004_gated_mig_layout`, `0005_image_pins` (`provision_image_pins`: images a deploy keeps in its site's cache until its workflow ends)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `metal`         | `0001_nodes`, `0002_addresses` (`metal_nodes.addresses`: the machine's own IPs, from inspection or `metal.enroll`'s `addresses`), `0003_tenant_root`, `0004_bmc_credentials`, `0005_batch` (`metal_nodes.batch_id`, the bulk verb's batch), `0006_bmc_driver_options` (`metal_nodes.bmc_driver_options`, a `jsonb` object or NULL: `bmc.driverOptions`, sent on every BMC operation; 3f19859), `0007_discovered` (`metal_discovered`, each site agent's PXE discovery list keyed by agent and MAC, and `metal_discovery_reports`, the digest of each agent's last stored list; 3d26428), `0008_mac_source` (`metal_nodes.mac_source`: `operator`, `inspection` or `discovered`; `discovered_mac` and `discovered_mac_at`, a candidate awaiting `metal.confirmMac`; 8de6e2f), `0009_power_cut` (`metal_nodes.power_cut_at`: when metal last sent the BMC a power action that ends the running system, so `metal.ensureRamdisk` trusts only a ramdisk session that connected since)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `vps`           | `0001_instances`, `0002_sealed_user_data`, `0003_provider_bootstraps`, `0004_sandboxes` (`sandbox_templates`, `sandboxes`, `instances.sandbox_id` and `warm_template_id`, and `paused` allowed in `instances.desired_state`), `0005_sandbox_snapshots` (`sandbox_snapshots`, `instances.restore_from`, `sandboxes.from_snapshot`), `0006_warm_claims` (`instances.claimed_at`), `0007_template_env` (`sandbox_templates.env`), `0008_template_network` (`sandbox_templates.network` and `instance_version`, `instances.network_policy` and `warm_template_version`), `0009_snapshot_fixes` (`sandbox_snapshots.principal_instance_id` and `purged`), `0010_sandbox_metric_indexes`, `0011_host_certificates` (`host_certified` on instances and snapshots), `0012_host_cert_validity` (`host_cert_valid_before` on both), `0013_sandbox_expiring` (`sandboxes.expiring_at`), `0014_sandbox_extending` (`sandboxes.extending_request` and `extending_at`), `0015_snapshot_agent` (`sandbox_snapshots.agent_id`, backfilled from each snapshot's sandbox: a delegated restore takes only its own agent's snapshots; 7b01053), `0016_warm_secret_versions` (`instances.warm_secret_versions`: the `secret_refs.version` of each secret a warm guest was given, so a rotation retires it; 2c7072b), `0017_identity_rebind` (`instances.identity_rebind_ref`, `rebind_cert_valid_before` and `identity_rebound_ref`: a memory restore's own sealed guest identity, the validity of the host certificate in it, and the bundle the host confirmed it applied; 5346523), `0018_placement` (`flavors.placement` and `instances.host_placement`, both `jsonb`: a flavor's `numaAffine`, `pinned` and `hugepages`, and where on its host an instance landed, which the next placement on that host reads to keep pinned CPUs exclusive; 9d55b36), `0019_ip_pool_dns` (`ip_pools.dns`, `inet[]`, default empty: the DNS servers a pool's IPAM hands its guests with the gateway; 387c22f), `0020_volumes` (`sandbox_volumes`: a tenant's persistent volume, its size (1 to 16,384 GiB), serial, host and `diskGiB` allocation, state `ready`, `deleting` or `deleted`, and the host generation it left at, with a unique live name per tenant; `volume_attachments`: which instance attaches which volume, in order, read-only or not, at which mount path; `sandbox_templates.volumes`, `jsonb`; 2a09202), `0021_volume_resize_hold` (`sandbox_volumes.resizing_request` and `resizing_at`: the one resize whose new size is pending in the IAM projection while it is decided, taken over after 60 s; 6cb30e9), `0022_volume_storage` (`host_volume_storage`: each runtime host's volume store as its bcd last reported it in `StatusReport.volume_storage`, total, available, reserved, committed and outstanding bytes and `reported_at`, which volume bookings are checked against; 131f5ee) |
| `events`        | `0001_webhooks`, `0002_delivery_retention`, `0003_webhooks_by_tenant` (an index for a deleted tenant's purge)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `fleet`         | `0001_names`, `0002_applied`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `rollout`       | `0001_rollouts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `audit`         | `0001_checkpoints`, `0002_archives`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

The [modules reference](/docs/reference/modules) is generated from a deployment with every module and lists the
same migrations.

A module you write owns its own prefix; see [modules](/docs/extending/modules).

### Upgrading [#upgrading]

Additive migrations ("expand") run before new code starts; PLAN.md puts destructive ones ("contract") in the
following release, once no process of the old release is left. Released migrations never change, so a database
migrated by release N+1 refuses release N; roll back code only together with the database it expects.

## One clock [#one-clock]

All SQL reads time through `compute_now()`, never `now()`, and column defaults use it
too. `scripts/check-sql-now.mjs` fails CI on a bare `now()` in `packages/*/src`, except in `core/0001_clock`, which
defines the function. In production `compute_now()` is the server's clock, and each process anchors its business
time to it, re-sampling at every lease renewal and raising `ClockSkew` beyond 1 s.

For tests and the simulator, `useSimClock(sql, startMs)` points `compute_now()` at `compute_sim_clock`,
`setSimClock(sql, ms)` moves it forward (never back), and `restoreSystemClock(sql)` points it back. These are
database-wide. `computeNowMode(sql)` reports `system`, `sim`, `custom` or `missing`, and `doctor` reports a
`compute_now()` that differs from the database clock.

## The runtime tables [#the-runtime-tables]

These tables are how processes cooperate. You never write them by hand, but you will read them when you debug.

| Table            | Role                                                                                                                                  |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `leases`         | `name`, `owner`, `epoch`, `expires_at`. Names: `shard:<n>`, `pool:<id>`, `site:<id>:boot`, `singleton:<job>`, `gateway:<id>`.         |
| `workers`        | One row per worker process with its heartbeat; rendezvous hashing over the live rows assigns the 256 shards.                          |
| `work_queue`     | The coalescing work queue: one row per `(kind, key)` with `shard`, `gen`, `not_before`, `claimed_until`. |
| `commands`       | Every command with its per-node `seq`, state, deadline and signature.                                                                 |
| `outbox`         | Events written in the transaction that caused them: `seq`, `txid` (`pg_current_xact_id()`), `topic`, `payload`.                       |
| `outbox_cursors` | Each sink's position, saved after every delivered batch.                                                                              |
| `authz_changes`  | The authorization projection's change feed, in the outbox's `(txid, seq)` shape.                                                      |
| `gateways`       | The gateway directory: region, `internal_url`, version, `max_streams`, sessions, `draining`, heartbeat.                               |

### Leases and fencing [#leases-and-fencing]

A lease is acquired or renewed with one guarded update, and the epoch goes up whenever the owner changes:

```sql
UPDATE leases SET owner = $me, epoch = CASE WHEN owner = $me THEN epoch ELSE epoch + 1 END,
       expires_at = compute_now() + make_interval(secs => $ttl)
 WHERE name = $name AND (owner = $me OR expires_at < compute_now()) RETURNING epoch;
```

Shard, pool and site leases live 15 s, gateway leases 20 s, and both renew every 5 s. Each fenced transaction
starts with `SELECT 1 FROM leases WHERE name = $name AND owner = $me AND epoch = $e AND expires_at > compute_now()
FOR SHARE`, so a process that lost its lease either sees that and aborts, or holds up the takeover until its
transaction ends. A bare `EXISTS` would not fence under `read committed`.

### The work queue [#the-work-queue]

Enqueuing happens in the state-change transaction and coalesces: a key already queued gets `gen = gen + 1` and the
earlier `not_before`. A worker claims due keys of its shards with `FOR UPDATE SKIP LOCKED` and sets
`claimed_until`; success deletes the row only `WHERE gen = $claimedGen`, so a key dirtied again while it was being
processed survives and no wake-up is lost. Reconcilers also re-enqueue everything they own every `resyncMs`.

### The outbox [#the-outbox]

Commit order is not `seq` order, so readers never trust `seq` alone:

```sql
WHERE txid < pg_snapshot_xmin(pg_current_snapshot())
  AND (txid, seq) > ($cursorTxid, $cursorSeq)
ORDER BY txid, seq
```

Nothing can commit behind the cursor. The same cursor serves the SSE `Last-Event-ID`, `authz_changes` and the
audit checkpoints. Delivery is at least once, with one lease per sink. Order by the table's own columns
(`o.txid, o.seq`), never by a text alias of `txid`, which would sort `"10"` before `"2"`. The `retention` module
prunes rows every cursor has passed; see [retention](/docs/operations/retention).

## Lock order [#lock-order]

Any transaction that locks several of these rows takes them in this order:

```text
quota_usage → pools → nodes → devices → node_resources → pool_capacity_rollup
```

Reserve charges quota first and release refunds it first, so concurrent reservations and releases never wait on
each other in a cycle. The order was set by the capacity torture test, which found a quota and rollup deadlock
before it was fixed. Module code that touches capacity rows (node verbs, `nodes.delete`, join approvals, which lock
the pool first) follows it too.

## PGlite [#pglite]

`pglite({ dataDir?, extensions? })` from `better-compute/pglite` runs PostgreSQL in WebAssembly. `better-compute
dev`, the test suites and the examples use it.

* `@electric-sql/pglite` is an optional peer dependency, loaded on the first statement. `btree_gist` is always
  included.
* Without `dataDir` the database lives in memory. `BETTER_COMPUTE_DATABASE_URL=pglite:DIR` or `pglite:memory`
  selects it from the environment.
* There is one connection. Transactions and standalone statements run one at a time behind an async mutex, so
  code in a transaction that waits for another top-level transaction deadlocks. Concurrency tests belong on real
  PostgreSQL.
* It runs `ANALYZE` after migrations and periodically, so query plans in development match PostgreSQL's.

PGlite is for development and tests. The production-shaped `examples/postgres` refuses `pglite:` URLs. See
[local development](/docs/operations/local-development).

## Backups [#backups]

Back up the database and the PKI bundle together. The bundle holds the CAs every node certificate chains to and the
keys daemons pin; `doctor` tells you to restore it from backup if it cannot be read, since creating a new one makes
every node rejoin. See [PKI](/docs/operations/pki).


# Deploying on AWS (/docs/operations/deploy-aws)

> The reference deployment of the control plane on AWS, as three CloudFormation stacks rendered by aws deploy-templates, from the VPC to the ECS Fargate services.



`@better-compute/aws` renders the reference deployment of the control plane on AWS as three CloudFormation JSON
templates. They lay out the [split-roles topology](/docs/operations/deployment#split-roles) on AWS services: the
api behind an L7 load balancer, the gateway behind an L4 load balancer with TLS passthrough, workers with no
listener, and Aurora PostgreSQL with IAM authentication. Nothing here is required: the templates are a starting
point you can read, diff and change, not an installer.

```bash
better-compute aws deploy-templates --out cfn            # network.json, data.json, control-plane.json
better-compute aws deploy-templates --out cfn --zones 2 --nat single --readers 0 --name bc-stage
```

The command makes no AWS call. It writes only new files, and refuses before writing anything if one exists.

## The three stacks [#the-three-stacks]

| Stack           | What it creates                                                                                                                                                                                                                                                              | Imports           |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `network`       | A VPC across 2 or 3 zones with a public and a private /20 per zone, NAT per zone (or one), the S3 gateway endpoint, interface endpoints for Secrets Manager, KMS, ECR, CloudWatch Logs and STS, VPC flow logs                                                                | none              |
| `data`          | Aurora PostgreSQL 16 or later on Serverless v2: IAM database authentication, TLS forced, `btree_gist` allowed, storage under its own KMS key, the master password managed by Secrets Manager, a writer and 0 to 2 readers                                                    | `network`         |
| `control-plane` | ECS Fargate services `api`, `worker` and `gateway` plus a `migrate` task; an ALB with an ACM certificate and WAF; an NLB in TCP passthrough; task roles; the deployment secret and the sealed PKI bundle; one KMS key; S3 buckets; an EventBridge bus; log groups and alarms | `network`, `data` |

Each stack exports what the next one needs as `<stack name>-<output>` (for example `bc-prod-network-PrivateSubnetIds`),
and the later stacks take the earlier stack names as the `NetworkStack` and `DataStack` parameters. Structural
choices (zones, NAT mode, readers, the deployment name) are command flags; everything that differs between
environments (the image, the domains, the certificate, task counts, sizes) is a stack parameter.

### Processes [#processes]

| Service          | Command                                                      | In front                               | Security group admits                                              |
| ---------------- | ------------------------------------------------------------ | -------------------------------------- | ------------------------------------------------------------------ |
| `api`            | `serve --roles api --port 3000 --no-migrate`                 | ALB, HTTPS 443, health check `/readyz` | 3000 from the ALB only                                             |
| `worker`         | `serve --roles controllers,scheduler,sinks,iam --no-migrate` | none                                   | nothing                                                            |
| `gateway`        | `serve --roles gateway --no-migrate`                         | NLB, TCP 443 to 8443                   | 8443 from the NLB; the internal API port from api and worker tasks |
| `migrate` (task) | `migrate`                                                    | none                                   | nothing                                                            |

Every task runs in the private subnets without a public address, with a read-only root file system, all Linux
capabilities dropped, `awslogs` in non-blocking mode, and the deployment circuit breaker with rollback. All of them
also wear the data stack's client security group, the only group the database admits on 5432.

The ALB uses the `ELBSecurityPolicy-TLS13-1-2-2021-06` policy, redirects HTTP to HTTPS, drops invalid headers, and
has a WAF web ACL in front: a per-IP rate limit (`WafRateLimit` requests per five minutes), the AWS IP reputation
list, the common rule set, and known bad inputs. The common rule set's 8 KiB body rule only counts, since API bodies
can be larger. The idle timeout is an hour, for interactive tunnels.

The NLB never terminates TLS: the gateway authenticates each daemon by its client certificate, which a terminating
balancer would strip. Its target group turns client IP preservation off, so tasks only need to admit the NLB's own
security group.

### Secrets and keys [#secrets-and-keys]

The templates take no secret as a parameter (`lintTemplate` refuses `NoEcho` parameters):

* `BETTER_COMPUTE_SECRET` is generated by Secrets Manager (64 characters) and reaches the tasks as an ECS secret.
* The **sealed PKI bundle** (the JSON `better-compute pki init` writes) goes into the `PkiBundleSecret` secret,
  which the tasks get as `BC_PKI_BUNDLE`. The stack creates it with a random placeholder, and you put the real
  bundle into it.
* The database master password is generated and kept by RDS in Secrets Manager. The processes never use it: they
  connect as an IAM-authenticated user with `rdsAuthTokenProvider`.

One rotating KMS key encrypts the two secrets, the blobs, snapshots and audit buckets, every log group and the alarm
topic. The database has a key of its own. Both keys are kept when a stack is deleted.

### Restoring under a new stack [#restoring-under-a-new-stack]

The secrets, the platform key, the log groups and the blobs, snapshots and audit buckets outlive a deleted stack, and
a restored database depends on all of them: data sealed with the old `BETTER_COMPUTE_SECRET` (the PKI bundle too) is
unreadable under any other, snapshot records point at `s3://<old snapshots bucket>/…`, and stored objects are
encrypted with the old key. So a new control-plane stack adopts them instead of creating new ones:
`ExistingDeploymentSecretArn`, `ExistingPkiBundleSecretArn`, `ExistingPlatformKeyArn`, `ExistingBlobsBucket`,
`ExistingSnapshotsBucket`, `ExistingAuditBucket` and `AdoptLogGroups=true`. Every role may then decrypt with the old
key, api and worker may also write with it, and the adopted buckets get the new stack's TLS-only policy. A restore
keeps the deployment name: tenant secrets live under `better-compute/<name>/`, the only prefix the task roles may
read. The repository's runbook has the command.

### Task roles [#task-roles]

Every role trusts only `ecs-tasks.amazonaws.com` calling from this account. A base policy, shared by all three,
allows `rds-db:connect` as the one database user, `secretsmanager:GetSecretValue` under `better-compute/<name>/*`,
`kms:Decrypt` with the platform key, and reading snapshots (NodeSpec signs snapshot download URLs wherever it renders). The secret prefix carries the
deployment name, so a staging deployment in the same account and region can neither read nor overwrite production's
tenant secrets; the config module passes the same prefix to `awsSecretsManagerStore`. On top of that:

| Role    | Adds                                                                                                                                                                                    |
| ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| api     | Encrypting with the platform key; writing secrets under `better-compute/<name>/*`; reading the downloads bucket; reading and writing blobs and snapshots; `events:PutEvents` on the bus |
| worker  | Encrypting with the platform key; writing secrets; blobs and snapshots; `s3:PutObject` and `s3:PutObjectRetention` on the audit bucket; `events:PutEvents`                              |
| gateway | Nothing                                                                                                                                                                                 |

The gateway reads tenant secrets because `serve --roles gateway` runs the secret broker in process: a daemon's
`SecretRequest` is answered there, and the broker releases a ref only when it is bound to work the asking node is
doing right now, sealed to that node. It never writes secrets, objects or events, and never encrypts.

Every role that reads a bucket also has `s3:ListBucket` on it. Without it S3 answers a read of a missing key with
403 AccessDenied instead of 404, so a download that was never published would be an error rather than not found.
On the blobs and snapshots buckets the listing is confined to the stores' prefixes (`s3:prefix` like `blobs/*` and
`snapshots/*`, passed to the tasks as `BC_BLOBS_PREFIX` and `BC_SNAPSHOTS_PREFIX`), so no task enumerates keys
outside them. Only the downloads bucket, which holds public bcd binaries, is listed whole.

No statement uses a wildcard resource or a service-wide action. The migrate task has a role of its own that allows only `rds-db:connect`: an upgrade runs the next image's migrate step before that image serves anything, so it gets no secret, bucket, bus or KMS permission. A configuration whose loading calls AWS under `migrate` needs those permissions added.

### Buckets [#buckets]

| Bucket    | Encryption | Notes                                                                                                   |
| --------- | ---------- | ------------------------------------------------------------------------------------------------------- |
| downloads | SSE-S3     | bcd binaries. Read only by its CloudFront distribution (origin access control), never by the bucket URL |
| blobs     | SSE-KMS    | Support bundles and other blobs. Never public, never behind the CDN                                     |
| snapshots | SSE-KMS    | `s3SnapshotStore`                                                                                       |
| audit     | SSE-KMS    | Object Lock in compliance mode with `AuditRetentionDays` default retention, for `s3AuditSink`           |

All of them block public access, enforce bucket-owner ownership, are versioned, and refuse any request that is
not made over TLS. Blobs, snapshots and audit are kept when the stack is deleted, and a restore adopts them.

## Deploying [#deploying]

The [deployment guide in the repository](https://github.com/better-compute/better-compute/tree/main/deploy/aws)
has the commands. In short:

1. **Image.** Build an image whose entrypoint is the `better-compute` CLI and which contains your configuration
   module (below), push it to ECR, and note its digest. The template accepts only `<repository>@sha256:<digest>`.
2. **Certificate.** Request an ACM certificate for the API domain in the stack's region.
3. **Network and data.** Deploy `network`, then `data` with `NetworkStack` set and `DataApi=Enabled`.
4. **Database user.** With the Data API and the master secret, create the IAM user once:
   `CREATE ROLE bc_app LOGIN`, `GRANT rds_iam TO bc_app`, `GRANT CREATE ON DATABASE better_compute TO bc_app`,
   `GRANT CREATE, USAGE ON SCHEMA public TO bc_app`. Then update the stack with `DataApi=Disabled`.
5. **Control plane at zero.** Deploy `control-plane` with `ApiCount=0 WorkerCount=0 GatewayCount=0`.
6. **PKI bundle.** Put the sealed bundle into the `PkiBundleSecretArn` secret.
7. **Migrate.** Run the migrate task definition once (`aws ecs run-task`, with the `TaskSubnetIds` and
   `MigrateSecurityGroupIds` outputs).
8. **Scale up and point DNS.** Update the stack with the real counts. With `HostedZoneId` set, the stack writes
   alias records for both domains; otherwise point them at `ApiLoadBalancerDnsName` and `GatewayLoadBalancerDnsName`.

Each later release is a new image digest, and its [expand migrations](/docs/operations/upgrades) run before any
process of it: set `MigrateImage` to the new digest (only the migrate task definition changes), run the migrate
task, then set `Image` to it and clear `MigrateImage`. The services roll with the circuit breaker, which rolls back
a release whose tasks never become healthy.

## The configuration in the image [#the-configuration-in-the-image]

The tasks pass everything a configuration needs as environment variables: `AWS_REGION`, `BC_PUBLIC_URL`,
`BC_GATEWAY_PUBLIC_HOST`, `BC_GATEWAY_INTERNAL_PORT`, `BC_DATABASE_HOST`, `BC_DATABASE_PORT`, `BC_DATABASE_NAME`,
`BC_DATABASE_USER`, `BC_EVENT_BUS`, the four bucket names (`BC_DOWNLOADS_BUCKET`, `BC_BLOBS_BUCKET`,
`BC_SNAPSHOTS_BUCKET`, `BC_AUDIT_BUCKET`), the blob and snapshot store prefixes (`BC_BLOBS_PREFIX`,
`BC_SNAPSHOTS_PREFIX`), `BC_AUDIT_RETENTION_DAYS`, `BC_KMS_KEY_ARN`, `BC_DEPLOYMENT` and
`BC_ROLE`, plus the secrets `BETTER_COMPUTE_SECRET` and `BC_PKI_BUNDLE`. A configuration on AWS wires them into
the platform-service factories:

```ts
import { readFileSync } from 'node:fs';
import pg from 'pg';
import { defineConfig } from '@better-compute/server';
import { postgres } from '@better-compute/postgres';
import { vps } from '@better-compute/vps';
import { ambientCredentials } from '@better-compute/aws-core';
import {
  awsSecretsManagerStore, eventBridgeSink, rdsAuthTokenProvider,
  s3AuditSink, s3BlobStore, s3SnapshotStore,
} from '@better-compute/aws';

export default defineConfig(async ({ command, env = process.env } = {}) => {
  const aws = { region: env.AWS_REGION!, credentials: ambientCredentials({ region: env.AWS_REGION }) };
  const sse = { kind: 'aws:kms', keyId: env.BC_KMS_KEY_ARN } as const;
  // The task roles may list these buckets only under these prefixes.
  const snapshots = s3SnapshotStore({ ...aws, bucket: env.BC_SNAPSHOTS_BUCKET!, prefix: env.BC_SNAPSHOTS_PREFIX!, sse });
  // A restore rendered before the download credentials load cannot be fetched: serve only once they have.
  if (command === 'serve' && !(await snapshots.warm())) throw new Error('snapshot download credentials did not load');
  const host = env.BC_DATABASE_HOST!;
  const port = Number(env.BC_DATABASE_PORT);
  const user = env.BC_DATABASE_USER!;
  return {
    database: postgres({
      pool: new pg.Pool({
        host, port, user, database: env.BC_DATABASE_NAME,
        // The RDS certificate bundle, shipped in the image.
        ssl: { ca: readFileSync('/etc/better-compute/rds-global-bundle.pem', 'utf8'), rejectUnauthorized: true },
        password: rdsAuthTokenProvider({ ...aws, hostname: host, port, username: user }),
      }),
    }),
    secret: env.BETTER_COMPUTE_SECRET!,
    baseURL: env.BC_PUBLIC_URL!,
    // The prefix the task roles may use: better-compute/<deployment name>.
    secrets: [awsSecretsManagerStore({ ...aws, prefix: `better-compute/${env.BC_DEPLOYMENT}` })],
    sinks: {
      events: [eventBridgeSink({ ...aws, eventBusName: env.BC_EVENT_BUS })],
      audit: [s3AuditSink({ ...aws, bucket: env.BC_AUDIT_BUCKET!, prefix: 'audit', sse })],
    },
    downloads: { blob: s3BlobStore({ ...aws, bucket: env.BC_DOWNLOADS_BUCKET! }) },
    supportBundles: { blob: s3BlobStore({ ...aws, bucket: env.BC_BLOBS_BUCKET!, prefix: env.BC_BLOBS_PREFIX!, sse }) },
    modules: [vps({ snapshots })],
    // …your tenancy, IAM, PKI (from env.BC_PKI_BUNDLE) and modules, as in examples/postgres.
  };
});
```

`ambientCredentials` (from `@better-compute/aws-core`) reads the task role through the ECS container credentials
endpoint. It ships with the aws-core credential sources of the AWS foundation phase (A0); `@better-compute/aws`'s
own `envCredentialSource` reads only `AWS_ACCESS_KEY_ID` and its siblings, so it cannot use a task role. Name the
file `better-compute.config.ts` (the example is TypeScript), or drop the `!` and `as const` for a `.mjs` file.

The snapshot store signs restore URLs inside NodeSpec's transaction, from credentials it holds in memory. Until they
load, a restore rendered for a host keeps its `s3://` identifier, which bcd cannot fetch, and the instance fails. So
the example builds the store first and, for `serve`, refuses to start when `warm()` resolves `false`; ECS then
replaces the task. `migrate` renders no NodeSpec and skips the wait.

The `host` export's `gateway` should advertise each task's own address as its internal URL (from the
ECS task metadata endpoint), listen on `BC_GATEWAY_INTERNAL_PORT`, and name `BC_GATEWAY_PUBLIC_HOST`. The
[observability on AWS](/docs/operations/aws) page covers CloudWatch metrics through EMF: the task logs already go
to CloudWatch Logs, so `emfMetricsExporter` needs no extra permission.

## Alarms [#alarms]

The stack publishes to its SNS topic (optionally with an `AlarmEmail` subscription) when the api answers 5xx
responses, an api or gateway target is unhealthy, no worker task is running, or Aurora runs near its maximum
capacity. Add your own on the same topic.

## Access logs [#access-logs]

Set `AccessLogsBucket` to a bucket in the same account and region, prepared for log delivery (its policy lets the
Elastic Load Balancing log delivery principal for your region and `logging.s3.amazonaws.com` write to it), and the
stack turns on ALB access logs under `alb/<name>-api/` and S3 server access logs for the four buckets under
`s3/<name>-<bucket>/`. Empty (the default) turns them off. The gateway's NLB has no access logs to give: it is a TCP
passthrough, and NLB access logs cover TLS listeners only. Keep the logs bucket outside the stack, so its retention
is yours and deleting the stack never deletes the evidence.

## Limits and choices [#limits-and-choices]

* `control-plane.json` is larger than CloudFormation's 51,200-byte inline limit, pretty-printed or minified with
  `--compact`: deploy it with `--s3-bucket`. The command says when a body is over, and offers `--compact` only when
  the minified body would fit.
* The templates use the `AWS::Partition`, `AWS::Region` and `AWS::URLSuffix` pseudo parameters for ARNs, but they
  are tested against the commercial `aws` partition only. CloudFront and some interface endpoint names differ in
  `aws-cn`.
* One set of stacks is one region. Gateways in further regions (daemons dial the gateways of their own region)
  need a network of their own and a gateway-only variant of the control-plane stack, which the templates do not
  render yet; every region keeps writing to the one database.
* Not rendered: AWS App Runner (an alternative to ECS for the API), AWS Global Accelerator in front of the
  gateways, RDS Proxy, and an AWS Budgets action. The first three are optional; the processes connect to the
  Aurora endpoint directly, and running them behind RDS Proxy is not verified (the migrator and some reconcilers
  take advisory locks, which pin proxy sessions). The live suite's README recommends a Budgets action for its
  sandbox account, set up by hand.
* The templates are checked by `lintTemplate` and unit tests only. CloudFormation's own `ValidateTemplate` runs
  in the AWS live suite (`tests/live/aws/templates.live.test.ts`), which needs sandbox credentials.
* Deletion protection is on for the database and both load balancers. Turn it off on purpose before deleting a
  stack.


# Deployment (/docs/operations/deployment)

> Lay out the api, gateway and worker processes, put the right load balancers in front, run the better-compute serve host or embed the handler in your own app.



Every process of a deployment loads the same configuration and differs only in the roles it runs. That leaves
two decisions: how many processes of each role you run and where, and whether you run them with the
`better-compute serve` host or embed the control plane in an application of your own.

## Topologies [#topologies]

### One process [#one-process]

`better-compute serve` with the default `--roles all` runs the API, every worker role, and the gateway when the
configuration exports `host.gateway`. It suits a small installation, a staging environment or a single site.
Nothing changes in the database when you later split it: leases and shard ownership already keep every job
single-writer.

```bash
better-compute migrate
better-compute serve --port 3000 --no-migrate
```

### Split roles [#split-roles]

The production shape runs each role in its own processes, all from one configuration. This is what
`examples/postgres` does:

```bash
better-compute migrate                                                    # once per deploy
better-compute serve --roles api --port 3000 --no-migrate                 # behind an L7 load balancer
better-compute serve --roles controllers,scheduler,sinks,iam --no-migrate # workers, no listener
better-compute serve --roles gateway --no-migrate                         # mTLS on :8443, behind an L4 balancer
```

Run as many of each as you need:

* **api** processes are stateless. Put them behind any HTTP load balancer.
* **Worker** processes register in the `workers` table and split the 256 `shard:<n>` leases by rendezvous hashing;
  singleton jobs (the CommandSweeper, outbox dispatch to each event sink, each IAM job) run under their own
  `singleton:<name>` lease on whichever worker holds it. Shard and singleton leases live 15 s and renew every 5 s.
* **Gateway** processes each hold a `gateway:<id>` lease (20 s, renewed every 5 s) and a row in `gateways`. A node's
  session row names the gateway that holds it, and dispatch joins on it, so any number of gateways can share a
  region.

With separate processes, give each gateway an internal control API (`host.gateway.internal`) and the api a
`notifier`. Then the api wakes the owning gateway right after it enqueues a command, relays tunnels, and pushes
revocations. Without them dispatch waits for the gateway's 1-second loop, and tunnels cannot open; `doctor` reports
both (`host.gateway.internal`, `host.gateway.notifier`, `host.gateway.tunnels`).

### A highly available region [#a-highly-available-region]

Section 14 of PLAN.md sizes a highly available region as:

| Component   | Count      | Notes                                                                                             |
| ----------- | ---------- | ------------------------------------------------------------------------------------------------- |
| api         | 3          | Behind an L7 load balancer                                                                        |
| gateway     | 3 or more  | Behind an L4 load balancer with TLS passthrough                                                   |
| worker      | 2 or more  | Shards move by rendezvous hashing when a worker joins or leaves                                   |
| PostgreSQL  | HA pair    | With PgBouncer in transaction mode                                                                |
| Better IAM  | embedded   | In the api and worker processes; compute is the IAM host family, and the `iam` role runs its jobs |
| Site agents | 2 per site | One holds the `site:<id>:boot` lease, the other stands by                                         |

It also lists an object store, an OTLP collector with a time-series database, and KMS or Vault. None of those is
required to start: the local CAs in the PKI bundle are the form the code ships, meant for development and small
deployments.

### Regions and sites [#regions-and-sites]

A region is a set of gateways with the same `host.gateway.region` (the value in `sites.region`). Daemons dial the
gateways of their own region; sites need only outbound 443. Every region's gateways write to the same database as
the api and workers.

An air-gapped installation (local object store, image mirror, `bcd` downloads) is designed in PLAN.md. Of it, the
api's `GET {basePath}/downloads/bcd/{version}/{os}-{arch}` route is built: set `downloads: { dir }` or
`downloads: { blob }` and the install script fetches `bcd` from the control plane itself. See
[installing bcd](/docs/daemon/install#serving-the-binaries).

## Load balancers [#load-balancers]

| In front of          | Kind                           | Why                                                                                                                                                                           |
| -------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| api                  | L7 (HTTP) with TLS termination | The API is plain HTTP on `--port`; `serve` itself never terminates TLS for it. `baseURL` must be the public origin, since cookie requests must come from exactly that origin. |
| gateway              | L4 (TCP) with TLS passthrough  | The gateway terminates mutual TLS itself and authenticates the daemon by its client certificate. A balancer that terminates TLS strips it.                                    |
| gateway internal API | none, or internal L4           | Each gateway advertises its own `host.gateway.internal.url` in its `gateways` row; the api dials that URL directly.                                                           |

For networks whose TLS-inspecting proxies strip client certificates, the gateway can also serve a WebSocket
fallback (v0.4) at `wss://<publicHost>/v1/session` (`host.gateway.ws`, default port 443), where the daemon proves its
identity in-band.

## The serve host [#the-serve-host]

`better-compute serve` (`packages/cli/src/host/**`) is a thin Node.js host around `compute`. On start it:

1. loads the configuration and resolves the `host` export;
2. runs `compute.initialize()` (migrations, registry checks, module start) unless `--no-migrate`;
3. installs the IAM event handlers when `host.iam` is set (give it `readTenants: iamTenantReader(iam, { credential })`
   so a tenant suspension or deletion reaches the whole subtree; see [tenancy](/docs/guides/iam/tenancy#the-iam-reaction));
4. starts the worker roles with `compute.start({ roles })`;
5. starts the gateway role: `createGateway`, the revocation feed, the mTLS listener, and the WebSocket and internal
   listeners when configured;
6. opens the HTTP listener for the `api` role.

### Routes [#routes]

The HTTP listener answers, in order:

| Path                                        | Answer                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /healthz`                              | Always `200 {"ok":true}`: the process is up.                                                                                                                                                                                                                                                                                                                                                                                    |
| `GET /readyz`                               | `200 {"ok":true}` when `SELECT 1` succeeds, else `503 {"ok":false}`.                                                                                                                                                                                                                                                                                                                                                            |
| `GET /.well-known/better-compute`           | The server's document (name, version, base path, agent protocol range) plus JoinService's public keys as `join` when `pki.join` publishes them, and `gateway: { url, serverName?, servingCaPin? }` when the options set `gateway.url` (the pin from `pki.servingCa`), which `bcd join --server` reads on a machine without `config.toml`. Cached for 300 s. `host.http.wellKnown` adds fields but never overrides the server's. |
| `host.http.routes`                          | Your own fetch-style routes, for example the events module's `GET /watch` and `GET /metrics`.                                                                                                                                                                                                                                                                                                                                   |
| `{basePath}/…`                              | The API: `POST {basePath}/{group}/{method}`, `GET {basePath}/health` and `GET {basePath}/.well-known/better-compute`. The default base path is `/api/compute`.                                                                                                                                                                                                                                                                  |
| `GET {basePath}/tunnels/{ticket}` (upgrade) | Interactive tunnels, mounted when the options set `tunnels: { issuer }`.                                                                                                                                                                                                                                                                                                                                                        |

Host routes must be absolute paths without a trailing slash, outside the base path, and must not reuse the
built-in paths; `doctor` reports a clash as `host.routes`. `GET /metrics` is added only when `host.metrics` has a
`bearerToken` of at least 16 characters; scrapers send `Authorization: Bearer <token>`. See
[observability](/docs/operations/observability).

### Health checks [#health-checks]

Use `/healthz` for liveness and `/readyz` for readiness of api processes. `{basePath}/health` also pings the store
and answers `{ status, database, latencyMs, time }`. Worker-only and gateway-only processes open no HTTP listener,
so check them through the database instead: `better-compute doctor` reports overdue work and a stalled sweeper
(`workers.queue`, `workers.sweeper`), gateways near their stream limit (`fleet.gateway-headroom`) and silent nodes
(`fleet.heartbeats`), and the `gateways` row of a live gateway has a fresh `heartbeat_at`.

### Shutdown [#shutdown]

On SIGINT or SIGTERM `serve` stops in this order:

1. The gateway drains: it tells each daemon to reconnect elsewhere (`Drain`, with a random delay of up to 60 s),
   closes its listeners with a 2 s grace, and releases its lease.
2. The HTTP listener closes, so no new requests arrive.
3. Tunnel relays close.
4. The worker roles stop and release their shard and singleton leases, and the worker leaves the `workers` table.

A process killed without that still loses nothing: its leases expire after 15 s (20 s for a gateway) and another
process takes them over, with a new epoch that fences out anything the old one still tries.

### The gateway role [#the-gateway-role]

`--roles gateway` (or `all` with `host.gateway` set) needs:

* `host.gateway.region`;
* `host.gateway.tls` with `key` and `cert` from the gateway serving CA (`pki issue-gateway`) and `ca`, the node CA
  roots, or `host.gateway.tlsFiles { key, cert, ca?, intervalMs? }` naming those files;
* a way to authenticate daemons: `pki.nodeCa` and `pki.join` in the options (from the PKI bundle), or
  `host.gateway.identity { roots, trustDomain, revocations }`. The role refuses to start without roots and a trust
  domain.

The host fills in the gateway's store, services, work queue, events, report handlers and the tunnel JWKS. Anything
else `createGateway()` takes goes in `host.gateway.options`, for example `maxStreams` (default 5,000) or vps's
`instanceStatusHandler` in `options.reports.status`. With `tlsFiles` (and `internal.tlsFiles` for the control API),
the gateway re-reads the files every 60 seconds and hot-reloads a renewed certificate for new handshakes while open
sessions keep theirs; a certificate given inline in `tls` is read once at start, so after `pki issue-gateway` writes a
new one, restart gateways one at a time. See [PKI](/docs/operations/pki#pki-issue-gateway).

In this release the gateway process builds `compute.gatewayServices` from the shared configuration, so it loads the
PKI bundle like every other process. See [security](/docs/operations/security#gateway-processes).

## Embedding in your application [#embedding-in-your-application]

`betterCompute()` returns a fetch handler and a Node handler, so you can serve the API from any framework:

  **Next.js:**

    ```ts title="app/api/compute/[...path]/route.ts"
    import { getStack } from '@/lib/stack';

    export const runtime = 'nodejs';
    export const dynamic = 'force-dynamic';

    async function handle(request: Request): Promise<Response> {
      return (await getStack()).compute.handler(request);
    }

    export { handle as GET, handle as POST };
    ```
  
  **Node:**

    ```ts title="server.ts"
    import { createServer } from 'node:http';

    await compute.initialize();
    await compute.start(); // every worker role
    createServer(compute.nodeHandler).listen(3000);
    ```
  
  **Hono:**

    ```ts title="server.ts"
    app.all('/api/compute/*', (c) => compute.handler(c.req.raw));
    ```
  
`apps/console` is the reference: Better IAM and Better Compute in one Next.js process on one origin (`/api/iam`
and `/api/compute`), with `acceptIamCookie: true` so the browser's IAM session cookie authenticates compute calls
that carry `X-Better-Compute: 1` from that origin. Next.js route handlers cannot take a WebSocket upgrade, so its
`server.mjs` wraps Next and hands `GET /api/compute/tunnels/{ticket}` to `handleTunnel` from
`better-compute/server`.

When you embed, you take over what `serve` does for you:

* call `compute.initialize()` before serving and `compute.start({ roles })` in the processes that run workers;
* register `retentionModule()` from `better-compute/postgres` yourself; the CLI adds it when the configuration does
  not (see [retention](/docs/operations/retention));
* serve your own health and readiness routes, and `/.well-known/better-compute` at the origin root if daemons use
  it (the server answers it under the base path);
* run the gateway with `createGateway` and `serveGateway` from `better-compute/gateway`, or run it with
  `serve --roles gateway` from the same configuration;
* call `compute.stop()` on shutdown so leases are released at once rather than after they expire.

### The caller address [#the-caller-address]

IAM policies can condition on `request.sourceIp`, so compute hands IAM the caller's address only as the host
verified it (d269738, security review 14 L3). `compute.nodeHandler` (and so `serve`) takes the socket's peer
address. With `compute.handler`, pass it yourself as the second argument, `HandlerInfo`:

```ts
// behind no proxy, the socket's peer is the caller
const response = await compute.handler(request, { peer: { address: socket.remoteAddress } });
```

The address becomes `Credential.sourceIp`, which remote IAM receives as `x-forwarded-for`; the client's own
forwarding headers never reach IAM (e7092bf). Since 0eba0ad that holds for an embedded IAM too: the handler strips
`X-Forwarded-For`, `Forwarded`, `X-Real-IP` and the other client-address headers and sets `X-Forwarded-For` to the
verified address, and since 004579e the in-process API does the same with a credential's `headers` and `sourceIp`,
so no path in lets a caller name its own address. The IAM caches key on the address as well, so the authentication cache (30 s) and allowed read
decisions (10 s) of a token used from one address never serve it from another. Without an address, IAM sees none.

Behind a load balancer the socket's peer is the balancer, for every caller. Name your load balancers and reverse
proxies in `trustedProxies` (0873d20), addresses or CIDR ranges, IPv4 or IPv6:

```ts title="better-compute.config.ts"
betterCompute({
  // …
  trustedProxies: ['10.0.0.0/8', 'fd00::/8'],
});
```

| Socket peer                                                   | `request.sourceIp`                                                                                         |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Not a trusted proxy                                           | The peer; forwarding headers are ignored, as without the option                                            |
| A trusted proxy                                               | `X-Forwarded-For` read right to left, skipping trusted hops: the first address that is not a trusted proxy |
| A trusted proxy, and every hop is one too                     | The leftmost hop (an internal caller)                                                                      |
| A trusted proxy with no `X-Forwarded-For`, or a malformed hop | None: it fails closed, never the proxy's own address, which an internal allowlist might match              |

IPv4-mapped IPv6 peers (`::ffff:10.0.0.1`) are compared as IPv4. An entry that is not an address or CIDR is
`INVALID_CONFIG` at startup, and the default trusts no proxy. `serve` takes the option from the configuration file like
every other, and applies it to `compute.nodeHandler`; with `compute.handler`, pass the socket's peer as above and the
option applies the same way.

## Containers [#containers]

The repository has no container image for the control plane (`apps/docs/Dockerfile` builds the documentation
site). A container runs `better-compute serve` like any other process. It needs Node.js 22.12 or newer (22.18 or
newer for a `.ts` configuration), the configuration file, and the environment below; `serve` handles SIGTERM.

| Variable                                               | Needed by                                                                   |
| ------------------------------------------------------ | --------------------------------------------------------------------------- |
| `BETTER_COMPUTE_DATABASE_URL`, `BETTER_COMPUTE_SECRET` | Every process (the examples read them; your configuration decides)          |
| `BETTER_COMPUTE_ROLES`, `PORT`, `HOST`                 | `serve` flags `--roles`, `--port`, `--host`                                 |
| `BETTER_COMPUTE_CONFIG`                                | A configuration outside the working directory and its parents               |
| `BETTER_COMPUTE_PKI`                                   | The PKI bundle path (default `.better-compute/pki.json`), mounted read-only |

Run `better-compute migrate` as a one-off job before rolling out new processes, and start them with `--no-migrate`.
See [configuration](/docs/operations/configuration#environment-variables).

## Upgrades [#upgrades]

PLAN.md orders an upgrade as:

1. expand migrations (`better-compute migrate`);
2. workers and api;
3. gateways, one at a time, each drained first;
4. daemon waves within each pool's disruption budget (see [rollouts](/docs/guides/rollouts));
5. contract migrations in the next release.

`migrate` refuses to run an older build against a database a newer build migrated, naming the migrations it does
not know, so a rollback of the code needs the matching database. See [database](/docs/operations/database#migrations).

## Examples [#examples]

Each directory under `examples/` is a workspace package with a `better-compute.config.ts`. All three run the same
deployment from `examples/shared/compose.ts`: every module (provision and metal, vps, events, fleet, and rollout and
audit when their keys exist) and a `host` export with `GET /watch`, `GET /metrics` and the gateway role when its
certificate exists.

| Example                  | Shows                                                                                                                                         | Needs                                                |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `examples/dev`           | `better-compute dev --sim 8`: simulated GPU machines join, are adopted and serve reservations, on PGlite                                      | Node.js 22.18 or newer                               |
| `examples/postgres`      | api, worker and gateway processes on PostgreSQL from one configuration, with the `pool-lease` placer; `docker-compose.yml` runs `postgres:17` | PostgreSQL 16 or newer, a PKI bundle for the gateway |
| `examples/emulated-site` | Bare metal on QEMU/KVM with Redfish BMCs from sushy-tools and PXE through a `bcd site` agent                                                  | Linux, KVM, libvirt, sushy-tools and a `bcd` build   |

`tests/examples-smoke.test.ts` checks that each configuration loads, migrates and passes `doctor`, and that
`dev --sim` adopts machines and serves `/watch`. The PostgreSQL and emulated-site walkthroughs are not run in CI
beyond that.

The PostgreSQL walkthrough joins machines from a signed release (5792b2b): the api role serves goreleaser's `dist`
through `BETTER_COMPUTE_DOWNLOADS_DIR`, `daemon install-script` takes `--release-manifest` and `--release-pins` so the
script pins the binary's digest from the manifest, and the join token comes from `join-tokens create --out`. A site
agent that drives IPMI BMCs adds `--install-plugins bcx-ipmi`, and `--unprivileged` runs `bcd node` beside a root
`bcd helper`:

```bash
better-compute daemon install-script --pool pool_… --site site_… \
  --gateway-url https://gw.eu-1.compute.example.com:8443 --base-url https://compute.example.com \
  --bcd-version 0.1.0 --release-manifest dist/manifest.json --release-pins sha256/… --out install.sh
better-compute join-tokens create --pool pool_… --ttl 2h --max-uses 8 --out join-token
```

The shared composition reads these variables:

| Variable                                                      | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BETTER_COMPUTE_DATABASE_URL`                                 | `postgres://…`, `pglite:DIR` or `pglite:memory`; default PGlite in `.better-compute/example-db`                                                                                                                                                                                                                                                                                                                                                                     |
| `BETTER_COMPUTE_SECRET`                                       | Required with PostgreSQL; PGlite examples fall back to a fixed development secret                                                                                                                                                                                                                                                                                                                                                                                   |
| `BETTER_COMPUTE_PKI`                                          | The bundle; with it JoinService, the audit module and the gateway role turn on                                                                                                                                                                                                                                                                                                                                                                                      |
| `BETTER_COMPUTE_GATEWAY_CERT`, `BETTER_COMPUTE_GATEWAY_KEY`   | `pki issue-gateway` output; with them and the bundle, `host.gateway`                                                                                                                                                                                                                                                                                                                                                                                                |
| `BETTER_COMPUTE_GATEWAY_PORT`, `BETTER_COMPUTE_REGION`        | Gateway port (default 8443) and region (default `dev-1`)                                                                                                                                                                                                                                                                                                                                                                                                            |
| `BETTER_COMPUTE_METRICS_TOKEN`                                | At least 16 characters; mounts `GET /metrics`                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `BETTER_COMPUTE_RELEASE_PINS`, `BETTER_COMPUTE_DOWNLOADS_URL` | Turn on the rollout module                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `BETTER_COMPUTE_DOWNLOADS_DIR`                                | goreleaser's `dist` directory (the bare `bcd` binaries and the plugin archives), absolute or relative to the working directory: the deployment then serves `GET /api/compute/downloads/{bcd,bcx-ipmi}/{version}/linux-{arch}` from it (`downloads: { dir }`), for install scripts, plugin installs and self-update. Without `BETTER_COMPUTE_DOWNLOADS_URL`, the rollout module's `downloadsUrl` becomes `{BETTER_COMPUTE_BASE_URL}/api/compute/downloads` (5792b2b) |
| `BETTER_COMPUTE_BASE_URL`, `BETTER_COMPUTE_ROOT_TENANT`       | Public origin and IAM root tenant                                                                                                                                                                                                                                                                                                                                                                                                                                   |

> **No IAM in the examples.** 
  The examples use `unconfiguredIam()`: no credential authenticates until you wire Better IAM through the IAM bridge.
  `better-compute dev --sim N` replaces it with an embedded, in-memory Better IAM and prints a root token.



# DigitalOcean (/docs/operations/digitalocean)

> Running compute on DigitalOcean accounts - token scopes, the per-token rate budget, an OAuth app and its redirect URI, provider costs and leaks, findings, Droplet metrics, and the opt-in live suite.



What an operator sets up and watches for DigitalOcean provider accounts. The provider itself (offers, Droplets,
networks, attach) is in the [providers guide](/docs/guides/vps/providers#digitalocean); this page is the account side.

## Token scopes [#token-scopes]

A personal access token, or an OAuth grant, needs these to run Droplets. `providerAccounts.check` (CLI
`better-compute digitalocean check <account>`) reads one list per scope and reports what the token lacks, read-only:

| Family   | Read scopes the check probes                 | Writes compute makes                              |
| -------- | -------------------------------------------- | ------------------------------------------------- |
| Compute  | `droplet:read`, `regions:read`, `sizes:read` | Droplet create, delete, power and rebuild actions |
| Networks | `vpc:read`, `firewall:read`                  | VPCs and firewalls of tenant networks             |
| Keys     | `ssh_key:read`                               | none (the account's key is named by fingerprint)  |
| Tags     | `tag:read`                                   | Tags on create (`tag:create`), claim and release  |

The operations features are optional; each needs its own read scope, and the check lists what is missing under
`optional`, never under `missing`:

| Feature                                         | Scope                                                   |
| ----------------------------------------------- | ------------------------------------------------------- |
| Provider costs (`costs: true`), billing summary | `billing:read`                                          |
| A project per tenant (`projectPerTenant`)       | `project:read`, plus `project:create`, `project:update` |
| Droplet metrics                                 | `monitoring:read`                                       |
| Security findings                               | `security:read`                                         |

A check is `ok` only when the account is `active` and nothing is `missing`. The token never appears in an answer,
row, log, event or error.

## Rate budget [#rate-budget]

DigitalOcean allows 250 requests a minute and 5,000 an hour per token, shared by everything that uses it. The
provider keeps each token under both: listing and sweeps (discovery, costs, metrics) spend at most 20% of a window,
single reads 30%, and writes whatever is left, so a busy cost or metrics pass defers before a create does. A call
over its share is `RATE_LIMITED` with `retryAfterMs`, and the reconcilers retry. Two deployments sharing one token
each keep their own budget, so give each deployment its own token.

The token's own count, as DigitalOcean's `ratelimit-*` headers report it, is the account-wide gauge:
`providerAccounts.check` (and `better-compute digitalocean check`) answers it as `rateLimit`, `{ limit, remaining,
resetAt }`, and the admin panel's **Provider accounts** page draws it under each checked account. When that count is
lower than the deployment's own (another process is using the token), the budget follows it, and sweeps and single
reads wait once less than half of the hour is left, keeping that half for writes.

## An OAuth app [#an-oauth-app]

Connecting an account through DigitalOcean's consent screen (the console's **Connect a DigitalOcean account**)
needs an OAuth app registered by the platform:

1. In DigitalOcean's control panel, under API → OAuth Applications, register an app. Its callback URL is the
   console's `https://<console>/provider-accounts/connect/callback`.

2. Pass the app to the provider, with the same URL in `redirectUris`, and a `rotate` that writes rotated refresh
   tokens back over the account's own secret:

   ```ts title="compute.ts"
   digitaloceanProvider({
     oauth: {
       clientId: process.env.DO_OAUTH_CLIENT_ID!,
       clientSecret: process.env.DO_OAUTH_CLIENT_SECRET!,
       redirectUris: ['https://console.example.com/provider-accounts/connect/callback'],
       rotate: rotateInto(secretStore),
     },
   });
   ```

3. `providerAccounts.connectSetup` answers the authorization URL for a registered redirect URI; any other redirect URI
   is `INVALID_INPUT`. The scope asked for is `read write` unless `oauth.scope` says otherwise: whether DigitalOcean's
   OAuth apps accept the granular scopes above is not verified yet.

DigitalOcean rotates the refresh token on every exchange, so a lost write-back strands the account. Every client of an
account in one process shares one access token until a minute before it expires (or until DigitalOcean refuses it),
so the refresh token rotates about once a month per process, not on every check or findings read. A refusal while the
store still holds the refused token waits about two seconds for another process's write-back (`oauth.raceWaitMs`),
then is `UNAVAILABLE` (`oauthRace`, retryable) the first time; refused again, it is `INVALID_CONFIG`
(`oauthGrant`), and the fix is to connect the account again. Only an `invalid_grant` answer counts as a refusal: a
429 from the token endpoint is `RATE_LIMITED` (with its `retry-after`), a 408 `UNAVAILABLE`, any other 4xx
`PROVIDER_ERROR`, all retryable. The token endpoint's parameters go in a form-encoded
body, never in the URL; `oauth.tokenParams: 'query'` is the fallback for DigitalOcean's documented query form.
The app's client secret is the platform's, held by the provider definition, never stored in an account.

The refresh token `providerAccounts.connect` sealed is compute's to dispose of. Deleting the account enqueues the
`vps.connect-secrets` sweep, which revokes the grant at DigitalOcean (`POST /v1/oauth/revoke`, RFC 7009, the stored
refresh token and the access token in use; `oauth.revokeEndpoint` overrides the URL) and deletes the sealed secret.
A connect that sealed a token but never stored its account (a refused account check, a crash, a restart) is
revoked and deleted the same way once it is 15 minutes old. A ref another account still names is kept. A revoke
DigitalOcean refuses is logged (`vps.connectSecrets`), and the secret is deleted anyway.

## Costs and leaks [#costs-and-leaks]

With `costs: true` on a platform account, the `vps.costs` worker reads DigitalOcean invoices (the month-to-date
preview, and final invoices of closed months) and attributes each Droplet item to the instance and tenant that own
it. Invoices carry no tags, so an item is matched by its Droplet id: to the live Droplet's labels, and after the
Droplet is deleted to the instance that held that id (instance rows keep it). A later read of the month, the final
invoice included, never takes an attribution away. An invoice item is a Droplet's month so far, so the leak check compares its hours with the hours the instance's
running and stopped intervals cover of its span (DigitalOcean bills a powered-off Droplet until it is destroyed). A
gap over `max(leakToleranceHours, 5%)` of the billed hours raises `billing.leak` once per instance and month, and
`billing.leaks` lists the raised ones for root. A leak usually means a Droplet compute no longer tracks: find it
with `better-compute digitalocean discover <account>`, then attach or delete it.

DigitalOcean spend reaches tenants' budgets through `billing.passThrough` rules on the platform account
([Billing](/docs/guides/vps/providers#billing)): an item of a tenant's Droplet that compute does not meter itself,
such as `Droplet Backups`, is billed to that tenant at cost times the rule's multiplier on a `reported` meter, so its
Better IAM budgets and budget alerts count it with the rest of its usage. Droplet hours, volumes and reserved IPs are
compute's own meters: a rule whose pattern matches them (`Droplets`) is `INVALID_CONFIG`, and they are never billed
twice. Items no tenant's Droplet explains (Spaces, an account-level charge) stay the platform's.

`providerAccounts.billing` reads the account's own bill at DigitalOcean (balance, month to date, final invoices and the
newest 50 billing history entries, scope `billing:read`), read-only, in micros of USD. Its history names payments, so
it is decided as `compute:providers:manage` on the account. A leak is what compute meters against what DigitalOcean
bills per Droplet; the bill is the account's total.

## Findings [#findings]

`providerAccounts.findings` reads the account's latest DigitalOcean security scan, read-only, as findings
(`security:<rule>`, with `critical`, `warning` or `info` severity). Without `security:read` it answers one
`security:unreadable` finding rather than an empty list that reads as a clean account. The admin panel's Provider
accounts page reads findings at most every ten minutes per account, and shows them with the billing leaks in its
attention queue.

## Droplet metrics [#droplet-metrics]

An attached Droplet runs no bcd. With DigitalOcean's metrics agent on it and `monitoring:read`, `doMetricsSource`
reads its CPU, memory and load for [host metrics of machines without bcd](/docs/operations/observability#host-metrics-of-machines-without-bcd).
With `vps({ externalMetrics: {} })` vps reads them for every live instance of the account on a schedule (the
account's `monitoring` left on), on the `sweep` budget.

## The live suite [#the-live-suite]

The emulator under `tests/support/do-emulator` covers the shapes of DigitalOcean's specification; a few behaviours
are only checked against a real account by an opt-in suite, which has not been run yet:

```sh
# Droplets, networks and the conformance cases: creates billable Droplets, always sweeps them.
BC_DO_LIVE=1 BC_DO_TOKEN=<token> BC_DO_SSH_KEY_FINGERPRINT=<md5 fingerprint of an account key> \
  node node_modules/vitest/vitest.mjs run tests/live/digitalocean.live.test.ts

# The operations reads only (probe, billing summary, this month's cost rows, findings): read-only, no sweep.
BC_DO_LIVE=1 BC_DO_TOKEN=<token> BC_DO_OPERATIONS=1 [BC_DO_METRICS_DROPLET=<droplet id>] \
  node node_modules/vitest/vitest.mjs run tests/live/digitalocean.live.test.ts -t operations
```

Optional: `BC_DO_REGION` (`nyc3`), `BC_DO_SIZE`, `BC_DO_IMAGE`, and `BC_DO_ATTACHABLE` (a hand-made Droplet the
claim and discover cases may tag and untag). Every resource a run creates carries its own tag or name prefix, and the
sweeper deletes only those.


# Requirements and sizing (/docs/operations/hardware-requirements)

> What the control plane needs and how far the measured numbers go in sizing it, the PostgreSQL version, what a node must have before bcd can run (Linux, systemd, cgroup v2, the NVIDIA driver, KVM and IOMMU for VPS hosts), the GPUs the inventory knows, and the network ports.



This page collects what each part of a deployment needs to run, and what is known about how big it has to be. The
short version: the control plane is Node.js processes and one PostgreSQL database; a node is a Linux server with
systemd and cgroup v2, plus the NVIDIA driver for GPUs and KVM with an IOMMU for virtual machines. Sizing beyond that
is not validated yet, and the page says where the numbers come from.

> **No validated sizing yet.** 
  PLAN.md fixes reference machines for its performance gates, but no benchmark has run on them, and no run on real
  PostgreSQL is recorded. Every number below was measured on a development workstation, mostly on PGlite in one
  process, and is a floor, not a capacity plan. See [scaling](/docs/operations/scaling) for every measurement and its
  conditions.

## The control plane [#the-control-plane]

### Software [#software]

| Requirement            | Notes                                                                                                           |
| ---------------------- | --------------------------------------------------------------------------------------------------------------- |
| Node.js 22.12 or newer | Every package's `engines`. A TypeScript configuration file needs 22.18 or newer. CI runs Node.js 22 and 24      |
| PostgreSQL 16 or newer | See [below](#postgresql)                                                                                        |
| PgBouncer, optional    | Transaction mode is supported: no session state, no `LISTEN`/`NOTIFY`, no session advisory locks                |
| Better IAM             | Embedded in the api and worker processes, or remote; see [installation](/docs/guides/installation#requirements) |

The control plane has no container image of its own; any host or container that runs Node.js runs
`better-compute serve` (see [containers](/docs/operations/deployment#containers)).

### PostgreSQL [#postgresql]

| Requirement                 | Why                                                                                                                                                                                                 |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL 16 or newer      | `migrate()` reads `server_version_num` and refuses anything below 160000 with `INVALID_CONFIG`. CI runs 17, and the reference machine runs 17                                                       |
| The `btree_gist` extension  | The first migration creates it; the booking exclusion constraint needs it. On a managed service that does not let the database owner create it, a superuser runs `CREATE EXTENSION btree_gist` once |
| `CREATE` on the database    | Migrations create tables, indexes and functions                                                                                                                                                     |
| One database per deployment | api, workers and gateways all connect to it; it is the only source of truth                                                                                                                         |

PGlite (PostgreSQL in WebAssembly) is for development, tests and the simulators: one connection, transactions
serialized, inside one process. It cannot be shared by separate processes, so it cannot run a split or highly
available deployment. Details are on [database](/docs/operations/database#requirements).

**Connections.** Each process holds its own pool, 10 connections by default (`postgres({ max })`, 1 to 100), so the
total is roughly the pool size times the number of processes. PLAN.md lists connections among the first limits at
scale and puts PgBouncer in front of PostgreSQL.

### Reference machines [#reference-machines]

`bench/REFERENCE.md` fixes the machines the PLAN.md §16 gates are quoted on:

| Role       | CPU     | Memory | Storage | Software                            |
| ---------- | ------- | ------ | ------- | ----------------------------------- |
| Gateway    | 8 vCPU  | 32 GiB | —       | One `gateway` process on Node.js 24 |
| PostgreSQL | 16 vCPU | 64 GiB | NVMe    | PostgreSQL 17                       |

Not fixed yet (decision D15): the instance type or CPU model, the network between hosts, the hosts for the api,
workers and load generators, PgBouncer's placement and the PostgreSQL configuration.

### What has been measured [#what-has-been-measured]

| Quantity                                        | Measured (workstation, mostly PGlite)                                                                                                                                               | Gate                                        |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| Sessions one gateway holds                      | 5,000 mTLS sessions at 10 s heartbeats, dispatch p99 24 to 26 ms, 0 drops                                                                                                           | 5,000 for 30 min, dispatch p99 below 500 ms |
| Gateway memory at 5,000 sessions                | 1,240 MiB RSS for the gateway process over 60 s with PGlite inside it; 2,016 MiB after 30 min, the growth traced to PGlite's WebAssembly memory while the JS heap stayed at 235 MiB | RSS below 2 GiB                             |
| Commands                                        | ack p99 378 ms at 100 commands/s on 1,000 nodes                                                                                                                                     | p99 below 1.5 s at v0.4                     |
| Placements                                      | 1,092 placements/s at 1,000 nodes and 464/s at 10,000 nodes (latest recorded run), pool-lease placer end to end                                                                     | 200/s at v0.4                               |
| Gateway kill                                    | re-home p99 13.25 s at 10,000 nodes over 3 gateways (simulated time)                                                                                                                | below 60 s                                  |
| Cold start                                      | 10,000 nodes connected, adopted and schedulable at 80 s (simulated time)                                                                                                            | below 3 min                                 |
| PostgreSQL CPU at 10,000 nodes and 100,000 GPUs | not measured                                                                                                                                                                        | below 50% on the reference PostgreSQL       |

### Sizing each tier [#sizing-each-tier]

**Recommendations**, derived from the defaults and the numbers above, not validated on production hardware:

| Tier       | Scales by                                                                                                    | Starting point                                                                                                                                                                                                                                                                                                           |
| ---------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Gateways   | Sessions: `maxStreams` is 5,000 per process by default, and `doctor` warns at 80% (`fleet.gateway-headroom`) | Enough gateways that the region's nodes fit in the survivors after losing one, and at least three per region (PLAN.md). For 10,000 nodes: three gateways hold about 3,333 each, and two survivors hold 5,000 each, exactly at the limit, so run four. Give each the reference 8 vCPU and 32 GiB until measured otherwise |
| api        | Requests; stateless                                                                                          | Three behind an L7 balancer for a highly available region (PLAN.md)                                                                                                                                                                                                                                                      |
| Workers    | Reconcile throughput; 256 shards split among them                                                            | Two or more, so a lost worker fails over (see [high availability](/docs/operations/high-availability)); add `scheduler` processes for the pool-lease placer                                                                                                                                                              |
| PostgreSQL | Writes per second, connections                                                                               | The reference 16 vCPU, 64 GiB and NVMe is what the 10,000-node gates are to be measured on; PLAN.md targets below 50% CPU there                                                                                                                                                                                          |

PLAN.md's design point is 10,000 nodes and 100,000 GPUs per region on one PostgreSQL, and its estimate of the load at
that size (liveness writes, reconnect storms, commands, events, placements) and the order in which limits are expected
to bite are on [what breaks first](/docs/operations/scaling#what-breaks-first).

**Storage.** `audit_events`, `inventory_snapshots` and `usage_windows` grow with time and are partitioned by month;
the retention module prunes and drops what every cursor has passed (see [retention](/docs/operations/retention)). No
growth rate per node has been measured.

## Nodes [#nodes]

### Platform [#platform]

| Requirement                                 | Checked by                                                            | Notes                                                                                                                                                     |
| ------------------------------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Linux on amd64 or arm64                     | The install script                                                    | `uname -m` of `x86_64`/`amd64` or `aarch64`/`arm64`; releases build only those two                                                                        |
| Root for the install                        | The install script                                                    | Also `systemctl`, `sha256sum`, and `curl` or `wget` with HTTPS                                                                                            |
| systemd as init                             | `bcd doctor` `systemd` (fail)                                         | bcd's hold slices, the helper's units and libvirt's scopes are systemd units                                                                              |
| cgroup v2 with `cpuset`, `cpu` and `memory` | `bcd doctor` `cgroups` (fail)                                         | Holds with limits are confined in cgroup slices; without `hugetlb` (warn) a hold's huge page limits are not enforced                                      |
| Controllers delegated at the cgroup root    | `bcd doctor` `delegation` (warn)                                      | `cpu`, `cpuset`, `memory`, `io`, `pids`, `hugetlb` in `cgroup.subtree_control`; systemd usually enables them once bcd's slices exist                      |
| A synchronized clock                        | `bcd doctor` `clock` (fail when the kernel reports it unsynchronized) | Certificates, exec tickets and signed intents compare times; run an NTP client such as systemd-timesyncd or chrony (e2a0cf0)                              |
| A persistent `data_dir`                     | —                                                                     | `/var/lib/bcd` by default: the node key, identity, journal and, on VPS hosts, instance disks and volumes                                                  |
| Outbound HTTPS to the gateway               | `bcd doctor` `gateway` (fail when no TCP connection within 5 s)       | Nodes only dial out; see [ports](#network-and-ports)                                                                                                      |
| TPM 2.0, optional                           | `/dev/tpmrm0`                                                         | With one, the node key is made in the TPM, and the `tpm` attestor can join the machine without a token; see [attestation](/docs/guides/nodes/attestation) |

No minimum kernel version is checked anywhere in bcd. The full list of `bcd doctor` checks is on
[the daemon overview](/docs/daemon#modes); run it before the first join (the install script does, and only warns).

### GPU hosts [#gpu-hosts]

bcd reads GPUs through `nvidia-smi` and needs no NVML library or cgo (cb987e5): a host build without `nvidia-smi`
reports no GPUs, and `bcd doctor`'s `nvml` check warns.

| Requirement                                  | Checked by                                                                     | Notes                                                                                                                       |
| -------------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| The NVIDIA driver, with `nvidia-smi`         | `nvml` (warn without it), `gpu-bus`                                            | The [`nvidia-driver` host unit](/docs/daemon/host-config#nvidia-driver) can install it from desired state                   |
| `nouveau` not loaded                         | `nouveau` (fail)                                                               | Blacklist it and rebuild the initramfs so the NVIDIA driver can bind the GPUs                                               |
| Every GPU on the bus bound and seen          | `gpu-bus` (fail when `nvidia-smi` sees fewer GPUs than are bound to `nvidia`)  | A GPU that fell off the bus (Xid 79) needs `gpu.reset` or a reboot                                                          |
| Persistence mode                             | `persistence` (warn)                                                           | Without `nvidia-persistenced`, every first CUDA call re-initializes the GPU                                                 |
| `nvidia-fabricmanager` on NVSwitch systems   | `fabric-manager` (fail when NVSwitches are present and the unit is not active) | CUDA fails on every GPU until it runs; the [`fabric-manager` host unit](/docs/daemon/host-config#fabric-manager) manages it |
| The NVIDIA Container Toolkit, for containers | —                                                                              | CDI specs for holds; see the [`container-toolkit` host unit](/docs/daemon/host-config#container-toolkit)                    |
| A healthy GPU                                | `gpu` per GPU                                                                  | A pending row remap or MIG change warns; failed row remapping or uncorrectable ECC errors fail                              |

On WSL 2 (`/dev/dxg` without `/dev/nvidiactl`), doctor says the paravirtual GPU runs CUDA but supports no VFIO
passthrough, `gpu.reset`, MIG or DCGM, and the `nvidia-driver` unit never installs a Linux driver there (2e71a60).

### VPS and sandbox hosts [#vps-and-sandbox-hosts]

| Requirement                                                                                                          | Checked by                                                       | Notes                                                                                                                                                                |
| -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/dev/kvm`                                                                                                           | `kvm` (warn)                                                     | Doctor says why it is missing: VT-x/AMD-V off in the firmware, the `kvm_intel`/`kvm_amd` module not loaded, or a VM without nested virtualization                    |
| An IOMMU, for GPU passthrough                                                                                        | `iommu` (warn)                                                   | The [`iommu` host unit](/docs/daemon/host-config#iommu) sets the kernel arguments                                                                                    |
| `vfio-pci`                                                                                                           | `vfio` (warn)                                                    | Loaded on first passthrough; doctor checks it is available                                                                                                           |
| Each GPU in an IOMMU group of its own                                                                                | `iommu-groups` (warn)                                            | Passthrough hands the VM the whole group, so a NIC or NVMe drive sharing a GPU's group makes that GPU unusable for VMs; move it to a CPU-attached slot or enable ACS |
| `libvirt-kvm`: `virsh`, `qemu-img`, `genisoimage` or `xorriso`, `nft`                                                | Doctor fails a host missing `qemu-img`, both ISO tools, or `nft` | Under `bcd helper`, also `dynamic_ownership = 0` and the group setup of the `libvirt` check                                                                          |
| `cloud-hypervisor`: the binary, `/usr/share/cloud-hypervisor/CLOUDHV.fd`, `systemd-run`, systemd 247 or newer, `nft` | `cloud-hypervisor`, `systemd`                                    | One confined systemd unit per VMM                                                                                                                                    |
| Huge pages, for flavors that ask for them                                                                            | `hugepages`                                                      | The [`hugepages` host unit](/docs/daemon/host-config#hugepages) reserves them                                                                                        |
| Disk for volumes                                                                                                     | —                                                                | A volume must fit in the data directory's free space less a reserve of 5% (at least 2 GiB) less every volume's unwritten space                                       |

### Site agents and Kubernetes [#site-agents-and-kubernetes]

A bare-metal site agent needs its provisioning network, the iPXE builds and a ramdisk image you supply, and its own
systemd unit for the privileged ports; see [site agents and the ramdisk](/docs/daemon/install#site-agents-and-the-ramdisk).
bcd on an existing Kubernetes cluster runs as a DaemonSet with its own host mounts; see
[bcd on Kubernetes](/docs/daemon/kubernetes).

### What has run on real hardware [#what-has-run-on-real-hardware]

| Hardware                                                | Status                                                                                                                                                                                                                                         |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A WSL 2 host with one GeForce RTX 5090 (driver R616)    | bcd built without `fakehw` joins a dev stack over mTLS, is adopted, reports the real inventory and GPU, and answers `bcd status` and `bcd doctor` (`tests/e2e/bcd-host-real.test.ts`, opt in with `BCD_E2E_REAL=1`; 0915ab7, 92e3385, 898a23c) |
| Data-center GPUs (H100, A100 and the rest)              | Not run. The fake hardware build reports `NVIDIA H100 80GB HBM3`; the `nvidia-smi` parsing is tested against fixtures                                                                                                                          |
| `libvirt-kvm` and `cloud-hypervisor` on a real KVM host | Not run; tested against fakes                                                                                                                                                                                                                  |
| Host config units on a real machine                     | Not run, apart from `TestRealKernel`'s sysctl, transparent huge pages and 2 MiB pages                                                                                                                                                          |
| TPM joins                                               | Tested with the software TPM only                                                                                                                                                                                                              |

## Supported GPUs [#supported-gpus]

The inventory normalizes each GPU's NVML name into a product and a family through a versioned table
(`gpuModels`, version 2, in `packages/inventory/src/gpu-models.ts`). Pools, policies and meters use the family. A name
the table does not know maps to the family `other`, which raises the `GpuModelUnknown` condition and keeps the node
from scheduling until a release adds it. The normalization rules are on [GPU naming](/docs/guides/nodes/inventory-and-health#gpu-naming).

| Family         | Product                         | NVML names                                              |
| -------------- | ------------------------------- | ------------------------------------------------------- |
| `H100`         | `H100-SXM5-80GB`                | NVIDIA H100 80GB HBM3                                   |
| `H100`         | `H100-PCIE-80GB`                | NVIDIA H100 PCIe                                        |
| `H100`         | `H100-NVL-94GB`                 | NVIDIA H100 NVL                                         |
| `H200`         | `H200-SXM5-141GB`               | NVIDIA H200                                             |
| `H200`         | `H200-NVL-141GB`                | NVIDIA H200 NVL                                         |
| `B200`         | `B200-SXM-180GB`                | NVIDIA B200                                             |
| `GB200`        | `GB200-SUPERCHIP-186GB`         | NVIDIA GB200                                            |
| `GH200`        | `GH200-SUPERCHIP-96GB`          | NVIDIA GH200 480GB, NVIDIA GH200 96GB HBM3              |
| `GH200`        | `GH200-SUPERCHIP-144GB`         | NVIDIA GH200 144G HBM3e                                 |
| `A100`         | `A100-SXM4-40GB`                | NVIDIA A100-SXM4-40GB                                   |
| `A100`         | `A100-SXM4-80GB`                | NVIDIA A100-SXM4-80GB, NVIDIA A100-SXM-80GB             |
| `A100`         | `A100-PCIE-40GB`                | NVIDIA A100-PCIE-40GB                                   |
| `A100`         | `A100-PCIE-80GB`                | NVIDIA A100 80GB PCIe                                   |
| `L40S`         | `L40S-PCIE-48GB`                | NVIDIA L40S                                             |
| `L40`          | `L40-PCIE-48GB`                 | NVIDIA L40                                              |
| `L4`           | `L4-PCIE-24GB`                  | NVIDIA L4                                               |
| `A40`          | `A40-PCIE-48GB`                 | NVIDIA A40                                              |
| `A30`          | `A30-PCIE-24GB`                 | NVIDIA A30                                              |
| `A10`          | `A10-PCIE-24GB`                 | NVIDIA A10                                              |
| `RTX-PRO-6000` | `RTX-PRO-6000-SERVER-96GB`      | NVIDIA RTX PRO 6000 Blackwell Server Edition            |
| `RTX-PRO-6000` | `RTX-PRO-6000-WORKSTATION-96GB` | NVIDIA RTX PRO 6000 Blackwell Workstation Edition       |
| `RTX-PRO-6000` | `RTX-PRO-6000-MAXQ-96GB`        | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition |
| `RTX-6000-ADA` | `RTX-6000-ADA-PCIE-48GB`        | NVIDIA RTX 6000 Ada Generation                          |
| `RTX-A6000`    | `RTX-A6000-PCIE-48GB`           | NVIDIA RTX A6000                                        |
| `RTX-5090`     | `RTX-5090-PCIE-32GB`            | NVIDIA GeForce RTX 5090                                 |
| `RTX-5080`     | `RTX-5080-PCIE-16GB`            | NVIDIA GeForce RTX 5080                                 |
| `RTX-4090`     | `RTX-4090-PCIE-24GB`            | NVIDIA GeForce RTX 4090                                 |

"Supported" here means the inventory names the card and pools can hold it. What a card can do beyond whole-GPU holds
depends on the card: MIG needs a MIG-capable part (see [fractional GPUs](/docs/guides/capacity/fractional-gpus)),
passthrough needs the IOMMU setup above. Only NVIDIA GPUs are inventoried.

## Network and ports [#network-and-ports]

Nodes only dial out. Everything the control plane sends a node travels down the session the node opened.

| From                                  | To                                                    | Port, default                                                                             | Protocol                                              | Notes                                                                                                                                   |
| ------------------------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Clients, browsers                     | api, through an L7 balancer                           | `--port`, 3000                                                                            | HTTP; the balancer terminates TLS                     | `serve` never terminates TLS for the API; `baseURL` is the public origin. `/healthz`, `/readyz` and `/metrics` are on the same listener |
| Daemons                               | Gateways, through an L4 balancer with TLS passthrough | The port in `gateway_url`; the gateway listens on 8443 by default (`host.gateway.listen`) | gRPC over HTTP/2, mutual TLS 1.3                      | A balancer that terminates TLS strips the client certificate                                                                            |
| Daemons behind a TLS-inspecting proxy | Gateway WebSocket fallback                            | 443 (`host.gateway.ws.listen`)                                                            | `wss://…/v1/session`                                  | Identity proven in-band; see [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04)                                         |
| api                                   | Each gateway's control API                            | 8444 (`host.gateway.internal.listen`)                                                     | HTTP/2, mutual TLS from the internal CA               | Pokes, tunnels, revocations; each gateway advertises its `internal_url`                                                                 |
| api, workers, gateways                | PostgreSQL (or PgBouncer)                             | Your database's                                                                           | PostgreSQL                                            |                                                                                                                                         |
| Nodes                                 | The control plane's downloads                         | HTTPS                                                                                     | `GET {basePath}/downloads/bcd/{version}/linux-{arch}` | Install script and self-update; the digest is checked against the signed manifest                                                       |
| VPS hosts                             | Image sources, snapshot URLs                          | HTTPS                                                                                     |                                                       | Base images and snapshots are SHA-256 verified; `http://` is refused                                                                    |
| Nodes with a `telemetry` section      | Your OTLP collector                                   | Its endpoint                                                                              | OTLP/HTTP                                             | See [node metrics](/docs/operations/observability#node-metrics-from-bcd)                                                                |
| Machines on the provisioning network  | Site agent                                            | UDP 67 and 4011 (proxy DHCP), UDP 69 (TFTP), `[boot] listen` (HTTP)                       |                                                       | Privileged ports; the install script's unit does not grant `CAP_NET_BIND_SERVICE`                                                       |
| Site agent                            | BMCs                                                  | HTTPS for Redfish                                                                         | Redfish, or IPMI over LAN through `bcx-ipmi`          | Name the BMC networks in `[site] bmc_networks` so `site.httpProxy` never reaches into them                                              |
| Instances                             | Their host's metadata service                         | `[imds] listen`, for example `169.254.169.254:80`                                         | HTTP                                                  | Answers only its own running instances                                                                                                  |
| Hosts and services                    | The workload identity issuer                          | 443                                                                                       | HTTPS                                                 | The issuer URL must be https on 443                                                                                                     |

  - [Scaling](/docs/operations/scaling): Every measurement and its conditions.

  - [Installing bcd](/docs/daemon/install): The install script and the files it writes.

  - [Host configuration](/docs/daemon/host-config): Drivers, IOMMU and huge pages from desired state.

  - [Deployment](/docs/operations/deployment): Roles, balancers and topologies.


# High availability (/docs/operations/high-availability)

> How worker processes share 256 shards and the singleton jobs through table leases, how a dead worker's or gateway's work moves and how fast, what fences a process that lost its lease, and why it all needs PostgreSQL.



Better Compute has no leader election, no consensus protocol and no coordination service. Every process that must
be the only one doing something holds a row in the `leases` table, and every write that depends on that holds the
lease row for the length of its transaction. Availability then comes down to three questions: how many processes of
each role you run, how quickly a dead process's leases move, and what stops a process that has lost its lease from
writing anyway. This page answers them from the code and the tests. The topology itself (how many of each role,
behind which balancer) is on [deployment](/docs/operations/deployment#a-highly-available-region).

> **Where the numbers come from.** 
  The failover times below come from `tests/realtime/lease-ha.test.ts`, two worker processes on one PGlite store on the
  wall clock, and from the gateway-kill benchmark in the fleet simulator, which runs in simulated time. Neither is a
  measurement on PostgreSQL or on separate hosts.

## What needs a lease [#what-needs-a-lease]

| Role                        | State                  | Single-writer by                                                     | Survives losing a process                                        |
| --------------------------- | ---------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------- |
| api                         | None                   | Nothing: stateless                                                   | Yes, behind a load balancer; clients retry with idempotency keys |
| Worker, sharded reconcilers | Work-queue claims      | `shard:<n>` leases (or `scheduler/shard:<n>`)                        | Yes: shards move to the surviving workers                        |
| Worker, singleton jobs      | Cursors, sweeps        | One `singleton:<job>` lease each                                     | Yes: the lease moves to another worker                           |
| Gateway                     | Sessions in memory     | One `gateway:<id>` lease per process, and the per-node session epoch | Its sessions are lost; nodes redial another gateway              |
| Site agent (bare metal)     | The active boot server | `site:<id>:boot`                                                     | Yes, with a standby agent at the site                            |
| PostgreSQL                  | Everything durable     | —                                                                    | Only through your PostgreSQL HA                                  |

Lease names are checked against `^[A-Za-z0-9._:@/-]{1,200}$` and TTLs are whole milliseconds up to a day. The name
`pool:<id>` is reserved in the code but held by nothing today: the pool-lease placer runs under the shard lease of
its pool's shard. The metrics database collector takes one more row, `metrics:command-latency`, so that only one
process observes command latency (see [the database collector](/docs/operations/observability#the-database-collector)).

## Leases [#leases]

A lease is a row `(name, owner, epoch, expires_at)`. Acquiring and renewing are the same guarded update, so there is
no separate election step:

```sql
UPDATE leases SET owner = $2,
    epoch = CASE WHEN owner = $2 THEN epoch ELSE epoch + 1 END,
    expires_at = compute_now() + make_interval(secs => $3::float8)
  WHERE name = ANY($1::text[]) AND (owner = $2 OR expires_at < compute_now())
  RETURNING name, epoch, expires_at, compute_now() AS now
```

* A process takes a lease only when it already holds it or the lease has expired. A live lease never changes hands.
* The epoch rises on every change of owner, never on a renewal. A missing row is first inserted unowned and expired,
  so the first holder gets epoch 1.
* All of a worker's leases are renewed in one statement per pass (`acquireMany`).
* Time is `compute_now()`, the database's clock, never the process's. Every renewal also samples it, and a process
  more than 1 second off raises `ClockSkew` (see [one clock](/docs/operations/database#one-clock)).
* Releasing sets `owner = NULL` and `expires_at = '-infinity'`, so a successor need not wait for the TTL.

### Fencing [#fencing]

Holding a lease is not enough: a process can pause (a long GC, a stalled VM) past its TTL and wake up believing it
still owns the shard. So every transaction a reconciler or singleton job opens through `ctx.sql` starts with the
fence, and a read-only transaction skips it:

```sql
SELECT 1 AS held FROM leases
 WHERE name = $1 AND owner = $2 AND epoch = $3 AND expires_at > compute_now() FOR SHARE
```

If the row no longer matches, the transaction fails with `CONFLICT` ("Lease … is no longer held") and nothing it
would have written commits. If it does match, the `FOR SHARE` lock makes a takeover's `UPDATE` wait until the
transaction ends, so the old holder either finishes its transaction before the new one starts or never starts it. A
bare `EXISTS` would not fence under `read committed`. Nested transactions are savepoints of the fenced one. See
[leases and fencing](/docs/guides/concepts/consistency#leases-and-fencing) for why this, rather than an advisory
lock, keeps PgBouncer's transaction pooling safe.

## Workers and shards [#workers-and-shards]

Every sharded reconciler key (a node, a pool) maps to one of 256 shards by FNV-1a hash (`shardOf` in
`@better-compute/core`). Worker processes share the shards like this:

1. **Heartbeat.** Each worker upserts its row in `workers` (`heartbeat_at = compute_now()`) on every membership pass.
2. **Live set.** The live workers of its group are those that heartbeated within `liveWorkerMs` (30 seconds).
3. **Rendezvous hashing.** For each shard, the owner is the live worker with the highest
   `hash32("<worker>#<shard>")` (FNV-1a with murmur3's finalizer; ties go to the lower id). Every worker computes the
   same answer from the same rows, with no coordination.
4. **Take and let go.** A worker releases the shard leases it holds but no longer owns, then acquires or renews the
   ones it owns, 15-second TTL.

Adding a worker moves only the shards rendezvous hashing now assigns to it, about 1/N of them; the others stay put.
A shard moves only when its holder lets it go on its next pass, or when its lease expires.

**Groups.** Each worker role that runs sharded reconcilers has a membership of its own. The `controllers` group uses
rows `<id>` and leases `shard:<n>`; the `scheduler` group, which runs reconcilers that declare `role: 'scheduler'`
(the pool-lease placer), uses rows `scheduler/<id>` and leases `scheduler/shard:<n>`. So a scheduler-only process
never takes node shards, and the placers scale apart from the controllers. A role with no sharded reconciler
registered holds no shard leases at all.

**Orphan shards.** A worker that stops renewing but whose heartbeat is not yet 30 seconds old is still in the live
set, so rendezvous hashing still names it. Since 4eb0b7e the others do not wait for it to leave: a desired owner
whose heartbeat is older than one lease TTL (15 seconds) is silent, and its shards are orphans. Every survivor tries
to take the orphans on each pass, which succeeds only once a lease has expired, so the first survivor to try after
expiry holds it. It keeps an orphan until the silent worker heartbeats again (the shard is then let go and its
desired owner takes it back) or drops out of the live set (rendezvous hashing then names a new owner among the
survivors). The acquire bumps the epoch, so fencing is unchanged. Rows of workers silent for 20 times `liveWorkerMs`
(10 minutes) are deleted.

### The renewal loop [#the-renewal-loop]

Renewals run on a timer of their own, the `lease-renewal` loop, never inside a reconcile batch, so a long batch
cannot starve them (4eb0b7e). The loop ticks every `max(50, min(1000, renewMs / 5))` milliseconds (1 second by
default); a tick costs nothing unless something is due, and then the membership pass and every due singleton are
renewed in one statement each. A membership pass is due every `renewMs` (5 seconds), or earlier when a held lease
has less than a third of its TTL left. A singleton is due once it was last taken `renewMs` ago.

| Setting                       | Value             | Configurable        |
| ----------------------------- | ----------------- | ------------------- |
| Shard and singleton lease TTL | 15 s              | No (`leaseTtlMs`)   |
| Renewal period                | 5 s               | No (`renewMs`)      |
| Live worker window            | 30 s              | No (`liveWorkerMs`) |
| CommandSweeper period         | 5 s               | No                  |
| Claimed work item lease       | 60 s              | `worker.claimTtlMs` |
| Level-trigger resync          | 5 min plus jitter | `worker.resyncMs`   |

The configurable worker options are on [configuration](/docs/operations/configuration#worker). The work queue's own
claims (`claimed_until`) are a second, independent safety net: a key claimed by a worker that died becomes claimable
again after `claimTtlMs`, and a successful run deletes it only at the generation it claimed, so no wake-up is lost.

### Singleton jobs [#singleton-jobs]

Jobs that must run in exactly one place take a `singleton:<name>` lease on whichever worker gets it first:

| Lease                              | Job                                                                                                                                                                                                                                      | Worker role                                               |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `singleton:command-sweeper`        | The CommandSweeper: expires and fails commands past their deadlines                                                                                                                                                                      | `controllers`                                             |
| `singleton:<reconciler>`           | Each reconciler declared `shardBy: 'none'`: among them the Liveness controller, site boot, `capacity.reservations`, `capacity.usage`, retention, the rollout controller, the quorum batches, audit checkpoints and the IAM bridge's jobs | `controllers` (or `scheduler` for a scheduler reconciler) |
| `singleton:outbox-dispatch:<sink>` | Outbox delivery to one event sink, from its stored cursor                                                                                                                                                                                | `sinks`                                                   |
| `singleton:iam:<job>`              | Each IAM job of the embedded Better IAM host                                                                                                                                                                                             | `iam`                                                     |

One lease per sink means several `sinks` workers split the sinks between them and a slow sink never holds the
others back. A new epoch on a sink lease means this process has just taken it over and resumes from the cursor the
previous holder saved after its last delivered batch, so delivery is at least once.

### Worker failover [#worker-failover]

`tests/realtime/lease-ha.test.ts` starts worker A with the `controllers` role, lets it take every shard and singleton
lease, starts worker B, then kills A's database connection (no release, nothing it was running commits) while keys
are enqueued continuously. It asserts that B holds every lease A held within the TTL plus a renewal period plus 2
seconds of slack (22 seconds), that B keeps working afterwards, and that no key ever ran on both workers at once.

| Measured on PGlite              | Before 4eb0b7e                                                      | Since 4eb0b7e                               |
| ------------------------------- | ------------------------------------------------------------------- | ------------------------------------------- |
| Singleton leases taken over     | about 13 s                                                          | about 13 s                                  |
| All 256 shard leases taken over | 30 to 34 s (waited for A's heartbeat to leave the 30 s live window) | about 18.7 s (orphans taken as they expire) |
| Lease renewal gap, p99          | 10.2 s                                                              | 5.2 s                                       |

The theoretical bound for a killed worker is the 15-second TTL plus up to one 5-second renewal period. A worker
stopped with SIGTERM releases its leases and leaves the `workers` table at once (see
[shutdown](/docs/operations/deployment#shutdown)), so its shards and singletons move on the survivors' next pass.

While no worker runs at all, nothing new is decided: gateways keep dispatching commands that are already pending
and keep ingesting results, but no reconciler runs and no command is swept. `doctor` reports work more than 5
minutes overdue as `workers.queue`, adding "and no worker is alive" when no worker heartbeated in the last minute,
and a stalled sweeper as `workers.sweeper`.

## Gateways [#gateways]

A gateway holds sessions in memory and nothing durable; see [memory and PostgreSQL](/docs/daemon/gateway#memory-and-postgresql).
Its availability rests on two fences.

**The gateway lease.** Each gateway process holds `gateway:<id>` (20-second TTL, renewed every 5 seconds; both
configurable through `host.gateway.options` as `gatewayLeaseTtlMs` and `leaseRenewMs`). The id is a fresh `gw_…` per
process unless you set one. Every routing transaction (the `Hello` transaction, dispatch) starts by re-reading the
lease `FOR SHARE`. When another owner holds it, or renewals have failed for half the TTL (10 seconds), the gateway
fences itself: it forgets the lease and closes every stream `UNAVAILABLE`. No other gateway ever takes a dead
gateway's lease over; it simply expires. See [the lease and fencing](/docs/daemon/gateway#the-lease-and-fencing).

**The session epoch.** Each node's `node_sessions` row carries a `session_epoch` that every `Hello` bumps while it
routes the node to its own gateway. Routing and liveness writes carry `AND session_epoch = $myEpoch`, so a gateway
whose node has said `Hello` elsewhere matches no row, learns that its session is a zombie and closes it. Results and
events are facts and are accepted from any epoch, deduplicated. See [epochs and fencing](/docs/daemon/sessions#epochs-and-fencing).

### Several gateways [#several-gateways]

Gateways share nothing but PostgreSQL. A command is routed to a gateway by a join on `node_sessions.gateway_id` at
dispatch time, so when a node reconnects through another gateway nothing needs rewriting, and `FOR UPDATE SKIP
LOCKED` lets every gateway dispatch side by side. Run three or more per region behind an L4 balancer with TLS
passthrough, each with headroom below `maxStreams` (5,000 by default) for the sessions of a lost peer: with three
gateways, losing one moves a third of the region's nodes onto the other two. See
[several gateways](/docs/daemon/gateway#several-gateways) and [admission and scale](/docs/daemon/gateway#admission-and-scale).

### Gateway failover [#gateway-failover]

When a gateway dies:

1. Its daemons see their streams end and redial with full-jitter back-off (0 to 1 second at first, doubling to 60
   seconds); the balancer sends them to a live gateway, whose handshake bucket (200 per second), `maxPendingHellos`
   (64) and retry pushback pace the herd.
2. Each `Hello` bumps the epoch, routes the node to the new gateway and replays in-flight commands; the daemon's
   journal deduplicates them.
3. Nodes that do not come back become `disconnected` once the dead gateway's lease expires (20 seconds), and
   `unreachable` only after the 40-second session lease plus a 120-second gateway-loss grace. Neither transition
   releases an allocation.

In the fleet simulator with three real gateways and the real control plane (`bench/multi-gateway`, simulated time),
killing one gateway re-homed its nodes with these results:

| Nodes  | Moved | Re-home p50 | p99     | Invariant violations |
| ------ | ----- | ----------- | ------- | -------------------- |
| 1,000  | 334   | 0.5 s       | 1.0 s   | 0                    |
| 10,000 | 3,334 | 3.25 s      | 13.25 s | 0                    |

PLAN.md's v0.4 gate is recovery under 60 seconds with no false unreachable marks and no duplicate executions. The
details, and the zombie-gateway case, are on [failure modes](/docs/daemon/failure-modes#gateway-crash).

A planned restart should drain instead: `serve` on SIGTERM sends every daemon `Drain` with a random delay of up to 60
seconds and releases the lease, so the region absorbs the reconnects gradually.

### Tunnels [#tunnels]

An interactive tunnel is relayed by the gateway that holds the node's session and dies with it. The caller opens a
new one. Nothing else about tunnels is replicated.

## Site agents [#site-agents]

A bare-metal site runs two site agents. The `site:<id>:boot` lease (15-second TTL) names the one connected agent that
answers proxy DHCP and iPXE and vouches for discovery; its epoch rises on every change of holder and is published to
the holder in its NodeSpec (`site.boot = { active, epoch }`). A holder that disconnects keeps the lease until it
expires and self-fences after the TTL; only then is the first connected standby (by node id) promoted. The lease
never changes hands while it is live, so two agents never both believe they are active at one epoch. BMC access and
the image cache work on both agents at all times. See [one active boot server per site](/docs/guides/bare-metal/sites#one-active-boot-server-per-site).

## PostgreSQL [#postgresql]

Every mechanism above is a table in one PostgreSQL database, so the database is the one component whose availability
is not provided by the code.

**Why PostgreSQL and not PGlite.** PGlite runs PostgreSQL in WebAssembly inside one process, on one connection, with
every transaction serialized. Separate worker, api and gateway processes cannot share it, so on PGlite there is
nothing to fail over to. `examples/postgres`, the production-shaped example, refuses `pglite:` URLs. Use PGlite for
development, tests and the simulators only; see [database](/docs/operations/database#pglite).

**What an outage does.** Nothing durable is lost, and the fleet stops changing until the database is back:

| Component | During the outage                                                                                        | After                                               |
| --------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| Gateways  | Cannot renew their leases, fence within 10 seconds, drop every stream, refuse new sessions `UNAVAILABLE` | Admit sessions again once a renewal succeeds        |
| Workers   | Every loop logs its failed pass and retries; leases expire                                               | Retake leases with new epochs                       |
| Daemons   | Back off up to 60 seconds, keep enforcing their last spec, keep results and events in the journal        | Replay on the next `Welcome`                        |
| Liveness  | A sweep gap above 20 seconds holds every timed transition back for 5 minutes                             | No node is marked unreachable for the outage itself |

The `pg` pool drops an idle connection that errors and reconnects on the next checkout, so a restarted or promoted
server needs no process restart. PLAN.md sizes this for a database failover of about 30 seconds. See
[PostgreSQL unavailable](/docs/daemon/failure-modes#postgresql-unavailable).

**Recommendations** (not enforced by anything in the code):

* Run PostgreSQL as a primary with a standby and automatic failover, behind PgBouncer in transaction mode, as PLAN.md's
  HA region does. `postgres()` keeps no session state, so it is safe behind it (see
  [pooling and PgBouncer](/docs/operations/database#pooling-and-pgbouncer)).
* Prefer synchronous replication. A failover to an asynchronous standby that lost recent commits is, for the lost
  transactions, a small point-in-time restore, with the consequences described in
  [restoring an older database](/docs/operations/backup-and-recovery#what-a-restore-cannot-bring-back).
* Keep clocks synchronized on every host. Leases use the database clock, but `ClockSkew` findings and certificates
  still compare times.

## Checklist [#checklist]

| Goal                            | What to run                                                                                                                                                                                                                                                                                                                |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No single api                   | Two or more `--roles api` processes behind an L7 balancer with `/readyz`                                                                                                                                                                                                                                                   |
| Workers fail over in about 20 s | Two or more worker processes with the same roles (`controllers,sinks,iam`, plus `scheduler` with the pool-lease placer)                                                                                                                                                                                                    |
| A gateway loss is a reconnect   | Three or more gateways per region behind an L4 balancer, each with room for a lost peer's sessions                                                                                                                                                                                                                         |
| PXE survives an agent           | Two site agents per site                                                                                                                                                                                                                                                                                                   |
| The database survives a host    | PostgreSQL HA with PgBouncer                                                                                                                                                                                                                                                                                               |
| You notice                      | `doctor`'s `workers.queue`, `workers.sweeper`, `fleet.gateway-headroom` and `fleet.heartbeats`, and `better_compute_gateway_lease_held` with the gateway loop series on [observability](/docs/operations/observability#what-to-alert-on). There is no metric for worker shard or singleton leases; read the `leases` table |

  - [Deployment](/docs/operations/deployment): Roles, balancers and the serve host.

  - [Failure modes](/docs/daemon/failure-modes): Each failure, what it costs and which invariant holds.

  - [Scaling](/docs/operations/scaling): How each tier scales and what breaks first.

  - [Backup and recovery](/docs/operations/backup-and-recovery): When the database itself is lost.


# Overview (/docs/operations)

> The processes of a Better Compute deployment, the roles each one runs, what talks to what, and what you need before the first node joins.



A Better Compute deployment is one configuration file, one PostgreSQL database and some number of Node.js
processes started from that configuration. Each process runs a subset of the same roles: the HTTP API, the daemon
gateway, and the worker roles that run reconcilers, deliver events and run Better IAM's
jobs. PostgreSQL is the only source of truth. Every process can be stopped, restarted or duplicated, and the
leases in the database keep each job single-writer.

This section covers running that system: how to lay the processes out, what to configure, what the database needs,
how the machine PKI works, and how to harden, observe and scale it.

> **What runs today.** 
  The control plane, the gateway and the CLI host are built and tested in TypeScript. The Go daemon `bcd` has a
  working v0.1 node agent, tested end to end against a real gateway, but no release: there is no published binary
  for the install script, and it has not run on a real GPU host. The `FakeDaemon` speaks the real protocol in the
  simulator, most tests and `better-compute dev --sim N`. The npm packages are at 0.2.0; see
  [installation](/docs/guides/installation).

## Process roles [#process-roles]

`better-compute serve` runs the roles you name with `--roles` (or `BETTER_COMPUTE_ROLES`). The default, `all`, runs
the API, every worker role, and the gateway when the configuration exports `host.gateway`.

| Role          | What it runs                                                                                                                                                                                                                                 | Listens on                                                                                                                                                      | Scale by                                                                                     |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `api`         | The HTTP API (`POST {basePath}/{group}/{method}`), admission transactions, `/healthz`, `/readyz`, `/.well-known/better-compute`, the host's own routes, and the tunnel upgrade `GET {basePath}/tunnels/{ticket}` when tunnels are configured | `--host`:`--port` (default `0.0.0.0:3000`), plain HTTP behind your TLS terminator                                                                               | Requests, behind an L7 load balancer                                                         |
| `gateway`     | Daemon sessions over gRPC on HTTP/2 mTLS, liveness, command dispatch, result and event ingestion, `Join`/`Renew`/`Secret` hand-off, the revocation feed, tunnels                                                                             | `host.gateway.listen` (default `0.0.0.0:8443`), plus the internal control API (default port 8444) and the WebSocket fallback (default port 443) when configured | Streams, behind an L4 load balancer with TLS passthrough                                     |
| `controllers` | Every reconciler not declared for the scheduler (node lifecycle, NodeSpec, liveness, inventory, reservations, workflows, webhooks, audit sinks, …) over 256 shards, and the CommandSweeper                                                   | nothing                                                                                                                                                         | Shards, spread over worker processes                                                         |
| `scheduler`   | Reconcilers that declare `role: 'scheduler'`: the pool-lease placer `capacity.placer`, under its own shard membership                                                                                                                        | nothing                                                                                                                                                         | Pool shards, spread over scheduler processes                                                 |
| `sinks`       | OutboxDispatch to event sinks (`sinks.events` and modules' event sinks)                                                                                                                                                                      | nothing                                                                                                                                                         | Sinks, one `singleton:outbox-dispatch:<sink id>` lease each, spread over the `sinks` workers |
| `iam`         | Better IAM's jobs (`iamJobs`), each under its own `singleton:iam:<name>` lease                                                                                                                                                               | nothing                                                                                                                                                         | One holder per job                                                                           |

`--roles` takes `all` or a comma list: `api`, `gateway`, and any of `controllers`, `scheduler`, `sinks`, `iam`.
The worker roles are the server's `workerRoles` export; `compute.start()` with no argument runs all four.

```bash
better-compute serve                                          # all: api, workers, and the gateway if configured
better-compute serve --roles api --port 3000 --no-migrate
better-compute serve --roles controllers,scheduler,sinks,iam --no-migrate
better-compute serve --roles gateway --no-migrate
```

## What talks to what [#what-talks-to-what]

<Mermaid
  chart="flowchart LR
  subgraph clients[&#x22;Callers&#x22;]
    U[&#x22;users, agents, CLI, console&#x22;]
  end
  subgraph machines[&#x22;Your machines&#x22;]
    D[&#x22;bcd (node, site, ramdisk)&#x22;]
  end
  L7[&#x22;L7 load balancer (TLS)&#x22;]
  L4[&#x22;L4 load balancer (TLS passthrough)&#x22;]
  API[&#x22;api&#x22;]
  GW[&#x22;gateway&#x22;]
  W[&#x22;workers: controllers, sinks, iam&#x22;]
  DB[(&#x22;PostgreSQL&#x22;)]
  IAM[&#x22;Better IAM (embedded)&#x22;]
  U -->|&#x22;HTTPS&#x22;| L7 --> API
  D -->|&#x22;mTLS, gRPC over HTTP/2&#x22;| L4 --> GW
  API --> DB
  GW --> DB
  W --> DB
  API --- IAM
  W --- IAM
  API -.->|&#x22;internal mTLS: pokes, tunnels, revocations&#x22;| GW"
/>

* **Callers to the API.** JSON over HTTPS with `X-Better-Compute: 1`. Better IAM authenticates every caller and
  decides every call. See [security](/docs/operations/security).
* **Daemons to gateways.** Nodes only dial out, to `gw.<region>:443` through an L4 balancer that passes TLS
  through, so the gateway sees the client certificate. See [the gateway](/docs/daemon/gateway).
* **Everything to PostgreSQL.** Processes never call each other for correctness. The API writes commands, desired
  state and outbox rows in its transactions; gateways and workers read them from the database.
* **API to gateways, as an accelerator.** After a commit the API can poke the owning gateway (`notifier`), relay a
  tunnel, or push a revocation through the gateway's internal control API, over mutual TLS from the internal CA.
  Without it, gateways still converge from their 1-second dispatch loop.

## One configuration [#one-configuration]

Every process loads the same `better-compute.config.ts`. Its default export describes the control plane
(`defineConfig`, the options of `betterCompute()`); its named `host` export describes what `serve` runs around it
(`defineHost`: extra routes, metrics, tunnels, the gateway listener). Environment variables supply what differs
between environments. See [configuration](/docs/operations/configuration).

```ts title="better-compute.config.ts"
import { defineHost } from 'better-compute/cli';
import { postgres } from 'better-compute/postgres';
import { defineConfig } from 'better-compute/server';

export default defineConfig(({ env = process.env } = {}) => ({
  database: postgres({ url: env.BETTER_COMPUTE_DATABASE_URL! }),
  secret: env.BETTER_COMPUTE_SECRET!,
  baseURL: 'https://compute.example.com',
  iam, // bridge.embedded(iam, { rootTenantId })
  modules: [/* provisionModule(), metal(), vps(), eventsModule(), … */],
}));

export const host = defineHost(async ({ compute, env }) => ({
  gateway: { region: 'eu-1', tls: { key, cert, ca: nodeCaPem } },
}));
```

## Before the first node joins [#before-the-first-node-joins]

      ### A database and a secret [#a-database-and-a-secret]

    PostgreSQL 16 or newer with `btree_gist`, and a deployment secret of at least 32 random characters, the same for
    every process. See [database](/docs/operations/database).
  
      ### The machine PKI [#the-machine-pki]

    `better-compute pki init --trust-domain compute.example.com` writes the CAs and JoinService keys, sealed under the
    secret; `pki issue-gateway` issues the gateway's serving certificate. See [PKI](/docs/operations/pki).
  
      ### Better IAM [#better-iam]

    Until the IAM bridge is wired in, no credential authenticates and only public methods answer. See
    [Better IAM](/docs/guides/iam).
  
      ### Migrate, start, check [#migrate-start-check]

    ```bash
    better-compute migrate
    better-compute serve --no-migrate
    better-compute doctor --strict
    ```

    `doctor` checks the schema, the clock, the method registry, the PKI bundle and the `host` export: whether the
    gateway role can authenticate daemons, whether the API can reach gateways, and whether host routes fit beside the
    API. Once the schema is migrated it also reports running problems: overdue work, a stalled command sweeper, usage
    backlogs, gateway headroom, silent nodes, old daemons and expiring node certificates, plus each module's findings.
    See [deployment](/docs/operations/deployment) and [the doctor checks](/docs/operations/observability#doctor).
  
## In this section [#in-this-section]

  - [Deployment](/docs/operations/deployment): Topologies, the serve host, load balancers, embedding in Next.js and the examples.

  - [Configuration](/docs/operations/configuration): Every option, the host export, environment variables and the deployment secret.

  - [Database](/docs/operations/database): PostgreSQL requirements, migrations, PGlite, the runtime tables and pooling.

  - [PKI](/docs/operations/pki): The CAs, JoinService keys, the bundle, SPIFFE IDs, revocation and rotation.

  - [Security](/docs/operations/security): The threat model and a hardening checklist.

  - [Local development](/docs/operations/local-development): `better-compute dev` , PGlite and simulated machines.

  - [Testing](/docs/operations/testing): Fixtures, the FakeDaemon and the fleet simulator.

  - [Observability](/docs/operations/observability): Metrics, logs, traces and `doctor` .

  - [Retention](/docs/operations/retention): What is pruned, when, and what is kept.

  - [Scaling](/docs/operations/scaling): Shards, leases, gateways and what breaks first.


# Local development (/docs/operations/local-development)

> Run a whole control plane on your laptop with better-compute dev, join simulated GPU machines with --sim, and run the deterministic fleet simulator.



`better-compute dev` runs the API and every worker role in one process on `127.0.0.1`, with no setup: an embedded
PostgreSQL (PGlite), a generated deployment secret and migrations on start. `--sim N` adds a gateway and N
simulated GPU machines that join the way the real daemon does, so reservations, commands, liveness and tunnels
all run end to end on your laptop. `better-compute sim` is the other tool: a seeded, deterministic fleet
simulation that checks the fleet invariants and prints a digest you can reproduce.

> **Simulated machines, real control plane.** 
  The simulated machines are `FakeDaemon`s from `@better-compute/testing`: the daemon side of the protocol in
  TypeScript. Everything above the daemon is the real code: JoinService, the gateway, the runtime, the reconcilers,
  capacity and Better IAM. The dev gateway authenticates sessions by a header over cleartext HTTP/2 and listens on
  loopback only; never expose it. The Go daemon `bcd` is built and tested but not released, and `better-compute dev`
  has no flag to join it; the dev stack in code can (see [a real bcd](#a-real-bcd)).

## Start a server [#start-a-server]

```bash
better-compute dev                 # PGlite in .better-compute/dev-db, kept between runs
better-compute dev --memory        # a throwaway in-memory database
better-compute dev --memory --sim 8
```

`dev` always migrates first, then serves the API under `/api/compute` and runs the worker roles `controllers`,
`scheduler`, `sinks` and `iam`. It stops on SIGINT or SIGTERM, releasing its leases.

| Flag                    | Default                                  | Meaning                                                               |
| ----------------------- | ---------------------------------------- | --------------------------------------------------------------------- |
| `--host HOST`           | `127.0.0.1`                              | The address to listen on                                              |
| `--port PORT`           | `3000` (env `PORT`)                      | The port; `0` picks a free one                                        |
| `--data DIR`            | `.better-compute/dev-db`                 | Where PGlite keeps the database, when there is no configuration       |
| `--memory`              | off                                      | A throwaway in-memory database (`pglite:memory`)                      |
| `--config PATH`         | discovered (env `BETTER_COMPUTE_CONFIG`) | The configuration file                                                |
| `--sim N`               | `0`                                      | Join N simulated machines, 0 to 1000; needs `@better-compute/testing` |
| `--sim-variant VARIANT` | `hgx-h100`                               | Hardware of the simulated machines, see [below](#simulated-machines)  |

**Without a configuration** (and without `BETTER_COMPUTE_DATABASE_URL`), `dev` builds its own deployment: PGlite in
`--data`, and a generated secret in `.better-compute/dev-secret` (owner-only), so repeated runs keep their data.
`BETTER_COMPUTE_BASE_URL` defaults to `http://<host>:<port>`.

**With a configuration file**, `dev` uses that deployment. `--memory` then sets `BETTER_COMPUTE_DATABASE_URL` to
`pglite:memory` for the run, which the examples and the configuration template read. The `host` export's HTTP
routes and metrics are mounted; its gateway role stays off in `dev`.

Either way the CLI adds `retentionModule()` from `@better-compute/postgres` unless the configuration registers a
`retention` module itself. See [retention](/docs/operations/retention).

**IAM.** A deployment without Better IAM configured answers only public methods. `better-compute dev --sim N`
replaces the missing IAM with an embedded one; plain `dev` does not. See [Better IAM](/docs/guides/iam).

### PGlite [#pglite]

PGlite is PostgreSQL 18 compiled to WebAssembly, with `btree_gist`. It is one connection, so the executor
serializes transactions behind a mutex. PGlite has no autovacuum, so the executor runs `ANALYZE` after `migrate()`
and then every `analyzeEveryMs` (default 60,000) to keep query plans close to Postgres's. It is the database of
`dev`, the tests and the simulators, not of production; see [database](/docs/operations/database).

## Simulated machines [#simulated-machines]

`--sim N` joins N machines to the dev server's own control plane. Nothing is mocked above the daemon:

      ### Local keys and IAM [#local-keys-and-iam]

    `dev` creates throwaway JoinService keys and a node CA (`createDevJoinKeys()`, the dev equivalent of
    `better-compute pki init`), overriding any configured `pki.join`. Without configured IAM it starts an embedded,
    in-memory Better IAM (`createEmbeddedTestIam()` from `@better-compute/iam/testing`, on better-iam's SQLite adapter)
    with compute's plugin and resolver, and prints its root administrator's token. With configured IAM,
    `BETTER_COMPUTE_TOKEN` must hold a root administrator's token (`MISSING_ENV` otherwise).
  
      ### A loopback gateway [#a-loopback-gateway]

    A real gateway starts on cleartext HTTP/2 at `127.0.0.1` on a free port, wired to the control plane's gateway
    services, work queue, outbox and report handlers. Sessions authenticate by the `x-bc-test-node` header, and only
    for node ids that joined through this fleet and whose certificate is not revoked. Pokes after each commit reach the
    gateway in process (`devFleetNotifier()`).
  
      ### A site, a pool and a join token [#a-site-a-pool-and-a-join-token]

    As the root administrator, `dev` creates a shared site in region `dev-1` and a `metal-host` pool with `diag.nvml`
    acceptance (and an expected GPU count for the H100 and L40S variants), then mints one join token with `maxUses: N`.
  
      ### Join, session, acceptance, adoption [#join-session-acceptance-adoption]

    Each machine seals the token to JoinService's key and binds it to its CSR, calls the `Join` RPC through the gateway,
    verifies the signed JoinResponse against the pinned keys, then opens its `Session` stream. The runtime runs
    `diag.nvml`, adopts the node and pushes its NodeSpec. `dev` reports adoption once every node has applied a NodeSpec
    of generation 1 or later.
  
<Mermaid
  chart="sequenceDiagram
  participant F as FakeDaemon
  participant G as dev gateway (h2c)
  participant J as JoinService
  participant R as runtime
  F->>G: Join(sealed token, CSR, claims)
  G->>J: forward
  J-->>F: signed JoinResponse, node certificate
  F->>G: Session: Hello
  G-->>F: Welcome
  R->>G: command diag.nvml (poke)
  G-->>F: Command
  F-->>G: CommandUpdate SUCCEEDED
  R->>R: adopt, bump NodeSpec generation
  G-->>F: DesiredState"
/>

| `--sim-variant`        | Machines                          | Pool                                                           |
| ---------------------- | --------------------------------- | -------------------------------------------------------------- |
| `hgx-h100`             | 8x H100 SXM5 80GB                 | GPU model `H100`, expects 8 GPUs                               |
| `hgx-h100-gpu-missing` | the same board with a GPU missing | expects 8 GPUs, so acceptance fails and they are never adopted |
| `l40s`                 | 4x L40S                           | GPU model `L40S`, expects 4 GPUs                               |
| `cpu`                  | no GPUs                           | no GPU model, no GPU count check                               |

Standard error shows the progress (from `examples/dev`):

```text
better-compute listening on http://127.0.0.1:3000/api/compute
Dev gateway (h2c, header auth, loopback only): http://127.0.0.1:52011
Embedded IAM root tenant ten_…; root token (dev only): biam_ses_…
Joined 8 simulated hgx-h100 node(s) into pool_…
8 simulated node(s) adopted
```

The embedded IAM lives in memory for this run only, and its root token is a development credential.

### Look around [#look-around]

In a second terminal, as the root administrator (see the [CLI](/docs/guides/clients/cli)):

```bash
export BETTER_COMPUTE_URL=http://127.0.0.1:3000
export BETTER_COMPUTE_TOKEN=biam_ses_…   # the root token dev printed
export BETTER_COMPUTE_TENANT=ten_…       # the root tenant dev printed
better-compute nodes list
export POOL=pool_…                        # the pool dev created (pools list)
better-compute pools offer $POOL --to $BETTER_COMPUTE_TENANT --quota gpu=64
better-compute reserve --pool $POOL --gpus 4 --for 1h
source <(better-compute completion bash) # optional: tab completion for every command
```

The reservation binds through the node's desired state and turns `active` once the simulated daemon reports
`AllocationBound`; its CDI device names are in the reservation's `binding`. With a configuration that mounts the
events module's `GET /watch` (the examples do), `better-compute events watch --types "node.*,reservation.*"` prints
the tenant's events one JSON line each (`curl -N` on the same route shows the raw stream); cordon a node in another
terminal (`better-compute nodes cordon node_… --reason demo`) to see `node.cordoned` arrive. This is the walkthrough
of `examples/dev/README.md` (786cc80).

### Tunnels [#tunnels]

With `--sim`, `dev` issues exec tickets (the configured `tunnels.issuer`, or a throwaway one) and mounts
`GET /api/compute/tunnels/{ticket}`. The gateway verifies tickets and serves its control API on loopback, and every
simulated machine answers `OpenTunnel` with an echo shell, so [tunnels](/docs/guides/access/tunnels) work end to end:
`better-compute console --node node_…` opens a session on a simulated machine, where `size` prints your terminal
size, `exit` ends it with status 0, `false` with 1, and Ctrl-] detaches (`tests/cli-console.test.ts`).
Unless the deployment's `tunnels` options bring their own, it also adds a throwaway break-glass SSH CA (`tunnels.ssh`,
for `nodes.issueSshCert` and `better-compute ssh-cert`) and a separate throwaway sandbox SSH CA (`tunnels.sandboxSsh`),
so both can be tried against the simulated fleet. It adds these throwaway CAs only on a PGlite store: against a real
database every node's spec would start trusting a CA that dies with the process, so there `dev --sim` uses only the
CAs the deployment configures.
With the embedded IAM, an open tunnel is not re-authorized every 60 seconds: the embedded test IAM has no session
reviewer. `tunnels.open` still decides every ticket.

## The dev stack in code [#the-dev-stack-in-code]

The same pieces are exported from `@better-compute/testing` for your own hosts and tests.

| Export                                | What it does                                                                                                                                                              |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `createDevJoinKeys({ trustDomain? })` | A node CA and JoinService keys (`pki`, for `betterCompute({ pki })`), the `pins` a daemon checks and what the agent half needs. Default trust domain `compute.dev.local`. |
| `devFleetNotifier()`                  | A `Notifier` for `betterCompute({ notifier })` that turns `gw:<id>` publishes into `gateway.poke(nodeIds)` in process.                                                    |
| `attachDevFleet(options)`             | Adds a loopback h2c gateway and simulated machines to a running control plane. Demands `insecureHeaderAuth: true`.                                                        |
| `createDevStack(options?)`            | All of it on PGlite in one call: control plane, workers, gateway, tunnels, `retentionModule()`.                                                                           |
| `DEV_ROOT_TOKEN`                      | `bc_dev_root`: the token `createDevStack()`'s default fake IAM accepts as a root administrator (it allows everything).                                                    |

```ts title="dev-stack.test.ts"
import { afterEach, expect, it } from 'vitest';
import { createDevStack, type DevStack } from 'better-compute/testing';

let dev: DevStack | undefined;
afterEach(() => dev?.close());

it('adopts three simulated machines', async () => {
  dev = await createDevStack();
  const { poolId } = await dev.createPool({ expectGpuCount: 8 });
  const nodes = await dev.join({ poolId, count: 3 }); // sealed-token Join, then a Session each
  await dev.waitFor(nodes); // adopted, NodeSpec generation 1 or later applied
  expect(dev.daemons.size).toBe(3);
});
```

A `DevFleet` has `createPool({ gpuModel?, key?, expectGpuCount? })`, `join({ poolId, count?, variant?, seed? })`
(1 to 10,000 machines on one token), `joinSite({ siteId, count?, kinds?, seed? })` for site agents (1 to 16, through
`siteTokens.create`), `waitFor(nodeIds, { provisionState?, timeoutMs? })` (default `adopted`, 30 s), the running
`daemons`, the `gatewayUrl`, and `close()`. `createDevStack()` takes `database`, `iam`, `modules`, `baseURL`,
`secret`, `trustDomain`, `worker` (default `pollMs` 100), `region`, `gateway` and `daemon` options, and
`signedIntent: true` (5071dbc), which gives the stack a fresh command-signing root (`pki.commandSigning`), so every
command and spec is signed and joining daemons receive the roots in `JoinResponse.trust`.

### A real bcd [#a-real-bcd]

Two options let a daemon outside the process, such as a real `bcd`, join the dev stack:

| Option                                                 | What it adds                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `externalTls: { host?, port?, dnsNames?, servingCa? }` | A second gateway listener authenticated as in production: HTTP/2 over mTLS, a serving certificate from a dev serving CA (`servingCaCertificate`, what the daemon pins; names `localhost` and `gw.dev.local` by default), client certificates checked against the dev node CA. Its URL is `externalGatewayUrl`. Its port (a free one unless given) and serving CA are chosen before the control plane starts and published in `/.well-known/better-compute` as `gateway` (`https://<host or 127.0.0.1>:<port>`, `serverName` the first DNS name, and the CA's pin). Safe to bind beyond loopback |
| `admitExternalJoins: true`                             | On the loopback h2c listener, also accepts the `x-bc-test-node` header of nodes that joined on their own, once their current certificate is in `node_certs` (polled every 250 ms, or `admit(nodeId)` at once). bcd reaches it only as a fakehw build with `dev_insecure_h2c = true`                                                                                                                                                                                                                                                                                                             |

For `externalTls`, write a `config.toml` with `gateway_url` set to the listener, `serving_ca_spki_pin` from
`servingCaCertificate`, the join pins from `pins`, and `gateway_server_name = "gw.dev.local"` when you dial an IP
address; put `pins.joinEncryptionJwk` next to it as `join-encryption.jwk`. Then pipe a join token into `bcd join`
and start `bcd node`. `tests/e2e/bcd.test.ts` does exactly this from WSL; see
[testing](/docs/operations/testing#the-go-daemon). Since the stack publishes its gateway, a machine without any
`config.toml` can also join with `bcd join --server <the stack's base URL>`, which takes the gateway from the
well-known document: a host build also needs `--serving-ca-pin`, `--join-signing-pin` and `--join-encryption-pin`
(from `servingCaCertificate` and `pins`), while a fakehw build may join with `--server` alone, taking the pins from
the document on trust (and accepts an `http://` server); see
[joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document).

For many real daemons at once, a fakehw build runs `bcd sim --config <that config.toml> --nodes N` with one join token
allowing N uses on standard input: N node agents in one process, each a distinct fakehw machine with its own
identity under `data_dir/sim/<i>` (see [bcd sim](/docs/daemon#bcd-sim)).

Two fakehw behaviours help when you test placement against a real bcd:

* **Reserved huge pages show in the inventory** (cd7a311). The host units write under the fake root
  (`<data_dir>/fakehw-root`), and the fakehw inventory reads `nr_hugepages` back from there: the machine-wide pools,
  and each NUMA node's own file or an even share of the machine-wide pool (node 0 takes the remainder). A pool with
  `policy.hugepages` therefore books `hugepages2Mi` and `hugepages1Gi` on a fakehw node as it would on a real host.
* **The libvirt plan is written out** (9a6aaf4). The fakehw runtime still runs every domain itself, but also writes
  the libvirt domain XML bcd would define on a real host to `<data_dir>/fakehw-plans/<id>.xml` (`vcpupin`,
  `numatune`, huge-page backing, the memory balloon, volumes by serial, GPU hostdevs), or `<id>.error` when libvirt
  would refuse the spec, and removes it on destroy. Its balloons live on disk (`<runtime>.balloons`), so they survive
  a restart. `tests/e2e/bcd-placement.test.ts` reads both (see [testing](/docs/operations/testing#the-go-daemon)).

## The example [#the-example]

`examples/dev` is the whole deployment every example shares (`examples/shared/compose.ts`: provision and metal,
vps, events, fleet, retention, and rollout and audit when their keys are present) with eight simulated machines:

```bash
cd examples/dev
better-compute dev --memory --sim 8     # the package's "dev" script
better-compute dev --sim 8              # "dev:persistent": data in .better-compute/example-db
```

## The console locally [#the-console-locally]

`apps/console` is the Next.js console (Better IAM and Better Compute on one origin). `node server.mjs --dev` (its
`dev` script) runs Next.js plus the WebSocket upgrade for tunnels on `http://localhost:3100`. `--dev` is the only mode
in which the console generates its own secrets under `.console-data`: the session secret, the exec ticket and SSH CA
keys, and on first start the root administrator's password, printed once. Its compute store defaults to PGlite in
`.console-data/compute` and its IAM store to a libSQL file. The console does not start simulated machines. See
[the console](/docs/guides/clients/console).

## The fleet simulator [#the-fleet-simulator]

`better-compute sim` runs `--nodes` FakeDaemons on fake hardware against the in-memory gateway stub, entirely in
simulated time. A steady workload runs for `--duration` (every 2 simulated seconds: three commands drawn from
`diag.nvml`, `inventory.refresh`, `logs.collect` and `node.reboot`, a GPU booking and sometimes a release), with the
faults of `--chaos`. The simulator then quiesces for `--settle`, flushes usage and checks the invariants. Nothing reads
the wall clock, so the same seed prints the same digest.

```bash
better-compute sim
better-compute sim --seed nightly-42 --nodes 500 --duration 1h --chaos heavy --log sim.log
```

| Flag                  | Default | Meaning                                                  |
| --------------------- | ------- | -------------------------------------------------------- |
| `--seed SEED`         | `sim`   | Everything random derives from it                        |
| `--nodes N`           | `50`    | Simulated nodes, 1 to 5000                               |
| `--duration DURATION` | `10m`   | Simulated time under workload (`30s`, `10m`, `2h`)       |
| `--settle DURATION`   | `16m`   | Quiet time afterwards, so the sweeper and replays settle |
| `--chaos LEVEL`       | `light` | `none`, `light` or `heavy`                               |
| `--log PATH`          | none    | Write the event log to a new file (fails if it exists)   |

| Chaos   | Links (both directions)                                                               | Faults, drawn every                                                                                                               |
| ------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `none`  | perfect                                                                               | none                                                                                                                              |
| `light` | 5 ms latency, 20 ms jitter, 0.0001 drop (reset), 0.002 duplicate, 0.0005 reorder down | 15 s: crash 0.1, reboot 0.05, partition 0.1, reset 0.1, stall 0.05                                                                |
| `heavy` | 5 ms latency, 20 ms jitter, 0.0005 drop (reset), 0.01 duplicate, 0.002 reorder down   | 5 s: crash 0.3, reboot 0.1, journal loss 0.05, partition 0.3, isolate 0.1, half-open 0.1, reset 0.2, stall 0.2, gateway kill 0.02 |

It checks I1 (no overlapping bookings or node-held devices), I2 (at most one execution per command; after a journal
loss, in-flight commands that are not retry-safe fail instead of replaying), I3 (a delivered spec is reported
observed within 60 s while the node stays reachable), I4 (no node marked unreachable within 20 s of a gateway hearing
from it), I5 (no release caused by connectivity) and I8 (usage windows add up to booked time times atoms, with no
duplicate keys). The command's own help text still lists only I1, I2, I5 and I8; the checks come from the simulator's
`invariants()`. See [testing](/docs/operations/testing#invariants).

The report is JSON: `seed`, `nodes`, `chaos`, `simulatedMs`, `wallMs`, `digest` (SHA-256 of the event log),
`commands` (`total`, `open`), `faults` (`crashes`, `reboots`, `journalLosses`, `gatewayKills`), `violations` and the
first 20 violations. On a violation the command prints the report and exits with `SIM_INVARIANT_VIOLATED`; re-run
with the same `--seed` to reproduce it, and add `--log` to get the event log. `sim` needs `@better-compute/testing`.

> **The stub is not the server.** 
  `better-compute sim` exercises the daemon side and the protocol against `GatewayStub`, a minimal in-memory control
  plane. The runs against the real control plane and real gateways are `createControlPlaneSim` and the nightly
  matrix; see [testing](/docs/operations/testing#the-control-plane-simulator). `bcd sim` is unrelated despite the
  name: it runs real Go daemons in real time against a real gateway, with no chaos or invariant checks (see
  [bcd sim](/docs/daemon#bcd-sim)).

## Working on the repository [#working-on-the-repository]

The repository is a pnpm workspace; `docs/development.md` is the full guide.

* **Node.js 22.12 or later**, ESM only. CI runs Node 22 and 24 on Ubuntu and Windows.
* **pnpm 11.19.0**, pinned in `packageManager`. When your global pnpm is another major, run the pinned copy:
  `node node_modules/pnpm/bin/pnpm.mjs <args>`.
* **No database server.** PGlite backs the tests. Suites that need real Postgres skip unless
  `BETTER_COMPUTE_POSTGRES_URL` is set.
* **Go 1.26 or later** only for `daemon/`: `node scripts/daemon-test.mjs` runs `go test -tags nonvml,fakehw`
  without cgo or GPUs, natively on Linux and through WSL on Windows. The bcd end-to-end suite is opt-in on Windows
  (`BCD_E2E=1`); see [testing](/docs/operations/testing#the-go-daemon).

| `pnpm …`          | What it does                                                                                                                                                                                 |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `check`           | `typecheck`, `test`, `test:iam-remote`, `lint:repo` and `validate:packages`: what CI runs first                                                                                              |
| `test`            | `vitest run` over `tests/`, against the packages' sources (no build needed)                                                                                                                  |
| `test:iam-remote` | the IAM-dependent suites again with IAM over HTTP (`vitest.iam-remote.config.ts`)                                                                                                            |
| `typecheck`       | `tsc -b`, plus the client type tests                                                                                                                                                         |
| `lint:repo`       | the dependency rules, the one-clock rule (`compute_now()` only), the umbrella drift check, `proto-gen --check` and `--breaking`, and the v1.0 API surface freeze (`scripts/api-surface.mjs`) |
| `api:surface`     | records compatible API additions in `generated/api-surface.json`; see [the API surface freeze](/docs/operations/testing#the-api-surface-freeze)                                              |
| `test:postgres`   | the Postgres-only suites (needs `BETTER_COMPUTE_POSTGRES_URL`)                                                                                                                               |
| `bench:scale`     | the scale benchmarks in `tests/bench/` (`vitest.bench.config.ts`)                                                                                                                            |
| `sim:nightly`     | the nightly simulation matrix in `tests/nightly/`                                                                                                                                            |
| `build`           | `tsc -b`: every package's `dist/`                                                                                                                                                            |

Tests live in the root `tests/` folder as `tests/<area>-<topic>.test.ts`; `vitest.config.ts` maps every
`@better-compute/*` specifier and `better-compute/*` subpath to `packages/*/src`. See [testing](/docs/operations/testing).

  - [Testing](/docs/operations/testing): Fixtures, the FakeDaemon, the simulators and the benchmarks.

  - [Database](/docs/operations/database): PostgreSQL, PGlite and migrations.

  - [Scaling](/docs/operations/scaling): What the benchmarks measured so far.

  - [CLI reference](/docs/reference/cli): Every command and flag.


# Observability (/docs/operations/observability)

> Prometheus metrics with bounded labels, a bearer-protected /metrics route, ComputeSpan observers, trace context, health endpoints and doctor.



`@better-compute/observability` holds the pieces a host uses to watch a deployment: a Prometheus registry that
refuses tenant labels and caps its series, the standard `better_compute_*` metric set of PLAN.md section 14, a
`GET /metrics` handler behind a bearer token, wrappers that meter IAM calls, gateway services, attestors and
providers, `ComputeSpan` observers and W3C `traceparent` helpers. It is browser-safe (no Node.js imports), so every
package can depend on it. The CLI host mounts it for you when the configuration's `host` export has a registry.

> **Defined is not the same as recorded.** 
  `standardMetrics(registry)` fixes the names, labels and buckets of every section 14 series, so all packages record
  the same thing. The built-in pieces record: the gateway series sampled by `gatewayCollector`, the gateway service
  calls, outbox events by family, the deployment-wide series `databaseCollector` samples from the database (below),
  placements through `placementObserver`, and whatever you wrap (`meteredIamPort`, `meteredAttestor`,
  `meteredProvider`, spans). `gateway_sessions`, `gateway_handshakes_total`, `commands_swept_total`,
  `reconcile_duration_seconds`, `usage_dead_letters_total` and `security_events_total` are defined but no package sets
  them yet; they expose their `HELP` and `TYPE` lines with no samples until your host records them.

## Wire metrics into a deployment [#wire-metrics-into-a-deployment]

One registry per process. The configuration creates it, registers the standard series, passes it to what it meters,
and hands it to the host through `host.metrics`:

```ts title="better-compute.config.ts"
import { defineHost } from 'better-compute/cli';
import { createMetrics, eventCounter, meteredIamPort, standardMetrics } from 'better-compute/observability';
import { eventsModule } from 'better-compute/events';
import { defineConfig } from 'better-compute/server';

export const metrics = createMetrics(); // maxSeries 2000 per metric
const standard = standardMetrics(metrics);

export default defineConfig(({ env = process.env } = {}) => ({
  // …database, secret, baseURL
  iam: meteredIamPort(bridge.embedded(iam, { rootTenantId }), standard), // iam_calls_total, iam_call_duration_seconds
  modules: [eventsModule({ secret: env.BETTER_COMPUTE_SECRET!, countEvent: eventCounter(metrics) })],
}));

export const host = defineHost(async ({ env }) => ({
  metrics: { registry: metrics, bearerToken: env.BETTER_COMPUTE_METRICS_TOKEN }, // GET /metrics
}));
```

`better-compute serve` then:

* mounts `GET /metrics` (or `host.metrics.path`) on the API listener when `bearerToken` is set; without a token there
  is no route;
* when it runs the gateway role, meters `compute.gatewayServices` with `meteredGatewayServices` and samples the
  gateway at every scrape with `gatewayCollector`;
* when it runs any worker role, starts `databaseCollector` into the same registry: a first reading at start, then
  every 15 s, stopped with the server. It takes each command kind's lane from the registry and watches the host's
  own certificates as `cert_expiry_seconds{kind}`: `node-ca`, `command-signing` and `serving` (the gateway's TLS
  certificate).

`better-compute dev` mounts the configuration's `host.metrics` too. The examples read the token from
`BETTER_COMPUTE_METRICS_TOKEN`.

Scrape it with the token as a bearer credential:

```yaml title="prometheus.yml"
scrape_configs:
  - job_name: better-compute
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/better-compute.token
    static_configs:
      - targets: ['compute-api-1:3000', 'compute-api-2:3000']
```

## The registry [#the-registry]

`createMetrics({ prefix?, maxSeries?, forbiddenLabels?, now? })` returns a registry of counters, gauges and
histograms.

* **Prefix.** Every name gets `better_compute_` unless it already has it. Names and labels must match
  `[a-z_][a-z0-9_]*`; `le` is reserved.
* **No tenant labels.** A label named `tenant`, `tenant_id`, `tenantid`, `org`, `organization`, `identity` or `user`
  is refused with `INVALID_CONFIG` when the metric is defined. Per-tenant numbers belong in usage and audit, not in
  your TSDB.
* **Bounded series.** Each metric holds at most `maxSeries` label sets (default 2000). New label sets past the cap
  fold into one series whose labels all read `(other)`, and `better_compute_metrics_collapsed_total` counts every
  folded recording.
* **Caller-controlled values.** A label value that is not a string or is longer than 256 characters reads
  `(invalid)`; an empty one reads `(unknown)`.
* **Collectors.** `registry.collect(fn)` runs `fn` at the start of every `expose()`, for gauges sampled at scrape
  time. A collector that throws is skipped for that scrape.
* **Exposition.** `expose()` renders text format 0.0.4. `value(name, labels)` and `stats()` (`series`, `dropped`)
  serve tests and health checks.

`metricsHandler({ registry, bearerToken, timingSafeEqual })` answers `GET` only (405 otherwise), requires a token of
at least 16 characters, compares the SHA-256 digests of the presented and expected tokens with the constant-time
function you pass (`crypto.timingSafeEqual` in Node), answers 401 with `WWW-Authenticate: Bearer` on a mismatch,
and serves the text with `Cache-Control: no-store`.

## The standard series [#the-standard-series]

`standardMetrics(registry)` defines the section 14 series: sessions and handshakes, command latency and sweeps, the
work queue and reconcile durations, placements and booking conflicts, pool GPUs, node connectivity, backlogs, IAM
calls and resolver lookups, certificate expiry, clock offset, and security events. None carries a tenant label.
The [metrics reference](/docs/reference/metrics) lists every family with its type, labels, help text and buckets,
generated from `metricCatalog()`, so it always matches the code. This page covers who records them.

Two IAM series have hooks in the IAM bridge you can point at them: `bridge.plugin({ onSpill })` is called for every
event the plugin spills, and `bridge.resolveResource(fallback, { onLookup })` reports each lookup as `hit`, `miss`,
`timeout`, `unavailable`, `scope` or `fallback`. Hook failures never change a decision.

```ts
const plugin = bridge.plugin({ onSpill: () => standard.iamInboxSpill.inc() });
const resolveResource = bridge.resolveResource(undefined, {
  onLookup: (result) => standard.resolverLookups.inc({ result }),
});
```

### The database collector [#the-database-collector]

Some section 14 series describe the whole deployment, not one process. `databaseCollector` samples them from
compute's own tables with read-only, bounded statements:

```ts
import { databaseCollector, standardMetrics } from 'better-compute/observability';

const collector = databaseCollector(standardMetrics(registry), {
  sql: compute.context.sql,
  laneOf: (kind) => compute.context.registry.commandKind(kind)?.lane, // default label (unknown)
  certificates: () => [{ kind: 'serving', notAfter }], // host certificates besides node certificates
});
collector.start(); // every intervalMs (default 15 s); refresh() polls once; stop() ends it
```

| Series                               | Sampled from                                                                                                                                                                                                                                    |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `work_queue_depth{kind}`             | Due, unclaimed `work_queue` rows by kind                                                                                                                                                                                                        |
| `backlog{queue}`                     | Pending `commands`, unprocessed `iam_inbox` rows, unrecorded `usage_windows`, pending `webhook_deliveries`, events a sink gave up on (`outbox_dead_letters`), and each `outbox_cursors` reader's lag as `outbox:<name>`; each capped at 100,000 |
| `node_connectivity{state}`           | Nodes by connectivity                                                                                                                                                                                                                           |
| `generation_lag_nodes`               | Nodes whose observed generation is behind                                                                                                                                                                                                       |
| `pool_gpus{state}`                   | `pool_capacity_rollup` GPUs: total, allocatable, allocated, reserved, provisioning, unhealthy, maintenance, unreachable                                                                                                                         |
| `cert_expiry_seconds{kind}`          | Each node's newest live certificate, soonest per role as `node:<role>`, plus the `certificates` you pass                                                                                                                                        |
| `clock_offset_seconds`               | This process's clock minus `compute_now()`                                                                                                                                                                                                      |
| `command_latency_seconds{kind,lane}` | Commands settled since the last poll, `updated_at - created_at` (needs `commands.created_at`, `core/0020`)                                                                                                                                      |

A table an optional module did not create is skipped, and a label set that disappears reads 0 instead of going
stale. Latency starts at the first poll, so a restart does not observe history twice, and at most 5,000 settled
commands are observed per poll. Poll failures go to `onError` and never throw.

Any number of processes may run the collector, and `serve` starts it in every process with a worker role. Each one
reports the same deployment-wide gauges, so query them with `max` rather than `sum` across instances. The latency
histogram counts each command once. Only the holder of the `leases` row `metrics:command-latency` observes
`command_latency_seconds`: the row is taken or renewed on each poll with a TTL of three polls, and released on
`stop()`. A stopped collector never takes it again. Pass `latencyLease` to name another row, or `false` to observe in
every process.

### Series the instrumentation adds [#series-the-instrumentation-adds]

Beyond the standard set, these come from the collectors and wrappers that record them:

| Series                                                                                                                    | From                                                      | Labels                                                                            |
| ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `better_compute_gateway_open_sessions`                                                                                    | `gatewayCollector`                                        | `gateway`, `role`                                                                 |
| `better_compute_gateway_stream_headroom`, `_draining`, `_lease_held`, `_commands_in_flight`, `_sessions_upgrade_required` | `gatewayCollector`                                        | `gateway`                                                                         |
| `better_compute_gateway_loop_consecutive_failures`, `_loop_failures`, `_loop_last_ok_age_seconds`                         | `gatewayCollector`                                        | `gateway`, `loop` (dispatch, lease, liveness, specs, status)                      |
| `better_compute_gateway_service_calls_total`, `_service_seconds`                                                          | `meteredGatewayServices`                                  | `method` (join, renew, secret), `outcome`                                         |
| `better_compute_attestor_verifications_total`, `_verification_seconds`                                                    | `meteredAttestor`                                         | `attestor`, `outcome` (`admitted`, `deferred-<kind>`, `refused` or an error code) |
| `better_compute_provider_calls_total`, `provider_call_seconds`                                                            | `meteredProvider`                                         | `provider`, `method`, `outcome`                                                   |
| `better_compute_revocation_feed_consecutive_failures`                                                                     | `revocationFeedCollector`                                 | `feed`                                                                            |
| `better_compute_events_total`                                                                                             | `eventCounter` via `eventsModule({ countEvent })`         | `family`, `module`                                                                |
| `better_compute_placements_total`, `booking_conflicts_total`, `placement_seconds`                                         | `placementObserver` via `capacityModule({ onPlacement })` | `outcome`; none; `placer` (`inline`, `pool-lease`, `batch`)                       |
| `better_compute_spans_total`, `span_duration_seconds`                                                                     | `createObserver({ metrics })`                             | `kind`, `name`, `outcome`, `code`; `kind`                                         |

`gatewayCollector` and `meteredGatewayServices` are wired by `serve` for the gateway role; the others are yours to
wrap around what you register. Placements, for example, are observed by passing capacity's hook the standard
series (`booking_conflicts_total` counts replans forced by a booking conflict):

```ts
capacityModule({ onPlacement: placementObserver(standardMetrics(registry), registry) });
```

Outcome labels are bounded: ids, method names, error codes and catalog families. Help text and buckets for each of
these are in the [metrics reference](/docs/reference/metrics).

## Spans and traces [#spans-and-traces]

`createObserver({ onSpan?, metrics?, now? })` times units of work as a `ComputeSpan`, shaped like better-iam's
`IamSpan`:

```ts
interface ComputeSpan {
  kind: 'operation' | 'decision' | 'http' | 'command' | 'reconcile' | 'placement' | 'iam' | 'sink';
  name: string; // method, route, command kind or reconciler
  tenantId?: string; // for hooks only, never a label
  outcome: 'ok' | 'denied' | 'error';
  code?: string;
  status?: number;
  requestId?: string;
  durationMs: number;
}
```

`observer.span(kind, name, fn, extra?)` runs `fn`, classifies the outcome (401, 403 and 429, or the codes
`UNAUTHENTICATED`, `ACCESS_DENIED`, `CONFIRMATION_REQUIRED`, `RATE_LIMITED`, `CSRF_REJECTED` and
`UNTRUSTED_ORIGIN`, are `denied`; anything else that throws is `error`), fans the span out to `onSpan` hooks and feeds
the span metrics. Hooks and metrics can never affect the work: their failures are swallowed. A name the caller
controls (an unknown route, `NOT_FOUND`, `INVALID_INPUT`, `METHOD_NOT_ALLOWED`) is recorded as `(invalid)` or
`(unknown)`, so it cannot grow the series. The server does not create an observer itself; hosts use it to export
spans to their tracing system.

**Trace context.** `parseTraceparent`, `formatTraceparent` and `childTraceparent` handle W3C Trace Context version
`00` (`00-<32 hex trace id>-<16 hex parent id>-<2 hex flags>`), refusing all-zero ids and version `ff`. The protocol
carries `traceparent` on every `CommandBody` and `DesiredStateBody`, and the driver SDK passes it to drivers, so a
daemon's work can join the request's trace. A valid `traceparent` header on an API request (version `00`, non-zero
trace and parent ids) reaches the method as `MethodContext.traceparent`, and `ctx.commands.enqueue` in that method
defaults `CommandBody.traceparent` to it, so the commands a call enqueues carry the caller's trace. An invalid header
is dropped, not refused. Commands enqueued by reconcilers, and NodeSpecs, go out with it empty.

**Request ids.** The API takes the caller's `X-Request-Id` when it is 1 to 128 characters of letters, digits and
`._:-`, else it mints a `req_` id, and returns it in the response's `X-Request-Id`. IAM receives the same id in
the forwarded headers, and the admission context of a mutation's audit row records it, so a support ticket with the
header finds the call.

**Logs.** `betterCompute({ log })` takes a `Logger`; the default writes warnings and errors as JSON to standard error.
GPU telemetry does not go through the control plane (principle P8); the nodes send it straight to your collector.

## Node metrics from bcd [#node-metrics-from-bcd]

bcd exports its own metrics over OTLP/HTTP, in the JSON encoding, to the collector its NodeSpec's `telemetry`
section names (27c7ee2). Nothing reaches the gateway, the session or PostgreSQL.

**Where to.** `nodeTelemetryModule(options)` (53aa356) gives every node that section: `options` are the platform's
collector, and a pool's `policy.telemetry` wins field by field, with `resourceAttributes` merged by key. A node whose
result has no endpoint gets no section, and exports nothing. The module is not registered by default; add it to your
modules:

```ts title="compute.ts"
import { nodeTelemetryModule } from 'better-compute/observability';

betterCompute({
  // …
  modules: [
    nodeTelemetryModule({
      otlpEndpoint: 'https://otel.example.internal:4318',
      intervalSeconds: 30,
      headersRef: 'secret://root/sec_01k…', // the collector's auth headers
      resourceAttributes: { 'deployment.environment': 'prod' },
    }),
  ],
});
```

| Setting              | Rules                                                                                                                                           |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `otlpEndpoint`       | `https://`, or `http://` to `localhost`, `127.0.0.1` or `[::1]`; no credentials. bcd appends `/v1/metrics` unless the path already ends with it |
| `intervalSeconds`    | 5 to 3600; bcd's default is 60 seconds and its minimum 5                                                                                        |
| `headersRef`         | A secret reference to the headers: a JSON object, or `Name: value` lines                                                                        |
| `resourceAttributes` | At most 32, single-line values of at most 256 characters; keys under `bcd.` and `service.` are bcd's own                                        |
| `roles`              | Module option only: the node roles that export (default all, site agents included)                                                              |

The module checks its options when it is built (`INVALID_INPUT`). A pool whose `policy.telemetry` fails the same
checks is skipped, and its nodes keep the default; `better-compute doctor` (and the support bundle) reports such
pools as `pool-telemetry-invalid`, from `nodeTelemetryFindings(sql)`.

**What.** Every interval, bcd sends:

| Metric                                                                      | Unit                          | From                                                                                                                                                                                                                                                                                                                                                                                                             |
| --------------------------------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bcd.gpu.memory.used`, `bcd.gpu.memory.total`, `bcd.gpu.memory.reserved`    | bytes                         | `nvidia-smi`, per GPU                                                                                                                                                                                                                                                                                                                                                                                            |
| `bcd.gpu.utilization`, `bcd.gpu.memory.utilization`                         | ratio (0 to 1)                | `nvidia-smi`                                                                                                                                                                                                                                                                                                                                                                                                     |
| `bcd.gpu.temperature`                                                       | Celsius                       | `nvidia-smi`                                                                                                                                                                                                                                                                                                                                                                                                     |
| `bcd.gpu.power.draw`, `bcd.gpu.power.limit`                                 | watts                         | `nvidia-smi`                                                                                                                                                                                                                                                                                                                                                                                                     |
| `bcd.gpu.clock.sm`, `bcd.gpu.clock.memory`                                  | MHz                           | `nvidia-smi`                                                                                                                                                                                                                                                                                                                                                                                                     |
| `bcd.gpu.processes`                                                         | processes                     | `nvidia-smi`                                                                                                                                                                                                                                                                                                                                                                                                     |
| `bcd.gpu.mig.memory.used`, `bcd.gpu.mig.memory.total`                       | bytes                         | `nvidia-smi`, per MIG instance (`mig.id`)                                                                                                                                                                                                                                                                                                                                                                        |
| `bcd.gpu.ecc.errors`                                                        | errors, cumulative sum        | the inventory's aggregate ECC counters, `type` `correctable` or `uncorrectable`                                                                                                                                                                                                                                                                                                                                  |
| `bcd.gpu.rows.remapped`                                                     | rows                          | the inventory's row remapping, by `type`                                                                                                                                                                                                                                                                                                                                                                         |
| `bcd.gpu.xid.active`                                                        | conditions                    | bcd's active XID conditions per GPU                                                                                                                                                                                                                                                                                                                                                                              |
| `bcd.host.pressure.cpu`, `bcd.host.pressure.memory`, `bcd.host.pressure.io` | percent                       | PSI `some avg10`                                                                                                                                                                                                                                                                                                                                                                                                 |
| `bcd.host.cpu.steal`                                                        | percent                       | CPU steal                                                                                                                                                                                                                                                                                                                                                                                                        |
| `bcd.host.memory.total`, `bcd.host.memory.available`                        | bytes (`By`)                  | `/proc/meminfo`'s `MemTotal` and `MemAvailable` (3111284)                                                                                                                                                                                                                                                                                                                                                        |
| `bcd.host.hugepages.total`, `bcd.host.hugepages.free`                       | pages (`{page}`)              | `/sys/kernel/mm/hugepages`, each pool's `nr_hugepages` and `free_hugepages`, by `page.size` (the page size in bytes)                                                                                                                                                                                                                                                                                             |
| `bcd.instances`                                                             | instances (`{instance}`)      | every instance runtime's list of domains, managed or not, counted by `runtime` and `state` (`running`, `stopped`, `paused`, `error`)                                                                                                                                                                                                                                                                             |
| `bcd.instance.vcpu`                                                         | vCPUs (`{cpu}`)               | each domain's configured vCPUs, by `instance.id`, `runtime` and `managed`                                                                                                                                                                                                                                                                                                                                        |
| `bcd.instance.memory`                                                       | bytes (`By`)                  | each domain's configured memory, by the same attributes and `hugepages` (the backing page size in bytes, `0` for normal pages)                                                                                                                                                                                                                                                                                   |
| `bcd.instance.memory.balloon.reclaimed`                                     | bytes (`By`)                  | the memory a ballooned guest lends the host now: its configured memory less what its balloon leaves it, by `instance.id` and `runtime` (9bdd43a; `bcd.instance.memory.balloon` before). Only for running, managed guests whose spec gives them a balloon (`placement.balloon`); a guest without a balloon reading counts as fully deflated (0). See [memory ballooning](/docs/daemon/delivery#memory-ballooning) |
| `bcd.instance.cpu.time`                                                     | seconds (`s`), cumulative sum | live usage (0b21b4b): CPU time since the domain started, by `instance.id` and `runtime`                                                                                                                                                                                                                                                                                                                          |
| `bcd.instance.memory.used`                                                  | bytes (`By`)                  | live usage: the memory the domain uses now                                                                                                                                                                                                                                                                                                                                                                       |
| `bcd.instance.io.read`, `bcd.instance.io.write`                             | bytes (`By`), cumulative sum  | live usage: disk bytes read and written                                                                                                                                                                                                                                                                                                                                                                          |
| `bcd.instance.network.receive`, `bcd.instance.network.transmit`             | bytes (`By`), cumulative sum  | live usage: bytes the guest received and sent                                                                                                                                                                                                                                                                                                                                                                    |

**Instances.** The host and instance series come every interval from a node that has them: the memory and huge page
series from any Linux host, the `bcd.instance*` series only from a host that runs an instance runtime (VPS and
sandbox hosts). The configured series (`bcd.instances`, `bcd.instance.vcpu`, `bcd.instance.memory`) cover every
runtime, plugin runtimes included. The live usage series come only from runtimes that can read it
(`runtime.UsageReader`), for their running domains, and count from the guest's side:

| Runtime            | Where the usage comes from                                                                                                                                                                                                                                                                                                                     |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cloud-hypervisor` | The VMM unit's cgroup v2, `/sys/fs/cgroup/bcd-instances.slice/bcd-ch-<id>.service`: `cpu.stat` `usage_usec`, `memory.current`, and `io.stat`'s `rbytes` and `wbytes` over every device; plus the counters of the instance's taps, where a tap's transmitted bytes are what the guest received. Managed domains that are running or paused only |
| `libvirt-kvm`      | `virsh domstats --raw --state-running --cpu-total --balloon --interface --block`: `cpu.time`, `balloon.rss` (the QEMU process's resident memory) and the summed `net.*` and `block.*` byte counters                                                                                                                                            |

A domain whose usage cannot be read is left out of the live series. Plugin runtimes report none, and a fakehw
build's file-backed runtime reports synthetic usage (half the configured memory, a second of CPU per boot). Under
[`bcd helper`](/docs/daemon#privilege-separation) `libvirt-kvm` reads `domstats` on libvirt's read-only socket
(since 12e9ced; before it the read was refused there), and `cloud-hypervisor`, offered there since 7fe74a1, reads
the same cgroup and tap counters as the node's own user, with no `ch.*` operation: since 1c9684a
`TestCloudHypervisorUnderTheHelper` (`internal/runtime`) reads a helper-started domain's usage from a fake cgroup tree
and tap statistics and checks that root was asked nothing. The series are tested against a fake `/proc`, sysfs,
cgroup tree and captured `domstats` output (`internal/agent` `telemetry_host_test.go`, `internal/runtime`
`usage_test.go`). On the control-plane side, `tests/e2e/helper-vps.test.ts` runs sandboxes on FakeDaemon VPS hosts
under the helper and checks that their `bcd.instance.*` usage reaches the collector `nodeTelemetryModule` names, and
leaves the next export once they are deleted (see [testing](/docs/operations/testing)).

GPU points carry `gpu.uuid`, `gpu.index` and `gpu.model`. The resource carries `service.name=bcd`,
`service.version`, `host.name`, `bcd.node.id` and `bcd.site.id`, then the section's own attributes (which cannot
replace those). bcd fetches the headers with a `SecretRequest` that names workflow `host:telemetry` and step
`headersRef`; the secret broker releases the reference only while the node's current spec names it there (binding
rule 3; see [sealed secrets](/docs/operations/security#sealed-secrets)).

**Failures.** An endpoint that is not an http(s) URL, or a malformed resource attribute, fails the `telemetry`
section terminally; an empty endpoint stops the export. A collector's 4xx answer other than 429 drops that batch;
after anything else the export runs again on the next interval, with that interval's fresh readings (a failed batch
is never re-sent). bcd logs a failure once, not every interval, and logs when the
export resumes. The exporter restarts whenever the section changes. `bcd status` prints the endpoint, the last
export and the last error (ad8791f). It is tested against a fake collector, and since efdff1d end to end: the real
fakehw bcd in `tests/e2e/bcd.test.ts` posts its GPU metrics to a local OTLP receiver the test runs, named in
`NodeSpec.telemetry` (see [testing](/docs/operations/testing)). No collector product has received it yet.

## Host metrics of machines without bcd [#host-metrics-of-machines-without-bcd]

An attached VM runs no bcd, so it exports nothing of its own. A provider whose cloud measures its machines offers a
pull source instead (`ExternalMetricsSource`: DigitalOcean's is `doMetricsSource`, over its Monitoring API), and
`externalMetricsCollector(registry, bindings)` turns the samples into gauges:

```ts title="external-metrics.ts"
import { externalMetricsCollector } from '@better-compute/observability';
import { doMetricsSource } from '@better-compute/provider-digitalocean';

const collector = externalMetricsCollector(metrics.registry, [
  {
    source: doMetricsSource(config, context), // cpu, memory and load; filesystem and bandwidth opt in
    machines: () => attached, // [{ instance: 'ins_…', machineId: '<droplet id>' }]
  },
]);
setInterval(() => void collector.pull(), 5 * 60_000);
```

* **Series.** One gauge per metric, `better_compute_external_host_<metric>{source, instance}`: `cpu_utilization_ratio`,
  `memory_used_bytes`, `memory_total_bytes`, `load1`, `filesystem_used_bytes`, `filesystem_size_bytes`,
  `network_receive_bytes_per_second`, `network_transmit_bytes_per_second`, plus
  `external_metrics_last_pull_timestamp_seconds{source}` and `external_metrics_pull_failures_total{source, code}`.
* **Bounded.** A source cannot add series: only the machines asked for, known metrics and finite values are kept.
  A sample older than 15 minutes (`staleAfterMs`) reads NaN, so a machine that stopped reporting never shows a frozen
  value, and a machine no longer listed is dropped.
* **Failures.** A failing source is counted by error code and keeps its samples until they go stale; one source
  never stops another. Each pull has 60 seconds (`pullTimeoutMs`).
* **DigitalOcean.** The Droplet needs DigitalOcean's metrics agent (`monitoring: true` on the account installs it on
  new Droplets) and the token `monitoring:read`. Droplets are read four at a time (`concurrency`) on the account's
  `sweep` budget; when it runs out, or the caller's signal aborts at its deadline, the pull answers what it read and
  the next one starts at the first Droplet it read nothing for.

**Through vps.** A host does not have to build the machine list or the source itself. With
`vps({ externalMetrics: {} })` (every `5m` by default, at least `1m`; off unless set, since it spends the accounts'
rate budgets) the `vps.external-metrics` worker asks each provider account whose client reads metrics
(`ProviderImpl.readMetrics`: DigitalOcean unless the account turns `monitoring` off) about its live instances, and
keeps the newest sample of each metric in `instance_external_metrics`. Every process then serves the same gauges from
that table, and the provider is asked once per period, not once per replica:

```ts title="metrics.ts"
import { externalMetricsCollector, storedExternalMetrics } from '@better-compute/observability';

const collector = externalMetricsCollector(metrics.registry, [storedExternalMetrics(sql, 'digitalocean')]);
setInterval(() => void collector.pull(), 60_000); // a database read, no provider call
```

Only the instances asked for, well-formed metric names and finite values are stored; an instance that is no longer
live loses its rows, and a failed read is logged and waits for the next period while the stored samples go stale.
Each account's read has 45 seconds and asks for at most `machineLimit` instances (default 500); an account with more
is read a page per period in id order (logged once), and its live instances on other pages keep their rows.

## Health and readiness [#health-and-readiness]

The CLI host answers two routes beside the API on every `api` listener:

| Route          | Answers                                                                                |
| -------------- | -------------------------------------------------------------------------------------- |
| `GET /healthz` | 200 `{ "ok": true }` while the process serves requests                                 |
| `GET /readyz`  | 200 `{ "ok": true }` when `SELECT 1` succeeds on the store, else 503 `{ "ok": false }` |

Point the load balancer's health check at `/readyz`. A gateway-only or worker-only process has no HTTP listener;
watch it through the gateway series and leases instead.

## doctor [#doctor]

`better-compute doctor` runs `compute.selfCheck()` and adds the CLI's own findings. It only reads. It always prints
the report, and exits non-zero with `DOCTOR_FINDINGS` when any finding is an error, so a deploy step that runs it
stops on one; `--strict` exits non-zero on warnings as well, and informational findings never fail it. `--pki PATH`
names the PKI bundle to check.

```bash
better-compute doctor --strict --query "findings[].check"
```

| Check                          | Severity         | Finds                                                                                                                                                                                                                                                                |
| ------------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `migrations`                   | error            | A database never migrated, or migrations of this release and its modules not applied                                                                                                                                                                                 |
| `btree-gist`                   | error            | The extension the booking exclusion constraints need is missing                                                                                                                                                                                                      |
| `clock`                        | error or info    | `compute_now()` is not the database clock (info when it reads the sim clock)                                                                                                                                                                                         |
| `clock.skew`                   | warning          | This process's clock differs from `compute_now()` by more than 1 s                                                                                                                                                                                                   |
| `catalog.*`                    | error or warning | Methods whose actions are uncatalogued, mismatched or undeclared                                                                                                                                                                                                     |
| `database`                     | error            | The check itself could not run                                                                                                                                                                                                                                       |
| `workers.queue`                | warning          | Work items more than 5 minutes overdue and unclaimed, noting when no worker has heartbeated in the last minute                                                                                                                                                       |
| `workers.sweeper`              | warning          | Pending commands more than 2 minutes past `not_after` and not yet expired: no worker holds the CommandSweeper's lease                                                                                                                                                |
| `usage.dead-letters`           | warning          | Unresolved rows in `usage_dead_letters`                                                                                                                                                                                                                              |
| `usage.backlog`                | warning          | Closed usage windows unrecorded for more than 2 hours                                                                                                                                                                                                                |
| `fleet.gateway-headroom`       | warning          | Live, non-draining gateways holding at least 80% of their `max_streams`                                                                                                                                                                                              |
| `fleet.heartbeats`             | warning          | Connected nodes whose session sent nothing for more than 2 minutes                                                                                                                                                                                                   |
| `fleet.daemon-skew`            | warning          | Connected daemons whose newest protocol is below the control plane's minimum                                                                                                                                                                                         |
| `fleet.cert-expiry`            | warning          | Connected nodes with no unrevoked certificate valid for another hour: renewals are failing                                                                                                                                                                           |
| `iam.not-configured`           | warning          | No IAM is configured, so only public methods answer                                                                                                                                                                                                                  |
| `iam.projection`               | error            | The authz projection is not loaded, so every decision is denied                                                                                                                                                                                                      |
| `iam.inbox-spill`              | warning or error | IAM events spilled in memory, or dropped                                                                                                                                                                                                                             |
| `iam.inbox-backlog`            | warning          | More than 10,000 IAM events wait in `iam_inbox`                                                                                                                                                                                                                      |
| `iam.usage-dead-letters`       | warning          | Usage events that could not be recorded in IAM billing                                                                                                                                                                                                               |
| `iam.usage-backlog`            | warning or error | Usage windows over 2 hours unrecorded (warning), or windows of a past month that IAM's `closePeriod` will refuse (error)                                                                                                                                             |
| `events.webhook-dead-letters`  | warning          | Webhook deliveries that failed every attempt                                                                                                                                                                                                                         |
| `events.webhook-backlog`       | warning          | More than 1,000 deliveries overdue by 5 minutes                                                                                                                                                                                                                      |
| `capacity.reservations.stale`  | warning          | Reservations more than 10 minutes past their end and not released                                                                                                                                                                                                    |
| `capacity.mig.stuck`           | warning          | Nodes that have waited more than 24 hours for a MIG layout change                                                                                                                                                                                                    |
| `capacity.dedicated.unoffered` | warning          | Nodes dedicated to a tenant their pool is not offered to, so nobody can use them                                                                                                                                                                                     |
| `capacity.device_counts.drift` | error            | The trigger-kept per-node device counts the placement shortlist ranks on disagree with `devices` (a `TRUNCATE` bypasses the trigger)                                                                                                                                 |
| `vps.pools.runtime`            | error            | A `vps-host` pool runs a runtime this deployment does not list in `vps({ runtimes })`                                                                                                                                                                                |
| `vps.pools.single-tenant`      | error            | A container-runtime pool without `policy.singleTenant: true`, which cannot run instances                                                                                                                                                                             |
| `vps.pools.ipam`               | warning          | A `vps-host` pool without an IP range, so its instances get no address                                                                                                                                                                                               |
| `vps.hosts.capacity`           | warning          | Adopted VPS hosts without vCPU, memory and disk capacity rows: no inventory yet, or no worker runs `vps.hosts`                                                                                                                                                       |
| `vps.sandboxes.expiry`         | error            | Sandboxes more than two reconciler passes (30 s) past `expiresAt` and not deleting: no worker runs the `vps.sandboxes` reconciler, or it is failing                                                                                                                  |
| `vps.sandboxes.deleting`       | warning          | Sandboxes deleting for more than 10 minutes: their instance is stuck (its host must report it gone, or be declared lost)                                                                                                                                             |
| `vps.sandboxes.warm`           | warning          | Templates below `warm.min` settled warm instances for over 10 minutes with no refill in that time (quota, capacity, a vanished flavor or image), and warm instances not settled within 10 minutes                                                                    |
| `vps.sandboxes.network`        | error            | Open sandboxes with a `private` or `none` network on a host without the `bc.dev/cap.instance.network-policy` label (placed before the capability constraint, or its bcd since downgraded): their guests have full egress. Upgrade bcd there, or delete the sandboxes |
| `workflows-stuck`              | warning          | `provision`: active workflows with an in-flight step more than 5 minutes past its deadline. The engine fails an overdue step whenever it looks, so nobody runs their `workflow` reconciler                                                                           |
| `workflows-failed`             | info             | `provision`: workflows that failed in the last day; inspect with `workflows.get`, retry with `workflows.retry`                                                                                                                                                       |
| `metal-failed`                 | warning          | `metal`: machines `failed` with maintenance on, waiting for `metal.recover` or `metal.retire`, listed with their failure codes                                                                                                                                       |
| `metal-stalled`                | warning          | `metal`: machines more than 10 minutes in a workflow-driven state without an active workflow, so no worker runs the `metal` reconciler (a rescued machine resting after its rescue workflow is not stalled)                                                          |
| `pool-telemetry-invalid`       | warning          | Pools whose `policy.telemetry` fails `nodeTelemetryModule`'s checks; their nodes export to the default collector (or none) until it is fixed                                                                                                                         |
| `audit.checkpoints`            | warning          | Audit rows older than 3 hours not sealed by a checkpoint                                                                                                                                                                                                             |
| `history-partitions`           | warning          | `retentionModule()`: this or next month's partition of a history table is missing, so the retention pass is not running                                                                                                                                              |
| `history-default-partition`    | warning          | `retentionModule()`: rows of the last 13 months sit in a history table's default partition                                                                                                                                                                           |
| `retention-cursors`            | warning          | `retentionModule()`: a durable `outbox_cursors` row has not moved for a day, so its stream is not pruned behind it                                                                                                                                                   |
| `pki.*`                        | error or warning | The PKI bundle unreadable or not unsealable, or a CA expired (error) or expiring (warning)                                                                                                                                                                           |
| `host.*`                       | error or warning | The `host` export: routes, tunnels, IAM, and whether the gateway role can authenticate daemons, reach the API's pokes, tunnels and revocations                                                                                                                       |

[`system.selfCheck`](/docs/reference/api/system#selfcheck) serves the server's part of the report over the API
(action `compute:fleet:admin` on `compute-fleet`); the `pki.*`, `host.*` and `iam.not-configured` findings come from
the CLI only.

The `workers.*`, `usage.*` and `fleet.*` checks are one read-only statement the server runs once the schema is fully
migrated (they are skipped while a `migrations` finding stands). Each is a warning with a `count` and a `fix`, and
each maps to an operations check of PLAN.md section 14: queues, the sweeper, usage, gateway headroom, heartbeats,
daemon version skew and certificate expiry.

## What to alert on [#what-to-alert-on]

These follow from the series and findings above, and from PLAN.md section 8.6 on what breaks first.

* **The gateway is fenced or stuck.** `better_compute_gateway_lease_held` at 0, or
  `better_compute_gateway_loop_consecutive_failures` rising for any loop. A gateway that loses its lease drops its
  streams; nodes re-home to the others.
* **Stream headroom.** `better_compute_gateway_stream_headroom` near 0 (default `maxStreams` 5,000 per gateway):
  add a gateway before handshakes are refused.
* **The IAM lane.** `better_compute_iam_call_duration_seconds` p99 rising, or `iam_calls_total{outcome="error"}`.
  Every IAM call takes at least one global lock acquisition in embedded mode.
* **Backlogs.** `workers.queue`, `workers.sweeper`, `iam.inbox-backlog`, `usage.backlog`, `events.webhook-backlog`
  and `audit.checkpoints` findings: a worker role is not running or cannot keep up. Run `doctor --strict` on a schedule and alert on a non-zero exit.
* **Clock.** `clock.skew`: run NTP on every host. Business time stays anchored to the database, but ids and logs use
  the host clock.
* **Stale reservations.** `capacity.reservations.stale`: the `controllers` role is not running reconcilers.
* **Revocations.** `better_compute_revocation_feed_consecutive_failures` above 0: revoked certificates keep their
  sessions while the feed fails.
* **Series cap.** `better_compute_metrics_collapsed_total` rising means some label is not bounded.

  - [Scaling](/docs/operations/scaling): The numbers behind these limits.

  - [Retention](/docs/operations/retention): What the housekeeping pass prunes.

  - [Security](/docs/operations/security): The threat model and hardening.

  - [The gateway](/docs/daemon/gateway): Leases, loops and admission control.


# PKI (/docs/operations/pki)

> The local CAs, JoinService keys and signing keys a deployment owns, the pki CLI and bundle, SPIFFE IDs, node certificate lifetimes, revocation and rotation.



Better IAM identifies people and AI agents. Machines are identified by Better Compute's own PKI: a set of local
certificate authorities and keys that one deployment owns, created once with `better-compute pki init` and loaded by
every process at start. Everything is ECDSA P-256 with SHA-256, except the two SSH CAs, which are Ed25519 by default, and
the package needs nothing beyond `node:crypto`.

The design rule is that **no key does two jobs**. A CA that signs agent certificates cannot sign a gateway's server
certificate, and the key that signs `JoinResponse` is not the one join tokens are sealed to.

## The keys [#the-keys]

| Key                         | Signs or does                                                                                                                     | Lifetime                                             | Who trusts it                                                   |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- |
| Root CA                     | Only the three subordinate CAs                                                                                                    | 10 years                                             | Anchor above the CAs; can be kept offline                       |
| Node CA                     | Agent client certificates: one SPIFFE URI SAN (then only `urn:bc:` key statements), the role as `OU=bc-role:<role>`, `clientAuth` | CA 1 year; leaves at most 24 h (ramdisk at most 2 h) | Gateways, as the trust anchor for daemon mTLS                   |
| Gateway serving CA          | Gateway server certificates: DNS SANs, `serverAuth`                                                                               | CA 1 year; leaves at most 30 days                    | Daemons, by its SPKI pin                                        |
| Internal CA                 | Control-plane mutual TLS between api, workers and gateways                                                                        | CA 1 year; leaves at most 7 days                     | The control plane                                               |
| JoinService signing key     | `JoinResponse` and `SignedTrustBundle`                                                                                            | Certificate 10 years                                 | Daemons, by pin                                                 |
| JoinService encryption key  | Nothing: join tokens are sealed to it                                                                                             | —                                                    | Daemons, by pin                                                 |
| Command-signing root (v0.2) | 24 h worker keys, which sign every command                                                                                        | 5 years                                              | Daemons, through the trust bundle                               |
| Exec ticket key (v0.3)      | 60 s single-use tunnel tickets                                                                                                    | —                                                    | Gateways (their tunnel JWKS) and daemons (the JWKS in NodeSpec) |
| SSH CA (v0.3)               | Break-glass OpenSSH user certificates, at most 1 h                                                                                | —                                                    | Nodes' sshd, through NodeSpec                                   |
| Sandbox SSH CA (v0.5)       | `sandboxes.exec` certificates, at most 15 minutes, for `<account>@<instanceId>`                                                   | —                                                    | Sandbox guests' sshd, through the user data vps renders         |
| Audit signing key (v0.4)    | Per-tenant Merkle checkpoints of `audit_events`                                                                                   | Certificate 10 years                                 | `audit verify` and third parties, by pin                        |

Subordinate CAs never outlive the root: a CA certificate's `notAfter` is capped at the root's. Leaf lifetimes are
enforced by the CA itself, whatever a caller asks for.

The gateway holds none of these keys in its own code: it relays `Join`, `Renew` and secret requests to JoinService
and never signs anything. See [security](/docs/operations/security#gateway-processes) for what that means for the
process that runs it.

## The pki commands [#the-pki-commands]

### pki init [#pki-init]

```bash
export BETTER_COMPUTE_SECRET=$(openssl rand -base64 48)   # the same for every process
better-compute pki init --trust-domain compute.example.com
```

It creates every key above and writes one JSON file, mode 0600, never overwriting an existing one. The private keys
are sealed with AES-256-GCM under a key derived (HKDF-SHA256) from `BETTER_COMPUTE_SECRET`, with the trust domain in
the authenticated data. The sealed payload also carries a SHA-256 over every public field (certificates, trust
anchors, the trust version), so whoever can write the file but lacks the secret cannot add a command-signing root or
pin a trust version: loading refuses a sealed bundle whose public parts changed (`INVALID_CONFIG`; restore it from a
backup), while reformatting the JSON is harmless. Any CLI write-back re-seals in the current format. `pki init`
prints the pins, like `pki show`.

> **Re-seal bundles sealed before 482dea8.** 
  A bundle sealed in the former format (before the seal covered the public parts, commits 482dea8 and 7b9e9af) is
  refused wherever it is loaded (`INVALID_CONFIG`: "sealed in the former format"), and so is an unsealed bundle when
  `BETTER_COMPUTE_SECRET` is set, since a sealed one may have been replaced. Nothing authenticates such a file: whoever
  could write it could have added a trust anchor or a second SSH CA line. Once, with `BETTER_COMPUTE_SECRET` set:

  1. run [`pki reseal`](#pki-reseal). For such a bundle it prints the pins it would vouch for, with
     `authenticated: false`, and stops with `INVALID_STATE`;
  2. compare those pins with your records (install scripts, join tokens, pinned CAs);
  3. run `pki reseal --accept-unauthenticated`, which writes the bundle in the current format.

  `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1` only lets other readers, such as a server, load the legacy bundle until
  it is resealed; never leave it set on a running deployment. The commands that write the bundle back
  (`pki rotate-command-root`, `pki add-sandbox-ca`, `pki renew-ca`) refuse it with `INVALID_STATE`: only `pki reseal`
  vouches for an unauthenticated bundle. A former-format bundle that carries command-signing rotation fields cannot be
  re-sealed: restore the copy from before the rotation and run `pki rotate-command-root` again.

| Flag                    | Default                                           | Meaning                                                                                    |
| ----------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `--trust-domain DOMAIN` | required (`BETTER_COMPUTE_TRUST_DOMAIN`)          | The SPIFFE trust domain of agent identities                                                |
| `--pki PATH`            | `.better-compute/pki.json` (`BETTER_COMPUTE_PKI`) | Where to write the bundle                                                                  |
| `--offline-root PATH`   | none                                              | Write the root key to its own file (sealed like the bundle) and leave it out of the bundle |
| `--organization NAME`   | `better-compute`                                  | Subject organization of the CA certificates                                                |
| `--plaintext`           | off                                               | Store the private keys as PEM when `BETTER_COMPUTE_SECRET` is not set                      |

Without the secret and without `--plaintext` it refuses with `MISSING_ENV`. Keep the `--offline-root` file off the
servers: it is needed only to replace a CA.

### pki show [#pki-show]

`pki show` reads the public half of the bundle. With `BETTER_COMPUTE_SECRET` it first checks that half against the
seal and prints `authenticated: true`; without the secret it still prints the pins, with `authenticated: false` and
a warning that whoever can write the file chooses them. `audit verify` and `daemon install-script` read pins the same
way. It prints:

| Field                                                                              | Used by                                                                                                                                                                             |
| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `trustDomain`                                                                      | Install scripts, `bcd` configuration                                                                                                                                                |
| `servingCaSpkiPin`                                                                 | Daemons check the gateway's certificate against it                                                                                                                                  |
| `joinSigningKeyPin`, `joinEncryptionKeyPin`                                        | Daemons check `JoinResponse` and seal join tokens only to a matching key                                                                                                            |
| `joinEncryptionJwk`                                                                | The key daemons seal join tokens to                                                                                                                                                 |
| `auditSigningKeyPin`                                                               | `audit verify`                                                                                                                                                                      |
| `execTicketKeyId`                                                                  | The `kid` of the exec ticket key (its RFC 7638 thumbprint)                                                                                                                          |
| `sshCaPublicKey`                                                                   | The OpenSSH public key sshd trusts                                                                                                                                                  |
| `sandboxSshCaPublicKey`                                                            | The sandbox SSH CA's public key, which sandbox guests trust (`vps({ sandboxExec: { caPublicKey } })`)                                                                               |
| `commandSigningRootSha256`                                                         | The command-signing root, as trust bundles list it                                                                                                                                  |
| `commandSigningNextRootSha256`, `commandSigningPreviousRootSha256`, `trustVersion` | During a root rotation: the root published but not signing yet, the former root still trusted, and the trust bundle's version                                                       |
| `expires`                                                                          | When the root, node, serving, internal and command-signing certificates expire                                                                                                      |
| `certificates`                                                                     | The hex SHA-256 of every certificate (DER) of every stored chain, in file order. Pins name only a chain's first certificate, so this is what shows a certificate appended behind it |

Pins have `bcd`'s form, `sha256/<base64 of SHA-256(SPKI)>`. `better-compute daemon install-script` reads the bundle
and writes the serving CA pin and the JoinService pins and key into each machine's configuration.

### pki issue-gateway [#pki-issue-gateway]

```bash
better-compute pki issue-gateway --san gw.eu-1.compute.example.com --out /etc/better-compute/tls --days 14
```

It generates a P-256 key and issues a serving certificate from the bundle's serving CA. It writes `NAME.key` (mode
0600\) and `NAME.crt` (the certificate followed by the serving CA, stopping below the root) into `--out`, replacing
earlier files atomically.

| Flag          | Default                    | Meaning                               |
| ------------- | -------------------------- | ------------------------------------- |
| `--san NAMES` | required                   | DNS names, comma-separated            |
| `--out DIR`   | `.better-compute/gateway`  | Directory for the key and certificate |
| `--name NAME` | `gateway`                  | File name stem                        |
| `--days N`    | 30                         | Lifetime, from 1 to 30 days           |
| `--pki PATH`  | `.better-compute/pki.json` | The bundle                            |

### pki add-sandbox-ca [#pki-add-sandbox-ca]

```bash
better-compute pki add-sandbox-ca
```

Bundles from `pki init` already hold a [sandbox SSH CA](#the-sandbox-ssh-ca). For a bundle made before sandboxes,
this creates one (Ed25519, comment `<trust domain> sandbox ssh ca`), writes the bundle back (sealed when
`BETTER_COMPUTE_SECRET` is set; an offline root stays out) and prints `sandboxSshCaPublicKey`. A bundle that already
has one is `INVALID_STATE`. It writes the bundle back, so it refuses `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`
(`INVALID_STATE`): re-seal a former-format bundle with [`pki reseal`](#pki-reseal) first. Roll
the bundle out to every api replica, and give vps the public key as `sandboxExec.caPublicKey`. The only flag is
`--pki`.

### pki reseal [#pki-reseal]

```bash
better-compute pki reseal                            # an unauthenticated bundle: prints its pins and stops
better-compute pki reseal --accept-unauthenticated   # after comparing them with your records
```

It rewrites the bundle sealed with `BETTER_COMPUTE_SECRET` (required, at least 32 characters; `MISSING_ENV`
otherwise) in the current format, where the seal also covers every certificate and pin, so a file changed without the
secret no longer loads. It keeps every key and certificate: nothing is rotated.

* **An authenticated bundle** (already sealed in the current format under this secret) is simply resealed.
* **An unauthenticated bundle** (sealed in the former format, or unsealed while `BETTER_COMPUTE_SECRET` is set) is
  not sealed at first: resealing would vouch for whatever the file holds. The command prints the pins it would vouch
  for, the same fields as `pki show` with `authenticated: false`, and stops with `INVALID_STATE` ("nothing vouches for
  the pins above"). Compare them, and the `certificates` digests (which also reveal a certificate appended behind a
  pinned one), with your install scripts, join tokens and pinned CAs, then run it again with
  `--accept-unauthenticated`.

After sealing it prints the pins it now vouches for, with `authenticated: true`. It needs no
`BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED`.

| Flag                       | Default                    | Meaning                                                                          |
| -------------------------- | -------------------------- | -------------------------------------------------------------------------------- |
| `--accept-unauthenticated` | off                        | Seal a bundle nothing authenticates, after comparing the pins printed without it |
| `--pki PATH`               | `.better-compute/pki.json` | The bundle                                                                       |

### pki renew-ca [#pki-renew-ca]

```bash
better-compute pki renew-ca --ca node
better-compute pki renew-ca --ca serving --root-key /media/usb/root-ca.key
```

It issues a new certificate for the node, serving or internal CA from the root, for the same key and subject, and
writes the bundle back (sealed when `BETTER_COMPUTE_SECRET` is set). Certificates the CA already issued stay valid and
the serving CA's SPKI pin does not change, so nothing is re-pinned and no node rejoins. Roll the bundle out to every
process before the old certificate expires. It prints the CA, the new `notAfter` and serial, and for `serving` the
unchanged pin. Like every command that writes the bundle back, it refuses `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`
with `INVALID_STATE`.

| Flag              | Default                    | Meaning                                                                                                       |
| ----------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `--ca CA`         | required                   | `node`, `serving` or `internal`                                                                               |
| `--root-key PATH` | none                       | The offline root key (`pki init --offline-root`), when the bundle has none; it is not written into the bundle |
| `--days N`        | 365                        | Lifetime of the new certificate, 1 to 3650 days, capped by the root                                           |
| `--pki PATH`      | `.better-compute/pki.json` | The bundle                                                                                                    |

### pki trust-version [#pki-trust-version]

```bash
better-compute pki trust-version
```

It reads the trust bundle the deployment last published (`trust_bundle_state`; its version
[follows its content](#signed-intent)) through the deployment's configuration and compares it with this
configuration and PKI bundle (03bdbc9). It prints `version` (the published one), `configurationMatches` (whether
this bundle's roots and the configured quorum publish the same bundle), `nextVersion` (what this configuration would
publish: the same version, or the next one), `approverSetEpoch` (the published approver set's epoch),
`approverSetSha256` (this configuration's set digest, for [pinning the first set](#pinning-the-first-set), with
`publishedApproverSetSha256` while they differ), `approverSetBehind` and `approverSetSample` (connected nodes
enforcing another set, as the gateway records it, a node without keys included; see
[nodes that hold another set](#changing-the-approver-set); 6612496, 08eb62e), `quorumChanged`
and, when it did, `signQuorumForEpoch` (the epoch to sign the change for with
[`batch sign-approvers`](#changing-the-approver-set), one above the published one; c6214dc), and how many connected
daemons there are and how many are behind that version. Before any process has published a bundle it is `NOT_FOUND`. `--pki PATH`
names the bundle.

### Reloading gateway certificates [#reloading-gateway-certificates]

A running gateway picks a reissued certificate up without a restart when it reads it from files. Point
`host.gateway.tlsFiles` at them (and `host.gateway.internal.tlsFiles` at the internal-CA pair of the control API):

```ts title="better-compute.config.ts"
export const host = defineHost(() => ({
  gateway: {
    region: 'eu-1',
    tls: {}, // other listener options; key, cert and ca come from the files
    tlsFiles: {
      key: '/etc/better-compute/tls/gateway.key',
      cert: '/etc/better-compute/tls/gateway.crt',
      ca: '/etc/better-compute/tls/node-ca.pem', // node CA roots, for client verification
      intervalMs: 60_000, // the default
    },
  },
}));
```

`serve` reads the files at start over `tls` and then checks their modification time and size every `intervalMs`. A
changed set is loaded into a new secure context first and swapped onto the daemon listener and the WebSocket
fallback (or the control API): new handshakes get the renewed certificate, and open sessions keep theirs, so no daemon
is disconnected by a rotation. A half-written or mismatched pair is reported (`Gateway serving certificate not
reloaded: …`) and the listener keeps its certificate; the next change is tried again. Without `tlsFiles`, the
certificate in `tls` is read once at start, so after reissuing, restart gateways one at a time.

Outside `serve`, `serveGateway`, `serveGatewayWs` and `serveInternal` from `better-compute/gateway` return
`setTls({ key, cert, ca? })`, which applies new material over the listener's original TLS options (protocol versions
and the rest stay) and throws `INVALID_CONFIG`, changing nothing, when the key does not match its certificate.
`watchTlsFiles({ key, cert, ca? }, apply, { intervalMs?, onError?, onReload? })` polls the files and calls `apply`
with a changed set, for example `(tls) => served.setTls(tls)`.

## The bundle in code [#the-bundle-in-code]

The bundle is a JSON document (`format: "better-compute-pki"`, `version: 1`) with the certificates and public keys
in the clear and the private keys either sealed or, with `--plaintext`, as PEM.

| Function                                                              | Package              | Does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| --------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `loadPkiBundle(path, env)`                                            | `better-compute/cli` | Reads the file and unseals it with `env.BETTER_COMPUTE_SECRET` (and `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`, which loads a legacy bundle until it is resealed)                                                                                                                                                                                                                                                                                                                                          |
| `importPkiBundle(text, { secret?, rootKey?, allowUnauthenticated? })` | `better-compute/pki` | Loads a bundle; `INVALID_CONFIG` on a wrong secret, a damaged file, a key that does not match its certificate, changed public parts, or (unless `allowUnauthenticated`) a former-format seal or an unsealed bundle with a secret                                                                                                                                                                                                                                                                             |
| `exportPkiBundle(bundle, { secret?, omitRootKey? })`                  | `better-compute/pki` | Serializes a bundle, sealed when `secret` is given                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `exportRootKey(bundle, { secret? })`                                  | `better-compute/pki` | The root key alone, for keeping it offline                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `readPkiPins(text, { secret? })`                                      | `better-compute/pki` | The pins and expiry dates from the public half, the SHA-256 of every stored certificate (`certificates`), and `authenticated` (checked against the seal when `secret` is given)                                                                                                                                                                                                                                                                                                                              |
| `createPkiBundle({ trustDomain })`                                    | `better-compute/pki` | What `pki init` runs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `bootstrapPins(bundle)`                                               | `better-compute/pki` | `{ servingCaPin, joinSigningPin, joinEncryptionPin }` in bcd's `sha256/<base64>` form, taken from the bundle: what a new machine pins before it trusts anything the control plane says (`bcd join --serving-ca-pin --join-signing-pin --join-encryption-pin`), never the pins in the unsigned `/.well-known/better-compute`. Pass it to `provisionModule({ userData: { pins } })`, as the examples do; see [joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document) |
| `tunnelsOptions(bundle, { issuer })`                                  | `better-compute/pki` | The `tunnels` options: the exec ticket issuer, `ssh` (break-glass) and `sandboxSsh`, when the bundle has them                                                                                                                                                                                                                                                                                                                                                                                                |

Wire it into the options and the gateway:

```ts title="better-compute.config.ts"
const bundle = await loadPkiBundle(env.BETTER_COMPUTE_PKI ?? '.better-compute/pki.json', env);

export default defineConfig(() => ({
  // …
  pki: {
    nodeCa: bundle.nodeCa, // the gateway role's trust anchor and revocation set
    join: pkiJoinCrypto({
      trustDomain: bundle.trustDomain,
      nodeCa: bundle.nodeCa,
      signingKey: bundle.joinSigning,
      encryptionKey: bundle.joinEncryption.privateKey,
    }),
    commandSigning: bundle.commandSigning, // v0.2 signed intent
  },
  // The ticket issuer must be baseURL's origin: the api accepts only tickets from it.
  // Equivalent to { issuer: execTicketIssuer(bundle, …), ssh: bundle.sshCa, sandboxSsh: bundle.sandboxSshCa }.
  tunnels: tunnelsOptions(bundle, { issuer: 'https://compute.example.com' }),
  modules: [
    auditModule({ signer: bundle.auditSigning }),
    vps({ images, sandboxExec: { caPublicKey: bundle.sandboxSshCa!.publicKey } }), // sandbox guests trust it
  ],
}));
```

Bundles made before a key existed still load; they lack that key (`auditSigning`, `execTicketKey`, `sshCa`,
`sandboxSshCa`, `commandSigning` are optional), and `execTicketIssuer` then fails with `INVALID_CONFIG`.

`doctor` checks the bundle at `--pki`: `pki.unreadable` when it cannot be parsed, `pki.unseal` when it does not
open with this deployment's secret, `pki.ca-expired` for an expired CA, `pki.ca-expiring` for one that expires
within 30 days, and `pki.rotation` (info) while a command-signing root rotation is half done: a new root published
but not signing yet (run `--stage switch` once every daemon has the trust version), or a former root still trusted
(run `--stage drop` at least 24 hours after the switch). Doctor (and the support bundle) also compares the
configured approver quorum with the published one (03bdbc9): `pki.quorum-unsigned` (warning) when it differs and its
current approvers have not signed it for a higher epoch, since daemons holding approver keys would keep the old
quorum (the fix names the exact `batch sign-approvers --epoch`, the published epoch plus 1, and
`pki.approvers.epoch`; c6214dc), and `pki.quorum-change` (info) when the change is signed for a higher epoch or
introduces the first set.

## Trust domain and SPIFFE IDs [#trust-domain-and-spiffe-ids]

Every agent certificate carries exactly one SPIFFE URI SAN, first:

```text
spiffe://<trust-domain>/site/<siteId>/node/<nodeId>
```

The trust domain is lower-case `a-z`, `0-9`, `.`, `-` and `_`, up to 255 characters, such as `compute.example.com`.
The node id always comes from this SAN, never from a message field, so a daemon cannot speak for another node. The
gateway checks the SPIFFE ID's trust domain against its configured one.

Since d94f1f3 the SPIFFE ID may be followed by `urn:bc:` statements: what compute proved about the node's key when
it issued the certificate. They are never identities. The node CA accepts at most four URI SANs, the SPIFFE ID and
then only statements (`urn:bc:<kind>:<value>`, printable ASCII, `isNodeKeyStatement`), and no DNS SANs. The one
statement today is written after a certified `tpm` join:

```text
urn:bc:tpm-ak:<nameAlg>:<digest>
```

It is the TPM name (hex: the 2-byte name algorithm, then a digest of 20 to 64 bytes) of the attestation key that
certified the node key (`tpmAkUri`, `parseTpmAkUri` in `@better-compute/pki`; `pki.TPMAKName` in bcd). Every reader
of a node identity takes the one URI SAN that is not a statement: `nodeSpiffeIdOf(uriSans)` in TypeScript (mTLS
identity, proofs of possession, join responses, the gateway's renew relay) and `pki.NodeIdentity` in bcd, so a
certificate with two SPIFFE IDs, or none, has no identity.

The role travels in the subject as one `OU=bc-role:<role>` attribute (1 to 32 of `a-z`, `0-9` and `-`, starting
with a letter; a subject with two roles is malformed): `node`, `site`, `ramdisk` or `node-tenant` for agents,
`gateway`, `internal` and `worker` for control-plane certificates. No certificate carries an OID under the private
`2.25.<uuid>` arc any more, because Go's `crypto/x509` refuses to parse such certificates, so bcd could not have
read them. For one release, `parseCertificate` still reads the old role extension of certificates issued before the
change; the OU wins when both are present, and Go accepts only the OU.
Each role accepts only its own command kinds. A `node-tenant` certificate (a dedicated-root lease) runs as a node
that gets no secrets and no destructive platform commands, and never attests.

## Node certificates [#node-certificates]

JoinService issues a node certificate when a machine joins with a join token or
another attestor, and records it in `node_certs`:

| Column                            | Meaning                                                                                                |
| --------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `serial`                          | The certificate serial                                                                                 |
| `node_id`, `role`                 | Whose it is, and as what                                                                               |
| `identity_gen`                    | The node's identity generation when issued; a new generation revokes the old certificates              |
| `spki_sha256`, `der`              | The key and certificate, so node signatures can be checked later                                       |
| `not_before`, `not_after`         | Validity                                                                                               |
| `revoked_at`, `revocation_reason` | Revocation                                                                                             |
| `allowed_kinds`                   | When set, the only command kinds this certificate may receive; others are cancelled `kind-not-allowed` |

Lifetimes: `node` and `site` certificates live 24 h; `ramdisk` certificates at most 2 h, whatever the profile asks.
A daemon renews at about 60% of its lifetime with `Renew`, signing a proof of possession with its **current** key
over the new CSR, its node id and its clock. The api checks the chain, revocation, the SPIFFE ID, a ±60 s window and
a 120 s replay cache keyed by the signed statement. A certificate with a `urn:bc:tpm-ak:` statement also needs
`RenewRequest.key_attestation`, that AK's certification of the new key; the statement is read from the presented
leaf (byte for byte its `node_certs` row), never from the CSR, and copied into the new certificate (see
[renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key)). A machine that lost its certificate re-joins with the `x509pop`
attestor, which accepts a certificate of the current identity generation up to 7 days past its expiry. See
[attestation](/docs/guides/nodes/attestation).

## Revocation [#revocation]

A certificate is revoked by setting `node_certs.revoked_at`. Every path that ends an identity does it:
`nodes.revokeIdentity`, node deletion, a reclaim, metal retire and a dedicated-root lease.

Each gateway runs a revocation feed. It loads every revoked, unexpired certificate on start, then polls
`node_certs.revoked_at` every second, re-reading 60 s before the newest revocation it has seen so out-of-order
commits are still caught. After three failures in a row it polls every 5 s until one succeeds, and never stops the
gateway. A serial the set learns about is refused at the next session open, and a live session gets a
`RevocationNotice` and is closed. The target is a revoked node disconnected within 5 s.

## Signed intent [#signed-intent]

With `pki.commandSigning` set, every command the queue writes is signed by a worker key: a P-256 key the
command-signing root certifies for at most 24 h, with role `worker` (`OU=bc-role:worker`). A signer is valid only as
a `worker` certificate chaining to a command-signing root; there is no command-signing extended key usage. Each
process replaces its worker key after 60% of its lifetime.
Daemons receive the roots in their trust bundle, first in `JoinResponse` and then in `Welcome`, signed by the
JoinService key so a gateway can relay it but not change it. A daemon checks any signature that is present. Once it
holds command-signing roots, a daemon requires a valid signature on every command whatever its kind, since an
unsigned non-destructive kind could still act: bcd since 0d42de8 and the FakeDaemon since 12ccfa1 (review 7; pki's
`VerifyIntentOptions.required`). Before, each daemon's kind table decided, requiring one only for destructive kinds.

The same worker key signs desired state (v0.4). When the NodeSpec controller writes a new generation, it encodes the
`DesiredStateBody` with pki's shared encoder (`storedDesiredState`, spec hash = SHA-256 of the encoded NodeSpec),
signs it under purpose `desired-state` (`CommandSigner.signDesiredState`) and stores both in `nodes.spec_body` and
`nodes.spec_signature` (`core/0021_signed_specs`) in the same `UPDATE` that bumps the generation. The gateway holds no
key: it relays the stored bytes verbatim (`desiredStateFromStored`), and falls back to an unsigned rendering from
`nodes.spec` (`desiredStateBodyFromJson`, byte-identical to the writer's) only when no current body is stored.
`verifyDesiredState(desired, { roots, now, nodeId, required })` is the daemon-side check.

Signed intent also changes how GPU bindings count: with `pki.commandSigning`, capacity's `attestedBindings`
defaults to `pkiAllocationBindingCheck()`, which verifies each `AllocationBound` against the reporting node's own
certificate in `node_certs` before the reservation goes active. See
[attested bindings](/docs/guides/capacity/reservations#attested-bindings).

Rotating the root takes three steps, gated on the fleet:

1. add the new root beside the old one (`pki.commandSigningRoots`); workers still sign under the old root, and the
   trust bundle's version goes up;
2. once every connected daemon reports that version, switch `pki.commandSigning` to the new root;
3. once nothing is signed under the old root any more, remove it; the version goes up again.

**The version follows the content** (9c40cd5). A daemon takes a trust bundle only when its version is newer than the
one it holds, so a changed bundle that kept its old version used to reach no daemon, silently. The deployment now
records the digest of the bundle it publishes (the set of root hashes and the canonical quorum fields) and its
version in `trust_bundle_state` (migrations `quorum/0003_trust_bundle_state` and `quorum/0004_trust_bundle_approver_set`,
which also stores the published quorum). Each process settles the version once, at `initialize()` or before a
gateway-only process first serves `JoinResponse.trust` or `Welcome.trust`: unchanged content keeps the recorded
version, changed content (roots, approvers, tiers, the unbatched cap) takes the next one, and processes with the same
bundle agree under a row lock. A bundle that was published before and has since been replaced never comes back as
newer (0a31734, review 7): `trust_bundle_history` (migration `quorum/0006_trust_bundle_history`) keeps every published
digest with its version, so a stale replica that restarts on an old configuration serves that bundle at its old
version, which daemons ignore, and logs a warning, rather than publishing it as the next version and making daemons
trust again, say, a command-signing root the new configuration removed. It does not change the record. Only an
explicit `pki.trustVersion` above the recorded version publishes such a bundle again. `pki.trustVersion` is now a
floor, not something you must bump: the published version is never below it. [Approver-set
signatures](#changing-the-approver-set) no longer name a version: since c6214dc they sign the set's own epoch
(`pki.approvers.epoch`), so they stay valid on every later version.

`better-compute pki rotate-command-root --stage add|switch|drop` runs one step on the bundle and writes it back
(sealed when `BETTER_COMPUTE_SECRET` is set), printing the roots and the bundle's trust version (with the content
rule above, the version the deployment publishes may be higher, never lower):

```bash
better-compute pki rotate-command-root --stage add     # a new root beside the current one; trust version up
# roll the bundle out to every process; daemons get the new version in Welcome.trust
better-compute pki rotate-command-root --stage switch  # workers sign under the new root; the old one stays trusted
# at least 24 hours later (worker keys live at most 24 h)
better-compute pki rotate-command-root --stage drop    # the old root is no longer trusted; version up again
```

A step out of order is `INVALID_STATE`, and `doctor` reminds you of a rotation left half done (`pki.rotation`). So is
a run with `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`: the command writes the bundle back, and only
[`pki reseal`](#pki-reseal) vouches for an unauthenticated one.

Every daemon reports the trust bundle version it holds: `Hello.trust_version` at session start, and
`StatusReport.trust_version` again once it applies a newer bundle from `Welcome.trust` mid-session, so the
acknowledgment does not wait for a reconnect. bcd and the FakeDaemon both send it. The gateway records it as
`node_sessions.trust_version` (`core/0023_session_trust_version`; a status report only ever raises it). The
`--stage switch` step reads that column and refuses with `INVALID_STATE` while any connected node is behind, naming a
few (`1 of 2 connected node(s) have not applied it (node_old at 1)`), since those nodes would reject every command
signed under the new root. Since the published version follows the content, the step gates on the version the
deployment published for this bundle, not the bundle's own `trustVersion`: it recomputes the digest this bundle and
configuration publish and refuses (`Not switching: the deployment still publishes trust version N without the new
root`) until every process runs the `--stage add` bundle, then waits for daemons to hold that published version
(03bdbc9). Connected means a session on a gateway, not ended, seen within the last 2 minutes; a node
that connects later gets the current bundle in `Welcome` before any command. The check needs the deployment's
configuration or `BETTER_COMPUTE_DATABASE_URL` and refuses without either; `--force` skips it. While a rotation is
in progress, the `pki.rotation` finding of `doctor` and the support bundle says how many connected nodes already have
the new version. `addTrustRoot`, `trustVersionAcknowledged` and `removeTrustRoot` in `better-compute/pki` implement
the same gates for your own tooling.

In production the design keeps this root in KMS or Vault transit; the bundle form is for development and small
deployments.

### The destructive-batch quorum (v1.0) [#the-destructive-batch-quorum-v10]

Signed intent stops a gateway from forging commands, but not a compromised api or worker, which holds a worker key.
PLAN.md §13's answer for fleet-wide destructive work (reimage, wipe, reboot of many machines) is a quorum checked on
the machine: destructive commands carry a `batch_id` in the worker-signed body, and run only once k distinct operator
**approver keys**, pinned in the trust bundle and held off the control plane, have signed a `BatchApproval` over the
Merkle root of the batch's scope. Each command ships with its inclusion proof, so the executing agent checks
membership itself and neither the control plane nor a gateway can widen the batch.

> **Built end to end; workflow leaves are unbound.** 
  Built and tested end to end: the protocol seams, the `better-compute/pki` primitives, `pki.approvers` in the signed
  trust bundle with approver-set signatures for quorum changes, the server's `quorum` module (batches, their stored
  scopes, payloads and proofs, `commandBatches` approvals), the command queue's `createBatch` and
  `enqueue({ batchId })`, the gateway attaching each batched command's approval and inclusion proof (40be696),
  `commandBatches.create` and `commands.run({ batchId })` for batches over the API (46afe75), the approvers'
  `better-compute batch` commands, enforcement in both bcd and the FakeDaemon, and the producers: metal's workflows,
  `metal.bulk` and `metal.power`, the health reconciler's remediations and the MCP server's `propose_command_batch`
  all create their batches (see [who creates batches](#who-creates-batches)). What stays open: the batches built-in
  workflows create leave every leaf's payload unbound (any payload of its kind for its subject), because the bytes
  are only known later, and approvers see those leaves flagged; binding a leaf to the operation instead of its bytes
  is a post-v1.0 option in PLAN.md §13 (see the [roadmap](/docs/reference/roadmap)).

**The scope.** A batch's scope is a tree with one leaf per distinct `(subject, kind, payload)`:
`H(0x00 ‖ lp(batch_id) ‖ lp(subject) ‖ lp(kind) ‖ lp(payload_sha256))`, leaves ordered by hash, inner nodes
`H(0x01 ‖ left ‖ right)` and an odd node promoted (RFC 6962 style; golden vectors in
`packages/protocol/golden/batch-scope.json`). `payload_sha256` binds a leaf to exactly one operation, or is left
empty to allow any payload of that kind for that subject; the agent accepts a proof from either the bound or the
unbound leaf (77cdf4f; `command_batch_scope` and `command_batch_members` carry it, migration `quorum/0002_payload_digest`).
A leaf may also carry its payload itself, `{ typeUrl, value }` (at most 64 KiB a leaf, and 8 MiB for all of a
batch's leaves): its digest is then derived from those bytes (a `payloadSha256` given as well must match), and the
payload is stored (migration `quorum/0007_leaf_payload`) so approvers can decode the image, firmware or power action
they sign, not only a hex digest (2ca0ba4). The `typeUrl` must be `type.googleapis.com/` plus the kind's own payload
type from the registry (fc2df7d, review 8), so a creator cannot label bytes as a harmless message.

Every subject must name a machine of the batch's own tenant (fc2df7d): the node itself, the site agent in
`bmc:<agent>/<address>`, or the target in `node:<id>`. A subject naming another tenant's node is `NOT_FOUND` and a
malformed one `INVALID_INPUT`, since approvers sign under the batch's tenant while the daemon checks only the leaf:
otherwise a compromised worker could spend tenant A's approval on tenant B's machines.
The subject is what the command acts on, computed by the executing agent from the signed body:

| Kinds                                                    | Subject                                                                   |
| -------------------------------------------------------- | ------------------------------------------------------------------------- |
| Kinds a node runs on itself                              | its `node_id`                                                             |
| `bmc.*`, which a site agent runs against a machine's BMC | `bmc:` + the site agent's `node_id` + `/` + `BmcOperationRequest.address` |
| `site.httpProxy`                                         | `node:` + `HttpProxyRequest.target_node_id`                               |

Without the BMC subject, "power-cycle 412 machines" would collapse into one `(site agent, bmc.power)` leaf, and
approving it would let a compromised control plane power any machine behind that agent. BMC addresses are
site-local, so the subject names the site agent too: an approval for a BMC at site A never admits a command at site B.

**The wire.** `CommandBody.batch_id` is inside the worker signature; `Command.batch_approval` (`batch_id`,
`scope_root`, `scope_size`, `not_after`, `signatures`) and `Command.batch_proof` (`index`, `siblings`) travel outside
it, since approvers sign the approval themselves. The `TrustBundle` gains `approver_keys` (`kid`, `spki_der`),
`approver_threshold`, `quorum_tiers` (`max_scope_size`, `threshold`), `unbatched_destructive_per_hour` and
`approver_set_signatures` (see [changing the approver set](#changing-the-approver-set)), and
`CommandRejectReason` gains `BATCH_UNAPPROVED` and `UNBATCHED_LIMIT`. The canonical quorum bytes approvers sign for
a set change (`approverSetInput`: a u32 key count, then each key's length-prefixed `kid` and SPKI in `kid` order, the
u32 threshold, the u32 tier count and each tier's pair, the u32 unbatched cap; all big-endian) have a golden vector
beside `batch-scope.json`, `packages/protocol/golden/approver-set.json`, with the set digest and the signing input
for an epoch, so bcd pins its encoding to the same bytes (26ab0fc). `approver_set_epoch` (field 8, c6214dc) numbers
the quorum fields separately from the bundle's `version`. Approvals are signed under the purpose
`batch-approval`, and the signed parts end with the deployment's trust domain, which every verifier takes from its
own configuration: an approval signed for staging never admits a command in production. The server therefore needs
`pki.join` (the trust domain) to accept approvals, and refuses the quorum with `INVALID_CONFIG` without it.

**The primitives** in `better-compute/pki` (`packages/pki/src/quorum.ts`, and `approver-set.ts` for set changes):

| Function                                                                          | Does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `createApproverKey()`, `approverKeyOf(publicKey)`                                 | A P-256 approver key pair, and its trust-bundle entry `{ kid, spkiDer }` with `kid` = base64url SHA-256 of the SPKI                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `batchSubject(body)`, `isBmcKind(kind)`                                           | The subject above, from a command body; `INVALID_INPUT` when a relayed kind's payload is not its request or names no target                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `batchScope(batchId, entries)`                                                    | The tree over `{ subject, kind, payloadSha256? }` entries: `root`, `size` and `proofOf(subject, kind, payloadSha256?)`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `verifyBatchProof({ batchId, subject, kind, payloadSha256?, proof, root, size })` | Walks a proof to the root; the size comes from the signed approval, never the command                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `signBatchApproval(approval, privateKey)`                                         | One approver's `Signature` over `{ batchId, scopeRoot, scopeSize, notAfter, trustDomain }`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `requiredApprovals(size, threshold, tiers)`                                       | The first tier whose `maxScopeSize` covers the batch, else the threshold                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `verifyBatchApproval(approval, options)`                                          | Unexpired, a 32-byte root, and enough valid signatures by distinct keys of the set (unknown kids, duplicates and a key listed under another's kid count for nothing); returns the approving kids                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `verifyBatchCommand(command, approval, proof, options)`                           | The agent's check: its own batch, an approval that holds, and its `(subject, kind)` in scope; `ACCESS_DENIED` with `details.reason` (`no-approval`, `other-batch`, `expired`, `too-few-approvals`, `not-in-scope`, ...)                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `approverSetOf(bundle)`                                                           | A `TrustBundle`'s quorum fields as an `ApproverSet` (`approverKeys`, `approverThreshold`, `quorumTiers`, `unbatchedDestructivePerHour`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `approverSetDigest(set)`, `sameApproverSet(a, b)`                                 | SHA-256 of the canonical set bytes (`approverSetInput`), and whether two sets are the same quorum (keys in any order)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `signApproverSet({ epoch, set, trustDomain }, privateKey)`                        | One current approver's `Signature` over a new set at one approver-set epoch of one deployment: purpose `approver-set`, parts `[uint64_be(epoch), approverSetDigest(set), trust_domain]` (4835f4f, c6214dc); an empty trust domain or an epoch below 1 is `INVALID_INPUT`                                                                                                                                                                                                                                                                                                                                                                                    |
| `verifyApproverSetChange(current, next)`                                          | Whether a holder of `current` (with its `epoch`) may take `next` (`next` carries `epoch`, `approverSetSignatures` and the verifier's own `trustDomain`, never the bundle's word for it): nothing is needed without current approver keys or with an unchanged set; otherwise `next.epoch` must be higher and the current threshold of distinct current keys must have signed. Returns the approving kids; `ACCESS_DENIED` with reason `stale-epoch`, `approver-set-unsigned`, `no-threshold` when the current threshold is 0, which admits no change, or `no-trust-domain`. On a refusal a daemon keeps its quorum and still applies the rest of the bundle |

`TrustBundleInput` (for `signJoinResponse` and `signTrustBundle`) takes `approverSetSignatures` (`{ kid, sig }`)
beside the quorum fields (6a7f1b7).

**Publishing the keys.** `pki.approvers` puts the quorum in the join-key-signed trust bundle (`JoinResponse.trust`
and `Welcome.trust`), so a gateway can relay it but not change it. The bundle's version follows the change by itself
(see [the version follows the content](#signed-intent)). The first set needs nothing more, since a daemon without
approver keys takes it as it comes (it only adds restrictions); any later change needs the current approvers (see
[changing the approver set](#changing-the-approver-set)):

```ts title="better-compute.config.ts"
export default defineConfig(() => ({
  pki: {
    // nodeCa and join as above
    commandSigning: bundle.commandSigning, // required with approvers
    approvers: {
      keys: [aliceSpkiDer, bobSpkiDer, carolSpkiDer], // P-256 SPKI DER, distinct
      threshold: 2, // 1 to the number of keys
      tiers: [{ maxScopeSize: 5, threshold: 1 }], // small batches need fewer approvals
      unbatchedDestructivePerHour: 0, // the default: unbatched destructive commands are break-glass only
    },
  },
}));
```

The options are validated at start (`INVALID_CONFIG`): `commandSigning` must be set, keys must be distinct P-256
public keys, thresholds whole numbers from 1 to the key count, tiers' `maxScopeSize` at least 1. Tiers must ascend
strictly by `maxScopeSize`, their thresholds must never decrease, and none may exceed `threshold` (which applies to
batches above every tier), so padding a scope with extra leaves never makes it cheaper to approve (d566cbe). A daemon cannot count
fleet-wide, so per-daemon allowances for unbatched commands would let a compromised worker hit every node once;
fleet-wide limits come from the approvers instead, for example a tier whose single signature comes from a
rate-limited auto-approver service outside the api and workers.

The [examples](/docs/operations/deployment) read the keys from `BETTER_COMPUTE_APPROVER_KEYS` (the `.pub` files of
`batch keygen`, comma-separated), with `BETTER_COMPUTE_APPROVER_THRESHOLD` (required with keys, 1 to their count)
and `BETTER_COMPUTE_UNBATCHED_PER_HOUR` (default 0), and refuse keys without a command-signing root in the bundle.

**Batches on the server.** The built-in `quorum` module, always hosted, owns migration `quorum/0001_batches`
(`command_batches`, `command_batch_scope`, `command_batch_members`, `command_batch_approvals`).
`createCommandBatch(tx, { tenantId, scope, reason, notAfterMs? }, { now, quorum })` fixes a batch at creation: its
root, size (at most 50,000 leaves) and every leaf's encoded `BatchProof` are stored, it ends after 24 hours by
default, and it needs `requiredApprovals` for its size (0 without approver keys). A module reaches it through the
command queue: `ctx.commands.createBatch(tx, { tenantId, scope, reason, notAfterMs?, createdBy? })` stores a batch
with the trust bundle's quorum and answers `{ batchId, held, required, size }`, and `ctx.commands.enqueue(tx, {
..., batchId })` puts a command into it. The batch must belong to the node's owner and be pending or approved and
unexpired, and the command's `(subject, kind)` (pki's `batchSubject`) must be in its scope (`INVALID_INPUT`
otherwise); `batch_id` goes into the signed body, a `command_batch_members` row is written, and while a quorum that
needs approvals has not approved the batch the command is held (`not_before` infinity, no poke). The commands API
shows it (a299e96): a batched command's `CommandInfo` carries its `batchId`, and a held one `awaitingApproval: true`
without `notBefore`, until the batch is approved. `enqueue` records
which leaf the command uses: the one bound to its payload's SHA-256 when the scope has it, else the unbound leaf.
Without approver keys a batch is recorded for audit and its commands go out at once.

With approver keys and no break-glass allowance (`unbatchedDestructivePerHour` 0), `enqueue` refuses a destructive
command without a `batchId` up front, since every daemon would refuse it `UNBATCHED_LIMIT`: `INVALID_STATE` with
`details: { reason: 'batch-required', kind }`, naming `commands.createBatch` and the break-glass option (508728e).
Destructive means the registry's `CommandKindSpec.destructive`, so plugin kinds count, and `node.reboot` is one. It is
an early warning only; the daemon stays the judge. With an allowance, or without approver keys, unbatched
destructive commands are enqueued as before.

The gateway sends a batched command only once it can prove it (40be696): when the batch needs approvals, dispatch
selects a member only after the batch is approved, joining the member to its scope leaf and the batch in SQL (so
held rows never crowd the dispatch limit), and sends it with `Command.batch_approval` and the member's own leaf's
`batch_proof`, decoded from the stored bytes. A `Hello` replay attaches them again.

#### Who creates batches [#who-creates-batches]

With approver keys set, every built-in producer of destructive commands puts them in a batch before it sends
anything, so none of them trips `batch-required`:

| Producer                     | Batch                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Metal workflows (f66e236)    | A workflow the reconciler starts outside a bulk batch gets one of its own: one node, the destructive leaves of the kinds it can send. `startWorkflow` records those kinds in `workflows.command_kinds` (every agent step's command, and every control-plane step's `sends`), and `workflows.batch_id` holds the batch (migration `provision/0006_workflow_batch`); the engine puts the batch id on the workflow's destructive commands only            |
| `metal.bulk` (f66e236)       | One verb (`manage`, `inspect`, `provide`, `undeploy`, `rescue`, `retire`) on up to 10,000 nodes of one pool as one batch, whose scope covers the verb's whole chain of state profiles (inspect then commissioning, for example). Decided on the pool and on every node as `metal/<verb>`; the batch is parked on each node (`metal_nodes.batch_id`, migration `metal/0005_batch`) until the node rests. See [the metal API](/docs/reference/api/metal) |
| `metal.power` (478025a)      | A one-leaf batch, `bmc:<site agent>/<address>` and `bmc.power`; the answer carries `batchId` and `held`                                                                                                                                                                                                                                                                                                                                                |
| Health remediation (e594a4f) | A reboot or GPU reset workflow gets a batch whose every leaf's subject is the node (`batchNodeWorkflow`); a workflow that relays `bmc.*` or `site.httpProxy` is refused there, since its owner must scope it                                                                                                                                                                                                                                           |
| MCP (171ad2f)                | `propose_command_batch` creates a `node.reboot` batch over up to 1,000 nodes; `exec_command` takes its `batchId`. There is no approve tool: an agent only proposes, and people approve with their approver keys                                                                                                                                                                                                                                        |
| Your own code                | `commandBatches.create` and `commands.run({ batchId })` over the API, or `ctx.commands.createBatch` and `enqueue({ batchId })` in a module                                                                                                                                                                                                                                                                                                             |

`metalBatchScope(tx, registry, nodeIds, kinds)` builds metal's scopes: the destructive kinds only, one leaf per node,
and for a `bmc.*` kind one leaf per site agent of the node's site, since a site agent failover reroutes the command.
The kinds come from `profileCommandKinds(profile, steps)` in provision, which relies on each control-plane step
declaring what it `sends`: a plugin step that relays commands (Talos's `httpViaSite`) must declare
`sends: ['site.httpProxy']`, or its commands fall outside the workflow's batch and are refused at enqueue
(`INVALID_INPUT`, or `batch-required` when the workflow has no batch at all; see
[steps and profiles](/docs/extending/steps-and-profiles)). Metal's leaves leave the payload unbound: the workflow
builds its payloads later (a BMC credential rotated mid-workflow, user data rendered by an earlier step).

Over the API, a caller creates a batch with `commandBatches.create` and runs commands in it with
`commands.run({ ..., batchId })` (46afe75). `create` takes `tenantId`, a `reason` shown to approvers (at most 500
characters), the `scope` leaves (`{ subject, kind, payloadSha256?, payload? }`, `payloadSha256` as hex, `payload` as
`{ typeUrl, value }` with `value` in base64; 1 to 50,000 of them),
`notAfterSeconds` (60 s to 72 h, default 24 h) and an `idempotencyKey`. The batch id derives from the tenant, the
caller's identity and the key, and migration `quorum/0005_batch_request` stores the request's digest, so a retry
returns the same batch and approvers never sign two, while the same key reused for another request is `CONFLICT`. A
new batch publishes `command.batch.created`. `commands.run` then checks the command's leaf against the scope and holds
the command until approval, as `enqueue` does; with approver keys configured, even `node.reboot` needs such a batch.
See [running commands](/docs/guides/nodes/commands).

The `commandBatches` group (phase v1.0), decided on the tenant:

| Method        | Action                    | Does                                                                                                                                                                                                                                                                                                                     |
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `create`      | `compute:batches:create`  | A batch over the given scope, answered like `get`; idempotent per caller and key                                                                                                                                                                                                                                         |
| `get`, `list` | `compute:batches:read`    | A batch's state (`pending`, `approved`, `expired`, `cancelled`), root, size, required approvals, the approvals so far and its commands by state, and who created it: `createdBy`, plus `createdByAgent` and `delegationId` when an agent proposed it for that person (below); `list` pages by `state`, `limit`, `cursor` |
| `scope`       | `compute:batches:read`    | The leaves (`leafIndex`, `subject`, `kind`, `payloadSha256`, empty for any payload, and the stored `payload` with `value` in base64 when its creator gave one), up to 5,000 a page, for approvers to review; check a payload against its digest before trusting it                                                       |
| `members`     | `compute:batches:read`    | The batch's commands, each with its node, subject, kind, state, `reason` and `awaitingApproval` while held; filter `state`, paged by `limit` and `cursor` (`better-compute batch members`, the console's batch view)                                                                                                     |
| `approve`     | `compute:batches:approve` | Takes one `signature` (`kid`, `sig`): it must verify alone for a known approver key over exactly this batch, is stored, and once the stored signatures meet the threshold the batch turns `approved`, its encoded `BatchApproval` is stored, its held commands become due and `command.batch.approved` is published      |
| `cancel`      | `compute:batches:approve` | Ends a pending batch and cancels its held commands                                                                                                                                                                                                                                                                       |

**An agent's proposal reads as the agent's.** When the caller of `commandBatches.create` is an agent acting through
a delegation (MCP's `propose_command_batch`, or a delegated API call), the batch records the agent and the delegation
beside the person in `createdBy` (`command_batches.created_by_agent` and `delegation_id`, migration
`quorum/0008_batch_agent`; f3c5c4e, security review 14 M3), since its `reason` is then the agent's text, not the
person's. `batch approve` notes "Proposed by agent X for Y (delegation D): the reason is the agent's own text"
before the approver signs, `batch show` answers the same words as `proposedBy`, and the console's Batches page shows
them as a warning line (bd80209); server strings pass through the CLI's printable filter first.

The IAM decision comes on top of the signatures, never instead of them. A pending batch past its end reads as
`expired`, and the `quorum.batches` reconciler records that every minute and cancels its held commands; a deleted
tenant's pending batches are cancelled.

**The approvers' side.** The `better-compute batch` commands (b3fd712, 6a7f1b7; flags in
[the CLI reference](/docs/reference/cli#quorum)) never trust the control plane's word for what a batch holds:

| Command                                                                       | Does                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batch keygen --out PATH`                                                     | A P-256 approver key (PKCS#8 PEM, mode 0600, never overwritten) and its public key at `PATH.pub`; prints the `kid`. With `BETTER_COMPUTE_APPROVER_PASSPHRASE` set, the private key is encrypted under it                                                                                                                                                                                                                      |
| `batch list --tenant ID [--state STATE]`                                      | `commandBatches.list`: reason, state, size, approvals had and needed, commands by state                                                                                                                                                                                                                                                                                                                                       |
| `batch show --tenant ID --batch ID [--scope-out FILE]`                        | Reads the batch, pages its scope and recomputes the root, like `approve`; `--scope-out` writes the verified scope with the batch's expiry and trust domain, the input of `batch sign`                                                                                                                                                                                                                                         |
| `batch approve --tenant ID --batch ID --key PATH`                             | Review, then sign and submit (below)                                                                                                                                                                                                                                                                                                                                                                                          |
| `batch approve ... --signature FILE`                                          | Submits a signature made offline by `batch sign`, after checking it signs this batch's id, root, size, expiry and trust domain (`SCOPE_MISMATCH` otherwise)                                                                                                                                                                                                                                                                   |
| `batch sign --key PATH --scope FILE`                                          | Signs offline from a `--scope-out` file (or a `{ subject, kind }` list): the same review, then prints `{ kid, sig }` with the signed fields. `--not-after` and `--trust-domain` default to the file's. Signing a bare `--root` and `--size`, which it cannot check, needs `--unchecked`                                                                                                                                       |
| `batch create --tenant ID --reason TEXT --scope FILE [--hours N] [--key KEY]` | `commandBatches.create` from a JSON array of `{ subject, kind, payload }` leaves (def72c8): a `payload` given as `{ json }` is encoded as the kind's own payload message, or given as `{ typeUrl, value }`, so the leaf binds exactly that operation and approvers see it; leaves without one are warned about. `--hours` is 1 to 72 (default 24); the idempotency key is printed, and repeating it retries the same creation |
| `batch members --tenant ID --batch ID [--state S] [--limit N] [--cursor ID]`  | `commandBatches.members`: the batch's commands with node, subject, kind, state and `awaitingApproval` (45d217e)                                                                                                                                                                                                                                                                                                               |
| `batch cancel --tenant ID --batch ID`                                         | `commandBatches.cancel`                                                                                                                                                                                                                                                                                                                                                                                                       |
| `batch sign-approvers --key PATH --epoch N ...`                               | Signs a change of the approver set for its epoch (see [changing the approver set](#changing-the-approver-set))                                                                                                                                                                                                                                                                                                                |

`batch approve` refuses a batch that is not `pending`, pages the scope (up to 5,000 leaves a page) and recomputes the
root from the listed `(subject, kind, payload digest)` leaves. Leaves that do not give the root and size the server
asks approvers to sign, or more leaves than the size, are `SCOPE_MISMATCH` ("the control plane answered two
different batches"); a page that does not advance, or an answer without a valid root and size, is
`INVALID_RESPONSE`. It then shows the reason (and, for an agent's proposal, who proposed it for whom), every subject by kind (at most 12 per kind, then a count), and the
approvals so far with the window's end, and signs the batch id, root, size, expiry and the deployment's trust domain.
The review is enforced, not only shown:

* Without `--yes` it stops after the summary and prints the root; `--root HEX` approves only if the batch still has
  the root you reviewed (`SCOPE_CHANGED` otherwise).
* `--expect FILE` (subjects one per line, a JSON array of subjects, or `{ subject, kind }` commands) approves only
  a batch that holds exactly those; `SCOPE_UNEXPECTED` names what is missing or not expected.
* Leaves that admit any payload are marked "any payload" and refused as `UNBOUND_SCOPE`, naming them, unless
  `--allow-unbound`.
* Bound leaves' operations are shown decoded, grouped by the payload's digest (each line shows its first 12 hex
  digits), with long strings shortened in the middle so every field shows; `--operations-out FILE` writes every
  distinct operation in full with its subjects (`batch sign` and `batch approve`; b118616). A payload with fields
  this build cannot decode, or whose re-encoding does not give the bytes back, is refused
  (`payload-unknown-fields`), and a kind whose payload type the CLI does not know is opaque and needs
  `--allow-opaque`; it is never decoded as the type the server labels it.
* A window that has closed, or that stays open longer than `--max-hours` (default 72, at most 720), is refused
  `INVALID_STATE`, since an approval admits its commands until then.
* An approver key readable by others is refused (POSIX; `INVALID_STATE`), as ssh does; an encrypted one needs
  `BETTER_COMPUTE_APPROVER_PASSPHRASE` (`MISSING_ENV`). A key that already approved the batch is reported and
  nothing is signed.

Once the batch holds the required approvals it is released and its commands run.

**On the machine.** bcd and the FakeDaemon (`intent.quorum`, or the quorum of its signed trust bundle) enforce it
the same way. With approver keys in the trust bundle, only kinds the daemon's own table marks destructive are
checked: one with a `batch_id` runs only with a `batch_approval` of enough distinct approver signatures (the tier for
its scope size, else the threshold, never fewer than one; a `kid` must be the SHA-256 of its key; duplicates count
once), not past its `not_after` in server time, and a `batch_proof` that proves the command's own `(subject, kind)`
leaf, the subject taken from the signed body; else it is `BATCH_UNAPPROVED`. Each leaf runs at most once, since an
approval for N machines is N operations, not N kinds of operation to repeat. Both daemons key a used leaf by the
hash of the leaf the proof matched, which holds the batch id, so another approval of the same batch id over a
different tree cannot move a used leaf to a fresh index (the FakeDaemon keyed it by batch id and leaf index until
c589c5d): another command on a used leaf is `BATCH_UNAPPROVED` (the FakeDaemon's reason is `leaf-used`), while a
redelivery or re-execution of the same command keeps its leaf. The FakeDaemon enforces this (d566cbe), and so does
bcd, which keeps the leaves that ran, with their command, in `data_dir/batch-leaves.json` until the approval's
`not_after` (8b0f50c). One without a `batch_id` counts against a rolling-hour allowance
(`unbatched_destructive_per_hour`, default 0), beyond which it is `UNBATCHED_LIMIT`; redeliveries do not count twice,
and bcd keeps the count in `data_dir/unbatched-destructive.json` so a restart does not reset it. The verdicts apply
after the signature check. bcd keeps the whole trust bundle in `identity.json`.

Both daemons also hold these lines (3ff8131 and 8b0f50c in bcd, 6a7f1b7 in the FakeDaemon):

* **A quorum change needs the current approvers.** Once a daemon holds approver keys, it refuses a change of any
  quorum field (keys, threshold, tiers or the unbatched cap; removal included) unless the current threshold of
  distinct current approvers signed the new fields, since the JoinService key alone could otherwise drop the
  approvers or swap in its own. A current threshold of 0 admits no change. Both apply one
  [epoch rule](#changing-the-approver-set) (c6214dc, 5fe7cbf, e171f0d): a refused set keeps the held quorum while
  the rest of the bundle (roots, version) applies; the FakeDaemon logs `trust.quorum-refused` with the reason. A
  daemon without approver keys takes the first set as it comes, or only its [pinned](#pinning-the-first-set) set,
  and its epoch never goes back. Both report the set they hold in `Hello` and `StatusReport` (`approver_set_epoch`,
  `approver_set_sha256`).
* **The unbatched window counts on the local clock**, so a `Welcome` that moves server time forward cannot age the
  count out. Approvals still expire in server time.
* **Nothing is used before admission.** A batch leaf or an unbatched slot is recorded only once the command is
  accepted and checked again at admission, so a forged, re-targeted or expired delivery carrying a public approval
  and proof cannot burn a leaf or the cap (the FakeDaemon's `claimQuorum`, review 6 L1); bcd also checks a deferred
  command's approval again when it is finally admitted.

bcd alone refuses destructive commands when its stored bundle cannot be read, rather than switching the quorum off.
bcd also treats a `site.httpProxy` other than a `GET` as a write (review 9): under a quorum it runs only under a leaf
bound to its exact payload, never an unbound one, and never unbatched, whatever the hourly allowance; readiness
polls, which are `GET`s, are unaffected.
Both count `bmc.boot` and `bmc.rotate` as destructive, as metal's kind table does, so they are signed and batched
like power actions. See [signed commands](/docs/daemon/delivery#signed-commands-and-attested-results).

#### Changing the approver set [#changing-the-approver-set]

The trust bundle is signed by the JoinService key, which the control plane holds, so on its own it must not be what
changes the quorum. Adding or removing an approver, changing a threshold, a tier or the unbatched cap, or removing
the quorum altogether therefore needs the **current** approvers' signatures over the new set (77cdf4f, 6a7f1b7,
edbd63f). Since c6214dc (review 7 M1) they sign the set's own **epoch**, not the trust bundle version:
`TrustBundle.approver_set_epoch` numbers the quorum fields, goes up with every change of the set, and stays put when
roots rotate. `approver_set_signatures` are purpose `approver-set` over `[uint64_be(epoch),
SHA-256(approverSetInput), trust_domain]`. Signing the version instead meant signatures were published only while
that version was current, so a daemon offline during a set change met the next bundle (a root rotation, say) with
the new set and no valid signatures, refused the whole bundle for good, and kept trusting rotated-out roots. The
trust domain is the verifier's own (4835f4f, review 7): a staging set's published signatures would otherwise be
replayable into production when the two share approver keys.

One rule governs every change (5fe7cbf and e171f0d, review 10 M3; the rule in `common.proto`): a daemon holding set
`H` at epoch `e` takes `(N, e')` only when `e' > e` and at least `H`'s threshold of `H`'s keys signed `(e', N, its
own trust domain)`. That holds for `N == H` too: an unchanged set moves to a higher epoch only with its own
signatures, and is otherwise kept silently at `e`. So a daemon that held `S@1` and missed `S -> T@2` and `T -> S@3`
cannot later be moved to `T@2` by replaying that epoch's genuine signatures. The same set at the same epoch needs
nothing; anything else is `stale-epoch` or `approver-set-unsigned`. `approver_set_signatures` may mix the previous
set's signatures (the change) with the new set's own (which move daemons that already hold the new set to the new
epoch); only held keys count. A daemon without keys takes the first set as it comes, or only its pinned one (below),
never at or below an epoch it held, and keeps its epoch through a signed removal. When the quorum part is refused,
the daemon keeps its quorum and still applies the rest of the bundle (roots, version). A daemon that missed two set
changes whose keys do not overlap cannot verify the newer set with the keys it holds and needs re-provisioning (or
re-pinning). The first set needs no signatures. On a fleet whose daemons already hold approver keys:

      ##### Pick the epoch [#pick-the-epoch]

    The new set's epoch is the published one plus 1. With the new quorum in the configuration,
    [`better-compute pki trust-version`](#pki-trust-version) prints it as `signQuorumForEpoch`, and doctor's
    `pki.quorum-unsigned` finding names it too. Any higher epoch works as well; an older one is never taken back.
  
      ##### Each current approver signs [#each-current-approver-signs]

    At least the current threshold of current approvers each run, with their own current key and the whole new set.
    Members of the new set may sign too: their signatures move daemons that already hold the new set to its epoch, and
    the command says when a key counts only as a current approver:

    ```bash
    better-compute batch sign-approvers --key alice.pem --epoch 2 \
      --approvers alice.pem.pub,bob.pem.pub,dave.pem.pub --threshold 2 --tiers 5:1 --unbatched 0
    ```

    It validates the set (P-256 public keys from `batch keygen`, no key twice, a threshold from 1 to the key count, tiers
    ascending by size with thresholds that never decrease), says what it signs (a quorum of k of n with
    their kids, or, with no `--approvers` and `--threshold 0`, the REMOVAL of the quorum), notes when your own key is
    not in the new set, and prints `{ epoch, trustDomain, setSha256, set, signature: { kid, sig } }`. The trust domain
    is `--trust-domain`, else the PKI bundle's (`--pki`); without either it is a usage error. The examples refuse
    signature files made for another trust domain.
  
      ##### Publish the set with its signatures [#publish-the-set-with-its-signatures]

    Set the new quorum, its `epoch`, and the collected signatures as `pki.approvers.setSignatures`:

    ```ts title="better-compute.config.ts"
    pki: {
      commandSigning: bundle.commandSigning,
      approvers: {
        keys: [aliceSpkiDer, bobSpkiDer, daveSpkiDer],
        threshold: 2,
        tiers: [{ maxScopeSize: 5, threshold: 1 }],
        epoch: 2, // default 1; required with setSignatures: the epoch they sign
        setSignatures: [aliceSig, bobSig], // the { kid, sig } of sign-approvers (sig base64url or bytes)
      },
    },
    ```

    `pki.approvers.epoch` is a whole number of at least 1 (default 1), `setSignatures` without it is `INVALID_CONFIG`,
    and each signature must be `{ kid, sig }` with a 64-byte P-256 signature. The bundle's version still follows its
    content; `JoinResponse.trust` and `Welcome.trust` carry `approver_set_epoch` and the set's signatures on every
    version while the set is current, so a later root rotation keeps them and a daemon that missed a version still
    verifies the set. `trust_bundle_state` records the published set with its epoch.
  
A quorum change without valid signatures for a higher epoch is still published, since daemons without approver keys
take it, but each process logs a warning naming `batch sign-approvers --epoch <published + 1>`, and daemons that
hold keys keep their old quorum until a signed set reaches them. To fix it, have the current approvers sign that
epoch and set `pki.approvers.epoch` and `setSignatures`. `better-compute doctor` verifies `setSignatures` as a
daemon does (the published set's threshold, the epoch, the deployment's trust domain), so signatures that do not
verify are `pki.quorum-unsigned` with the reason (b118616).

**Nodes that hold another set.** A daemon can take a bundle's version while refusing its quorum part, so the trust
version alone does not show which approvers a node enforces. The gateway records the set each daemon reports holding
in `node_sessions.approver_set_epoch` and `approver_set_sha256` (migration `core/0026_session_approver_set`,
fa39201):

* `Hello` writes the reported set as it comes.
* A `StatusReport` with `trust_version` above 0 (the acknowledgement of a newer bundle) writes it only when that
  version is at least the one recorded, so a late, older report never pairs an old set with a newer version.
* A daemon that holds no approver keys (an empty digest) is recorded as `NULL` in both columns.

`better-compute doctor` and [`pki trust-version`](#pki-trust-version) read those columns for the connected nodes
(a session on a gateway, not ended, seen within the last 2 minutes) and compare them with the published set
(6612496, 08eb62e). While the published bundle has approver keys, a node lags when its digest is `NULL` (it
enforces no quorum at all, the worst lag), differs from the published set's, or its epoch differs. Without published
keys, any node that still holds a set lags. Doctor reports `pki.quorum-lag` (a warning, only when a published
approver set exists) as "N of M connected node(s) enforce another approver set than the published one (epoch E)",
followed by up to five nodes with the epoch each holds (0 for a node without keys). `pki trust-version` prints the
count as `approverSetBehind` and those nodes as `approverSetSample` (`nodeId`, `epoch`). Both leave the check out
before `core/0026` has run. The fix is to check the set's signatures, or to re-pin nodes that missed several
changes.

The examples read the signatures from `BETTER_COMPUTE_APPROVER_SET_SIGNATURES`: the JSON files `batch sign-approvers`
printed, comma-separated. They must all sign one epoch and exactly the quorum `BETTER_COMPUTE_APPROVER_KEYS`,
`BETTER_COMPUTE_APPROVER_THRESHOLD` and `BETTER_COMPUTE_UNBATCHED_PER_HOUR` configure (the examples set no tiers), or
the example refuses to start; that epoch becomes `pki.approvers.epoch`. `BETTER_COMPUTE_TRUST_VERSION` sets the
trust bundle version's floor, independently.

#### Pinning the first set [#pinning-the-first-set]

A daemon without approver keys (every new or reimaged node) takes the first set its join-key-signed bundle carries,
so a compromised api could hand it an empty set or its own key at k = 1, after which the operators' real set would
need the attacker's signature. The first set can be pinned outside the api (9b7a199, review 7 M2):
`better-compute pki trust-version` prints `approverSetSha256` (SHA-256 of `approverSetInput`), computed from this
configuration's approver keys rather than from the database the api writes, with `publishedApproverSetSha256` beside
it while the two differ (b118616, review 10 M4), and `daemon install-script --approver-set-sha256 HEX` writes it into
the node's `config.toml` as `approver_set_sha256` (an upgrade run without the flag keeps an existing pin, b6d401f).
A daemon with that pin and no keys takes only a matching set (`pin-mismatch` otherwise, while the rest of the bundle
still applies) and refuses destructive commands `BATCH_UNAPPROVED` (the FakeDaemon's reason is
`awaiting-pinned-set`) until it holds one, so it fails closed. bcd (`daemon/internal/pki/batch.go`,
`agent/quorum.go`) and the FakeDaemon (`intent.approverSetSha256`) both implement it.

#### Turning the quorum on [#turning-the-quorum-on]

PLAN.md §13 records what happens when a deployment upgraded from v0.5 sets `pki.approvers` for the first time, and
`tests/sim/upgrade.test.ts` proves it (80195df): no allocation moves, no VM reboots, and reservations stay active.

      ##### Upgrade first, then set the approvers [#upgrade-first-then-set-the-approvers]

    Run the v1.0 upgrade as usual (the quorum migrations as expand migrations, then workers, gateways and daemons), then
    set `pki.approvers` and restart the workers and roll the gateways again. Setting approvers moves no NodeSpec
    generation.
  
      ##### Cancel pending destructive commands [#cancel-pending-destructive-commands]

    A destructive command queued before the switch is still sent afterwards, but a daemon that holds the quorum by then
    rejects it `UNBATCHED_LIMIT`, so it never runs. To have nothing destructive in flight at the switch, cancel pending
    destructive commands first (`commands.cancel`).
  
      ##### Let the bundle roll out [#let-the-bundle-roll-out]

    The trust bundle's version goes up exactly once when approvers are set. Daemons take the first set without
    signatures, or only the set they were pinned to; check with `pki trust-version` that none is behind.
  
      ##### Batch destructive work from now on [#batch-destructive-work-from-now-on]

    A new unbatched destructive command is refused at enqueue (`INVALID_STATE`, reason `batch-required`) unless
    `unbatchedDestructivePerHour` allows it. The built-in producers batch their own work (see
    [who creates batches](#who-creates-batches)).
  
## Exec tickets and the SSH CA [#exec-tickets-and-the-ssh-ca]

The exec ticket key signs ES256 JWTs (`typ: bc-exec+jwt`) that live at most 60 s, name one node as audience
(`aud: node:<id>`), and carry a single-use `jti`. `tunnels.open` issues them after Better IAM allowed the call; the
gateway and then the daemon verify each ticket against the JWKS in the node's NodeSpec. Every api replica and the
gateway derive the same `kid` from the key, and `rotate()` keeps the previous key listed.

The SSH CA issues OpenSSH user certificates for break-glass access, valid at most 1 h and never beyond the
requesting session, with the principal `<account>@<nodeId>` so a certificate opens only the node it names. Every
issuance is a row in `ssh_certificates`; `nodes.revokeSshCert` adds the serial to the node's key revocation list in
its next NodeSpec. See [SSH](/docs/guides/access/ssh) and [tunnels](/docs/guides/access/tunnels).

## The sandbox SSH CA [#the-sandbox-ssh-ca]

The bundle's `sandboxSshCa` (v0.5) is a second OpenSSH CA, used only by `sandboxes.exec`. It signs user certificates
of 60 seconds to 15 minutes, never past the caller's session or the sandbox's expiry, for the one principal
`<account>@<instanceId>` (for a sandbox restored from a memory snapshot, the instance the memory was captured on,
since the resumed guest keeps that principal). It is kept apart from the break-glass CA so neither certificate opens
what the other is for: nodes never trust the sandbox CA, and sandbox guests never trust the break-glass CA.

* **Signing.** Pass it as `tunnels.sandboxSsh` (`tunnelsOptions(bundle, …)` does). Without it, `sandboxes.exec` is
  `UNSUPPORTED`. The tunnels module refuses a `sandboxSsh` whose key is the break-glass `ssh` CA's
  (`INVALID_CONFIG`: "tunnels.sandboxSsh must be a different CA from tunnels.ssh"), so the split cannot be undone by
  configuration.
* **Trust.** Pass its public key as `vps({ sandboxExec: { caPublicKey } })`. vps writes it into every sandbox
  instance's user data as `TrustedUserCAKeys`, with an `AuthorizedPrincipalsFile` holding that instance's own
  principal. vps and the tunnels module each refuse a key that is not one line (an ed25519 or ECDSA P-256 key for
  vps), since a second line would be a second trusted CA; the bundle loader refuses one too.
* **Older bundles.** `pki add-sandbox-ca` adds it; `pki show` prints it as `sandboxSshCaPublicKey`.
* **Revocation.** Guests have no agent and receive no revocation list, so the short lifetime and tunnel
  re-authorization are the controls. See [sandboxes](/docs/guides/vps/sandboxes#run-code-inside).
* **Host certificates.** Every `SshCertificateAuthority` can also sign OpenSSH host certificates:
  `issueHostCertificate({ publicKey, principals, keyId, validBefore })` returns `{ certificate, serial, validBefore }`
  with no options or extensions, for principals that are plain names (letters, digits, `.`, `_`, `-`; never a list
  or a wildcard) and a lifetime of at most `maxHostTtlSeconds` (default and maximum 30 days). Wire the sandbox CA's
  `issueHostCertificate` into `vps({ sandboxExec: { certifyHostKey } })` and sandbox guests present host keys it certified, which
  `sandbox exec` pins with `@cert-authority` instead of accepting them on first use; see
  [how the guest trusts it](/docs/guides/vps/sandboxes#how-the-guest-trusts-it).

## The audit key [#the-audit-key]

With the bundle's `auditSigning` key, `auditModule({ signer })` seals `audit_events` into signed, hash-chained,
per-tenant Merkle checkpoints every hour. `better-compute audit verify --tenant ID` checks a tenant's trail against
the key's pin, taken from `--pin`, `BETTER_COMPUTE_AUDIT_PIN` or the bundle's public half, and exits
`AUDIT_TAMPERED` on any problem. See [audit](/docs/guides/events/audit).

## Workload identity [#workload-identity]

Workload identity (v0.5, optional) gives hosted VPS instances short-lived OIDC tokens. Its key is not part of the
bundle: `createWorkloadIssuer({ issuer, defaultAudience, keys? })` from `@better-compute/pki` holds its own P-256
keys, and you pass the issuer as `betterCompute({ workloadIdentity: { issuer } })`.

* **Tokens.** ES256 JWTs with `iss`, `sub = "t:<tenantId>:instance:<instanceId>"`, `aud` (the default audience
  unless the request names an allowed one), `iat`, `nbf`, `exp`, `jti`, `tenant_id` and `instance_id`, living 60 to
  900 seconds. Extra claims may not shadow those.
* **Publication.** The issuer URL must be https on port 443, without query. `better-compute serve` answers
  `GET` and `HEAD` on `<issuer path>/.well-known/openid-configuration` and `<issuer path>/.well-known/jwks.json`,
  public, cached 5 minutes, with `access-control-allow-origin: *`; the JWKS is read per request, so an added key
  shows at once. The issuer URL has to reach that host.
* **Rotation**, in three steps so no token ever fails verification: `addKey()` publishes a new key while the old one
  still signs; once verifiers have re-read the JWKS, `promote(kid)` signs with it; `retire(kid)` drops the old key,
  refused until its last token has expired (15 minutes plus a minute after it stopped signing).
  `exportKeys()` returns the keys as PKCS#8 PEM with their state, for storage; handle them as secrets.

The host asks for a token over its session with a `WorkloadTokenRequest` signed by its node key (purpose
`workload-token`), and the answer is sealed to a one-time key (JWE `typ: bcworkload+jwe`, binding the request id,
the instance and the expiry). `createWorkloadTokenRequest` and `openWorkloadTokenResponse` are the host's half,
`openWorkloadTokenRequest`, `sealWorkloadTokenResponse` and `refuseWorkloadTokenRequest` the broker's. bcd carries
the host's half in Go (`NewWorkloadTokenRequest` and `OpenWorkloadTokenResponse` in `daemon/internal/pki`, checked
against fixtures the TypeScript side produces), and its metadata service (`daemon/internal/imds`) calls it for the
instances of whichever runtime bcd runs; neither runtime of host builds (`libvirt-kvm`, `cloud-hypervisor`) has run on real hardware yet,
so only fakehw builds' instances have been served end to end. What the broker checks is in
[VPS instances](/docs/guides/vps/instances#workload-identity).

## The workload issuer for provider accounts [#the-workload-issuer-for-provider-accounts]

A second, separate issuer makes the control plane itself an OIDC identity provider, so a tenant's GCP account can
be keyless: the tenant's workload identity pool trusts the control plane, and each provider account mints tokens for
one subject, `compute-account:<tenantId>/<provider>/<name>`, which vps derives from the account row (see
[keyless GCP accounts](/docs/guides/vps/providers#keyless-gcp-accounts)). Without it,
`providerAccounts.create` with `federationAudience` and `providerAccounts.federationSetup` are `INVALID_CONFIG`
(`reason: 'noWorkloadIssuer'`).

```ts
import { createPrivateKey } from 'node:crypto';
import { workloadIssuer } from '@better-compute/pki';

// PEM keys from the secret store, the same on every replica: never generated at startup.
const key = (pem: string) => createPrivateKey(pem);

betterCompute({
  baseURL: 'https://compute.example.com',
  // basePath defaults to /api/compute
  pki: {
    workloadIssuer: workloadIssuer({
      issuer: 'https://compute.example.com/api/compute/workload',
      signingKey: key(secrets.workloadSigningKey),
      // During a rollover only (see below):
      // nextKey: key(secrets.workloadNextKey),
      // previousKeys: [key(secrets.workloadPreviousKey)],
    }),
  },
  // …
});
```

* **The issuer URL** must be exactly `<public origin>{basePath}/workload`: the host serves the discovery document at
  `{basePath}/workload/.well-known/openid-configuration` and the keys at `{basePath}/workload/jwks.json`, public,
  unauthenticated and cached for 5 minutes (`cache-control: public, max-age=300`), and Google fetches both from the
  issuer URL itself. The origin must be `baseURL`'s. When the issuer is reached at another public origin (a tunnel,
  an API host split from the one browsers use), name it with `pki.workloadIssuerOrigin`, e.g.
  `'https://compute-public.example.com'`; the issuer's origin must then equal that instead. https is required (http
  only for `localhost` and `127.0.0.1`). `betterCompute()` refuses anything else with `INVALID_CONFIG`
  (`reason: 'workloadIssuerUrl'`), as it refuses an issuer URL equal to the v0.5 `workloadIdentity.issuer`'s, whose
  discovery document would be served at the same place.
* **The keys** are EC P-256 private keys. They must be persistent and identical on every replica: load them from the
  secret store at start. A key generated at startup changes on every restart and differs between replicas, so tokens
  one replica signs fail verification against the keys another replica published, and every tenant that uploaded
  keys is cut off at the next restart. Handle the PEMs as secrets; only public keys ever appear in `jwks.json`.
* **Tokens** are ES256 JWTs living 60 to 600 seconds (300 by default), with `iss`, `sub`, `aud`, `iat`, `exp` and
  `jti`. They are exchanged at Google STS at once and never logged.

### Rolling the issuer's key over [#rolling-the-issuers-key-over]

Tenants whose pool fetches the keys from `jwks.json` follow a rollover on their own, but tenants whose control plane
Google cannot reach uploaded a copy of the keys, and Google accepts no token signed by a key it was not given. So
the next key is published beside the current one well before it signs, and the rotation is announced:

1. **Publish the next key.** Add the new key as `nextKey` and roll that configuration out to every replica.
   `jwks.json` now lists both keys; the new one never signs. Announce the rotation to tenants with uploaded keys:
   they download `jwks.json` again and update their provider as soon as it lists the new key.
2. **Wait.** At least 30 days, so every tenant with uploaded keys has re-uploaded them, and in any case longer than
   the 5-minute `Cache-Control` of `jwks.json` plus Google's own JWKS caching.
3. **Promote.** Make the new key `signingKey` and move the old one to `previousKeys`, on every replica. Keep it there
   for at least the 600-second maximum token lifetime, so tokens signed just before the switch still verify.
4. **Drop the old key** from `previousKeys`. Tenants with uploaded keys may remove it from their provider too.

## Rotation and expiry [#rotation-and-expiry]

| What                                                | How it is replaced today                                                                                                                                              |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node certificates                                   | Automatically: daemons renew at about 60% of their 24 h lifetime                                                                                                      |
| Worker signing keys                                 | Automatically, at 60% of their 24 h lifetime                                                                                                                          |
| Gateway serving certificates                        | `pki issue-gateway` before the 30 days run out; gateways with `host.gateway.tlsFiles` hot-reload it, others need a restart one at a time                              |
| Command-signing root                                | `pki rotate-command-root`, the three steps above                                                                                                                      |
| Exec ticket key                                     | `rotate()` on the issuer; the previous key stays in the JWKS                                                                                                          |
| SSH CA, sandbox SSH CA                              | Not automated. A new sandbox CA reaches only instances created after it, since the key is in their user data                                                          |
| A bundle sealed in the former format                | Once: `pki reseal` prints its pins, then `pki reseal --accept-unauthenticated` after comparing them (see [pki reseal](#pki-reseal))                                   |
| Workload issuer keys                                | `addKey()`, `promote(kid)`, then `retire(kid)` once the old key's tokens expired                                                                                      |
| Provider-account issuer keys (`pki.workloadIssuer`) | Not automated: `nextKey`, at least 30 days, then `signingKey` with the old key in `previousKeys` (see [rolling the issuer's key over](#rolling-the-issuers-key-over)) |
| Sealed secrets                                      | `localSealedSecretStore` with several keys, then `reseal()` (see [configuration](/docs/operations/configuration#the-deployment-secret))                               |
| Node, serving and internal CAs (1 year)             | `pki renew-ca --ca node` (or `serving`, `internal`) before they expire (`doctor` warns 30 days ahead); see [pki renew-ca](#pki-renew-ca)                              |
| JoinService keys                                    | Not automated. Daemons pin them, so new keys mean new pins in every machine's configuration.                                                                          |

> **Losing the bundle.** 
  A bundle that is lost, or that no longer opens because the secret changed, cannot be recreated: a new
  `pki init` makes new CAs and new pins, and every node has to rejoin. Back up the bundle and the secret, and keep the
  offline root key apart from both.



# Retention (/docs/operations/retention)

> What the retention passes prune from PostgreSQL and when, how cursors hold them back, how history partitions work, and what is kept on purpose.



Most of Better Compute's tables hold current state and stay the size of your fleet. A few are logs that grow with
every change: the transactional outbox, the authz change feed, the IAM inbox, replay guards
for attestations, commands, inventory snapshots, audit rows, usage windows and webhook deliveries. This page says
which of them are pruned, by what, with which defaults, and which are kept on purpose.

| Table                 | Grows with                                      | Pruned by                   | Default                                                                        |
| --------------------- | ----------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------ |
| `outbox`              | every event                                     | `retentionModule()`         | behind every sink cursor, and older than 1 h                                   |
| `authz_changes`       | every change to the authz projection            | `retentionModule()`         | behind every cursor of the stream; with no cursor saved, rows older than 1 day |
| `commands`            | every command                                   | `retentionModule()`         | settled commands last updated more than 30 days ago                            |
| `attestation_uses`    | single-use attestations                         | `retentionModule()`         | once `expires_at` passed; rows without expiry are kept                         |
| `join_requests`       | deferred joins                                  | `retentionModule()`         | marked `expired`, not deleted                                                  |
| `inventory_snapshots` | every inventory report                          | `retentionModule()`         | superseded snapshots older than 180 days; empty past months dropped            |
| `usage_windows`       | hourly per allocation and meter                 | `retentionModule()`, opt-in | kept unless `history.usageOlderThanDays` is set                                |
| `webhook_deliveries`  | webhook deliveries                              | `events.prune`              | delivered and cancelled after 30 days, dead after 90                           |
| `iam_inbox`           | IAM audit events                                | the IAM inbox job           | processed rows older than 7 days                                               |
| `work_queue`          | dirty keys                                      | the runtime                 | a key is deleted when its run succeeds                                         |
| `workers`             | worker processes                                | the runtime                 | rows of workers long gone                                                      |
| `audit_events`        | every allowed mutation, join and secret release | nothing                     | kept; sealed by checkpoints; archived by hand                                  |
| `outbox_dead_letters` | poison events of event sinks                    | nothing                     | kept                                                                           |

## The retention module [#the-retention-module]

`retentionModule(options?)` from `@better-compute/postgres` is the store's housekeeping, one singleton
reconciler named `core.retention` with a periodic key. It runs under the `controllers`
worker role, in short transactions and bounded batches, so a pass never holds long locks.

The server cannot import the store adapter, so it cannot register the module itself. The CLI does it for you:
deployments built by `serve`, `dev` and `migrate` get `retentionModule()` unless the configuration registers a module
named `retention` already. `createDevStack()` registers it too. If you build `betterCompute()` yourself, add it:

```ts
import { retentionModule } from 'better-compute/postgres';

betterCompute({ database, modules: [retentionModule({ every: '10m' }), ...modules] });
```

| Option                           | Default | Meaning                                                                                                                                                                                           |
| -------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `every`                          | `5m`    | How often a pass runs                                                                                                                                                                             |
| `minAgeSeconds`                  | 3600    | Outbox and `authz_changes` rows stay at least this long after every cursor passed them                                                                                                            |
| `batch`                          | 5,000   | Rows per statement; a job repeats its statement until it deletes fewer                                                                                                                            |
| `maxRounds`                      | 20      | Most statements per job in one pass, so a large backlog drains over several passes                                                                                                                |
| `authzChangesMaxAgeSeconds`      | 86,400  | While no reader saved an `authz_changes` cursor, rows older than this go (at least 3,600); `false` keeps them                                                                                     |
| `commandsOlderThanDays`          | 30      | Settled commands last updated this long ago go (at least 1); `false` keeps them                                                                                                                   |
| `history.monthsAhead`            | 2       | Month partitions kept created ahead of the current month                                                                                                                                          |
| `history.inventoryOlderThanDays` | 180     | Superseded inventory snapshots older than this go; `false` keeps them all                                                                                                                         |
| `history.usageOlderThanDays`     | unset   | Recorded or voided usage windows that ended this long ago go (an allocation's once the allocation ended that long ago too; at least 7); a window with an open dead letter stays; unset keeps them |

A pass runs these jobs in order and returns how many rows each touched:

1. **History partitions**: `ensureHistoryPartitions(tx, { monthsAhead })` creates the month partitions from the
   current month through `monthsAhead` ahead.
2. **`outbox`**: `pruneBehindCursors(tx, 'outbox')`.
3. **`authz_changes`**: `pruneBehindCursors(tx, 'authz_changes')`, then `pruneUncursoredAuthzChanges(tx)`, which acts
   only while the stream has no saved cursor.
4. **`commands`**: `pruneSettledCommands(tx)` deletes commands in a settled state (`succeeded`, `failed`,
   `rejected`, `expired`, `cancelled`) last updated more than `commandsOlderThanDays` ago, through the
   `commands_settled` index (`core/0020_command_created_at`). Open commands are never pruned.
5. **`attestation_uses`**: `pruneAttestationUses(tx)` deletes rows whose `expires_at` has passed.
6. **`join_requests`**: `expireJoinRequests(tx)` marks `pending` and `approved` requests past their expiry as
   `expired`, which frees the one-pending-request-per-machine index for a new request.
7. **`inventory_snapshots`**: `pruneInventorySnapshots(tx)`, unless `inventoryOlderThanDays` is `false`.
8. **`usage_windows`**: `pruneUsageWindows(tx)`, only when `usageOlderThanDays` is set.
9. **Dropping emptied months**: `dropEmptyHistoryPartitions` for `inventory_snapshots` and `usage_windows`, each in
   its own short transaction.

When a pass touched anything it logs `retention pass` at info level with the counts. To schedule passes yourself,
call `runRetention(sql, options)`; it returns `{ outbox, authzChanges, commands, attestationUses,
joinRequestsExpired, partitionsCreated, inventorySnapshots, usageWindows, partitionsDropped }`.

The module also has a `selfCheck`, so [`doctor`](/docs/operations/observability#doctor) warns when retention is not
keeping up. `retentionFindings(sql, { staleCursorSeconds? })` returns the same findings for your own checks:

| Check                       | Warns when                                                                            | Fix                                                                              |
| --------------------------- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `history-partitions`        | this or next month's partition of a history table is missing: no pass is running      | register `retentionModule()` on a worker, or run `ensureHistoryPartitions`       |
| `history-default-partition` | rows of the last 13 months sit in a default partition                                 | let a pass create their months, which moves them out                             |
| `retention-cursors`         | a durable `outbox_cursors` row has not moved for a day (`staleCursorSeconds`, 86,400) | revive the sink or reader that owns it, or delete the row if it is gone for good |

> **Pruned commands free their idempotency keys.** 
  A command's idempotency key is unique per node, so a retry with the same key finds the old row. Once a settled
  command is pruned, its key is free again. Keep `commandsOlderThanDays` longer than any caller retries a key.

## Cursors decide what goes [#cursors-decide-what-goes]

The outbox and the authz change feed are read by cursor: a `(txid, seq)` position that only moves forward. Readers
that must not miss a row save their position in `outbox_cursors` (`core/0009_runtime`), one row per reader and
stream. Retention deletes only rows that **every** saved cursor of that stream has passed, and only when they are
older than `minAgeSeconds`, measured with `compute_now()`.

<Mermaid
  chart="flowchart LR
  A[&#x22;oldest rows&#x22;] --> F[&#x22;floor: the lowest saved cursor&#x22;] --> B[&#x22;rows a reader still needs&#x22;] --> H[&#x22;newest rows&#x22;]
  F -. &#x22;deletable when older than minAgeSeconds&#x22; .-> A"
/>

* **An outbox with no saved cursor is never pruned.** A reader that has not saved a position yet could need
  everything.
* **Outbox cursors come from event sinks.** Each sink in the `sinks` worker role saves its cursor as `sink:<id>` after
  every delivered batch, and a save never moves a cursor back. SSE watchers, `events.list` callers and the webhook
  fan-out cursor do not hold retention back; a client that resumes from a pruned position continues with the oldest
  row still there. A deployment without sinks saves no outbox cursor, so it keeps its whole outbox. See
  [events](/docs/guides/events).
* **No built-in reader saves an `authz_changes` cursor.** The IAM bridge's projection tails the feed with a cursor it
  keeps in memory and reloads the whole projection on start, so only a live tail more than a day behind could miss a
  pruned row. While no cursor is saved, rows older than `authzChangesMaxAgeSeconds` (one day) that committed before
  every open transaction go. If you add a durable reader, save its position with
  `saveCursor(tx, { name, stream: 'authz_changes', txid, seq })`: from then on only the cursor rule prunes the stream.
* **A stopped reader holds everything back.** A cursor row that no longer moves keeps every row after it. When you
  retire a sink or reader, delete its `outbox_cursors` row. `doctor` reports such a row as `retention-cursors` once it
  has not moved for a day.

`saveCursor` and `loadCursor` are exported from `@better-compute/postgres`; `txid` is passed as text, as readers
alias it (`txid_text`). See [database](/docs/operations/database) for the runtime tables.

## History partitions [#history-partitions]

`core/0018_partitioned_history` range-partitions `audit_events`, `inventory_snapshots` and `usage_windows` by UTC
month on `occurred_at`, `collected_at` and `window_start`. Month partitions are named `<table>_pYYYYMM`, and each
table has a `DEFAULT` partition, so an insert never fails for want of a month (a simulated clock, a skewed row).
Primary keys include the partition column: `(id, occurred_at)`, `(id, collected_at)`,
`(idempotency_key, window_start)`.

* **Created ahead.** Each pass creates the current month and the next `monthsAhead`, moving any rows of those months
  out of the default partition first (`compute_history_partition(parent, at)`, serialized by an advisory lock).
  `defaultPartitionRows(tx, table)` counts what the default partition still holds.
* **Pruned by row.** `pruneInventorySnapshots` deletes snapshots older than the window that a newer snapshot of the
  same node superseded; a node's newest snapshot and the one its inventory is applied from always stay.
  `pruneUsageWindows` deletes only windows that were recorded in billing, of allocations that ended more than the
  window ago; unrecorded windows always stay, and the minimum of 7 days leaves metering's 48 h lookback intact.
* **Dropped when empty.** `dropEmptyHistoryPartitions` detaches and drops month partitions of `inventory_snapshots`
  and `usage_windows` that are empty and ended more than the window ago. The default partition and current or future
  months are never dropped. It refuses `audit_events`.
* **Archived by hand.** `detachHistoryPartition(tx, table, name)` detaches one past month, leaving a standalone
  table to export and drop. It refuses the default partition and the current or a future month.
  `historyPartitions(tx, table?)` lists what exists.

## Attestation replay guards [#attestation-replay-guards]

`attestation_uses` (`core/0011_v02_security`) records evidence an attestor admits only
once, such as a cloud instance identity document or a site agent's discovery assertion, keyed by the attestor and the
SHA-256 of the evidence. Attestors insert the row in the join transaction of the final round, so a deferral does not
burn it, and a second join with the same subject is refused. Evidence that can no longer be presented needs no
guard, so the pass deletes rows once their `expires_at` passes. A row with no `expires_at` stays forever: that is how
an attestor says "never again". See [attestation](/docs/guides/nodes/attestation).

## Webhook deliveries [#webhook-deliveries]

`eventsModule()` prunes its own delivery history with a third reconciler, `events.prune` (hourly, migration
`events/0002_delivery_retention` adds the index it uses). It deletes delivered and cancelled deliveries 30 days after
they were created and dead letters after 90, in batches of 5,000, running again a second later while a batch
comes back full. Pending deliveries are never deleted, whatever their age.

```ts
eventsModule({ secret, retention: { deliveredDays: 14, deadDays: 60 } }); // or retention: false to keep everything
```

A period under one day is `INVALID_CONFIG` when the module is defined. `pruneWebhookDeliveries(sql, retention?,
limit?)` is exported for hosts that schedule it themselves. See [webhooks](/docs/guides/events/webhooks).

## What else cleans up after itself [#what-else-cleans-up-after-itself]

* **IAM inbox.** The IAM inbox job (`drainIamInbox`, worker role `iam`) marks rows processed and deletes processed
  rows older than `retainMs`, default 7 days (0 keeps them).
* **Work queue.** A successful run deletes its `work_queue` key only when the key's generation is still the one it
  claimed, so a key dirtied again mid-run survives and no wake-up is lost.
* **Workers.** Each worker heartbeats its `workers` row; rows of workers gone long after their shards moved are
  deleted, and a stopping worker deletes its own.
* **The daemon's journal.** On the node, done command results are kept 7 days, and the journal is capped at 64 MiB
  and 10,000 unacked events. See [delivery](/docs/daemon/delivery).

## What is kept on purpose [#what-is-kept-on-purpose]

**Audit rows and Merkle checkpoints.** `audit_events` records every allowed mutation with its admission context, plus
joins and secret releases. With `auditModule()` from `@better-compute/audit`, an hourly
`audit.checkpoint` pass seals new rows into signed per-tenant Merkle checkpoints, following the `(txid, seq)` commit
order. `verifyAuditCheckpoints` recomputes every root and size **against the rows as they are now**, so a deleted
row is reported as tampering, exactly like an edited or inserted one. Retention and tamper evidence therefore pull
in opposite directions, and the code chooses evidence: no pass ever deletes an audit row. To take an old month out,
mark it with `markAuditArchived(sql, { month })` from `better-compute/audit`, then detach its partition with
`detachHistoryPartition`; `verify` reports that month's checkpoints as archived instead of failed. The module's
`doctor` check `audit.checkpoints` warns when rows older than 3 hours are still unsealed. See
[audit](/docs/guides/events/audit#archiving).

**Usage windows, by default.** The `capacity.usage` reconciler writes hourly `usage_windows` rows (key
`cmp:<allocationId>:<windowStartMs>`) every 5 minutes, and the IAM bridge's `iam.usage` job records them in Better
IAM's billing, setting `recorded_at` and `outcome` (`recorded`, `duplicate`, `reattributed` or `dead-lettered`). The
rows are the proof that usage adds up to booked time times atoms (invariant I8) and what a billing dispute is settled
from, so they stay unless you set `history.usageOlderThanDays`. See [billing and usage](/docs/guides/iam/billing-and-usage).

**Event sink dead letters.** `outbox_dead_letters` (`core/0019_outbox_dead_letters`) holds the events an event sink
refused on every attempt. Nothing prunes or redelivers them; delete rows once you have handled them. See
[sinks](/docs/extending/sinks#poison-events).

  - [Database](/docs/operations/database): The runtime tables, migrations and pooling.

  - [Observability](/docs/operations/observability): Findings and series that show a growing backlog.

  - [Scaling](/docs/operations/scaling): What grows first at 10k nodes.


# Scaling (/docs/operations/scaling)

> The PLAN.md section 16 performance gates, what the benchmarks measured so far, how each tier scales, which placer to run, and what breaks first.



Better Compute is designed for 10,000 nodes and 100,000 GPUs per region on one PostgreSQL. PLAN.md section 16 turns
that into gates each phase must pass, measured with `pnpm bench:scale` on fixed reference hardware. This page lists
the gates, the numbers measured so far, how each process tier scales out, and which limits you meet first.

> **Indicative numbers only.** 
  Every number below was measured on a development workstation (64 vCPU, 255 GiB), mostly on PGlite in one Node.js
  process with the load generator in the same process, on 2026-09-24 and 25. None is on the reference hardware, and no
  run on real PostgreSQL is recorded yet (the machine had no Postgres). PGlite is a single serialized connection, so
  these runs understate what Postgres can do in throughput and overstate latency under load. Treat them as a floor and
  as a record of what each optimization changed, not as capacity planning.

## The gates [#the-gates]

`bench/REFERENCE.md` fixes the reference machines: the gateway on 8 vCPU and 32 GiB running one gateway process on
Node.js 24, and PostgreSQL 17 on 16 vCPU, 64 GiB and NVMe. The instance type, network, the hosts for api, worker and
load generators, PgBouncer placement and the Postgres configuration are not fixed yet (decision D15). Load generators
never share a host with the process they measure.

| Gate                | Phase | Target                                                                                                                                                                                          | Measured so far                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Gateway streams     | v0.1  | One Node gateway holds 5,000 or more mTLS sessions at 10 s heartbeats for 30 min, dispatch p99 below 500 ms, RSS below 2 GiB; otherwise decide D20 (a Go gateway) before v0.2                   | Sessions, dispatch and RSS criteria met on a workstation: 5,000 mTLS sessions with the gateway in its own process (PGlite), dispatch p99 26 ms, gateway RSS 1,240 MiB, 0 drops, for 60 s; for 30 min on PGlite, every criterion met (dispatch p99 24 ms, RSS 2,016 MiB, only just, grown by PGlite's WebAssembly memory). Left: the 30 min run on the reference host with Postgres out of process, to confirm the RSS criterion |
| IAM lane            | v0.1  | Embedded better-iam on adapter-postgres sustains 50 or more admissions/s (about 150 lock acquisitions/s) with lock wait below 50%; record the ceiling; below 200 admissions/s, propose R1 (D18) | not run: it needs Postgres                                                                                                                                                                                                                                                                                                                                                                                                      |
| Placements          | v0.1  | Placements/s recorded                                                                                                                                                                           | 35.7/s inline, 1k nodes, PGlite (see below)                                                                                                                                                                                                                                                                                                                                                                                     |
| Place to active     | v0.1  | p99 below 2 s with 1k FakeDaemons in the real-time harness                                                                                                                                      | 1.02 s at 5/s in the first runs; 101 to 191 ms at 5/s and 176 ms at 10/s in later runs                                                                                                                                                                                                                                                                                                                                          |
| Cold start          | v0.4  | 10k nodes in under 3 min                                                                                                                                                                        | simulated time only (`runColdStart`, 3 gateways, PGlite): 10k nodes all connected, adopted and schedulable at 80 s (connected p99 29.7 s, bound by the 30 s power-on spread), all back 40 s after a power cycle, 0 unreachable marks, 0 violations; not measured on the wall clock or on Postgres                                                                                                                               |
| Gateway kill        | v0.4  | Recovery under 60 s, 0 false unreachable marks, 0 duplicate executions                                                                                                                          | 10k nodes, 3 gateways, in simulated time: re-home p99 13.25 s, 0 invariant violations                                                                                                                                                                                                                                                                                                                                           |
| Placements at scale | v0.4  | 200 or more placements/s                                                                                                                                                                        | pool-lease end to end on PGlite: 1,092/s at 1k nodes; 464/s at 10k nodes in the latest recorded run (see below)                                                                                                                                                                                                                                                                                                                 |
| Commands            | v0.4  | p99 below 1.5 s with polling only                                                                                                                                                               | ack p99 378 ms at 100 commands/s on 1k nodes, PGlite                                                                                                                                                                                                                                                                                                                                                                            |
| Postgres CPU        | v0.4  | below 50% at 10k nodes and 100k GPUs on the reference Postgres                                                                                                                                  | not measured                                                                                                                                                                                                                                                                                                                                                                                                                    |

A phase closes only when its exit tests are green in CI; v0.4's exit also asks for every section 15 simulation
scenario green on three consecutive nights. See [testing](/docs/operations/testing#benchmarks) for the suites and
their knobs.

## What the benchmarks measured [#what-the-benchmarks-measured]

### Gateway sessions [#gateway-sessions]

`bench/gateway-streams`: 5,000 simulated daemons each open a real `Session` with a `Hello` over HTTP/2 (100 streams
per connection over h2c; with `BENCH_GW_TLS=1`, one mTLS connection per daemon with its own node certificate),
heartbeat every 10 s, and answer 50 commands/s enqueued at random. `BENCH_GW_SPLIT=1` runs the gateway and its store
in their own process, so RSS is the gateway's alone. All runs are on a 64 vCPU workstation with PGlite.

| Run                                                      | Ramp                                             | Dispatch p50 / p99 / max           | RSS max                                                        | Drops |
| -------------------------------------------------------- | ------------------------------------------------ | ---------------------------------- | -------------------------------------------------------------- | ----- |
| 60 s, h2c                                                | 24.8 s (bound by the 200/s handshake bucket)     | 4 / 22 / 381 ms                    | 1,115 MiB                                                      | 0     |
| 30 s, h2c, before admission control                      | 58 s, 9,880 opens for 5,000 sessions             | 15 / 924 / 7,606 ms                | 871 MiB                                                        | 5,464 |
| 60 s, h2c, with a reconnect storm, `maxPendingHellos` on | 25.5 s                                           | 5 / 32 / 101 ms                    | 924 MiB                                                        | 0     |
| 60 s, mTLS, storm, generator in process                  | 83.8 s (refused dials: the Windows accept queue) | 12 / 27 / 99 ms                    | 2,183 MiB (includes the generator's 5,000 client TLS sessions) | 0     |
| 60 s, mTLS, storm, split                                 | 32.1 s (120 refused dials)                       | 14 / 26 / 94 ms                    | 1,240 MiB (the gateway process, PGlite included)               | 0     |
| 30 min, mTLS, storm, split                               | 30.4 s                                           | 14 / 24 / 422 ms (90,000 commands) | 2,016 MiB (JS heap 235 MiB)                                    | 0     |

The split mTLS run meets the gate's sessions, dispatch p99 and RSS criteria; its storm brought all 5,000 back
(p99 38.8 s, all 39.0 s) with 3,464 of 3,464 commands succeeded, and the gateway's event-loop p99 was 25 ms. The
result file still records `met: false`, because the run lasted 60 s. The 30 min split run met every criterion on
PGlite, RSS only just: it grew from 1,240 to 2,016 MiB while the JS heap stayed at 235 MiB, and a 5 min run with
memory attributed put almost all of such growth in `arrayBuffers`, PGlite's WebAssembly database inside the gateway
process. Its storm brought all 5,000 back in 40.4 s with 90,480 of 90,480 commands succeeded. The 30 min run on the
reference host with Postgres out of process is what remains, to confirm the RSS criterion (D15, D20). The suite's test timeout follows `BENCH_GW_DURATION_MS`, so a 30 min run is not
cut off by the default timeout.

The failed run found that a storm could make the gateway fence itself: Hellos queued without bound behind a slow
store, the gateway's own lease renewal waited past its 20 s TTL, and every fenced transaction dropped its streams.
`maxPendingHellos` (default 64) now refuses Hellos beyond that with `RESOURCE_EXHAUSTED`, and daemons retry with
jitter. In the h2c storm run all 5,000 sessions came back, p99 32.5 s and all within 33.2 s, and 3,056 of 3,056
commands succeeded. In the h2c runs RSS includes the in-process load generator, and h2c leaves out TLS handshake
cost.

**Status reports** (318b396). `bench/gateway-status` applies status reports as the gateway's status job does, with the
production `xidStatusHandler`, over 10,000 nodes on PGlite, sequentially. bcd sends a full report at least every
10-minute drift pass, so the steady need is 16.7 reports/s; the steady report ran at 662/s (p99 3.27 ms), about 40
times that, and a fleet-wide burst drains in about 15 s. Raising two conditions on every node ran at 253/s. The
gateway now publishes `node.status` only when a report says something new: `statusKey(report)` hashes the report
without its gauges (host pressure and volume storage), and a session publishes only when the key changes (a new
session publishes its first report again). Every report still runs the status handlers, so conditions, pressure and
`observed_generation` stay current. Before, every drift report was an event, about 1.44 million a day at 10,000 nodes.

### Placements [#placements]

`bench/placements` measures one pool of 8-GPU nodes, one GPU per placement, sequential, on PGlite.

| 1k nodes                                                          | Placements/s | p50 latency        | p50 pool lock held |
| ----------------------------------------------------------------- | ------------ | ------------------ | ------------------ |
| Inline, whole pool loaded under the pool lock (baseline)          | 12.4         | 79 ms              | 78 ms              |
| Inline with the SQL shortlist, planned before the lock            | 35.7         | 27 ms              | 2.3 ms             |
| Pool-lease, batches of 64, in-memory index                        | 118.1        | \~540 ms per batch | the whole batch    |
| Pool-lease, in-memory shortlist, index kept between batches (5 s) | 350.2        | \~170 ms per batch | the whole batch    |

| 10k nodes                                   | Placements/s |
| ------------------------------------------- | ------------ |
| Inline                                      | 7.8          |
| Pool-lease, index rebuilt every batch       | 21.1         |
| Pool-lease, in-memory shortlist, index kept | 93.3         |

End to end through the pool-lease worker's `placeQueued` (reservation rows, an event and a work-queue write each):
224/s one reservation at a time at 1k nodes; with set-based commits (one bookings insert, one allocations insert and
one reservations update per batch), 1,092/s at 1k nodes and 161/s at 10k nodes. Batches that rank a pool once per
request shape were measured at about 500/s at 10k nodes when they landed, and the latest recorded 10k run has 464/s
(640 placements, `bench/placements/results`).

Two later changes keep the pool lock short at 10k nodes; their numbers come from the changes' own PGlite runs at
10k nodes and are not in `bench/placements/results` yet:

* **The index is loaded before the lock.** A batch first claims its queued reservation rows (`FOR UPDATE SKIP
  LOCKED`), then loads or checks out the pool's index, and only then takes the pool row lock for planning and the
  guarded commits, which catch any booking committed in between. Rebuilding the index no longer holds the lock that
  inline placements on the same pool (instances, gangs, BookingRepair) wait for: lock held p90 went from 875 ms to
  138 ms. The benchmark's `lockHeldMs` for `placeQueued` now measures the lock itself, from the pool's `FOR UPDATE`
  to the commit.
* **The index loads devices lazily.** `PoolIndex.load` reads per-class counts from `capacity_device_counts`, the
  booked devices with their bookings, and which nodes have link rows. Full device rows and link matrices are read
  per node the first time a placement view asks for that node, so a shortlist reads a handful of nodes and only
  gangs and the whole-pool fallback read everything. Index load p90 went from about 1 s to 256 ms, `placeQueued`
  to about 1,058 placements/s, and lock held p90 to 60 ms.

The shortlist
(`packages/capacity/src/shortlist.ts`) ranks nodes in SQL and plans only the best 48, and it is exact: when a node
outside the shortlist could reach the top N, the placer plans the whole pool. It reads per-node device totals from
`capacity_device_counts` (migration `capacity/0003_device_counts`), which a row trigger on `devices` keeps exact,
instead of aggregating every device of the pool per placement: inline at 10k nodes went from 7.7 to 7.8/s to 10 to
12.3/s on PGlite. `doctor` reports `capacity.device_counts.drift` if the counts ever disagree with `devices`.

### Place to active, joins and commands [#place-to-active-joins-and-commands]

`bench/realtime-spine` runs the whole v0.1 spine on the wall clock: a started control plane, a gateway over h2c and
1k FakeDaemons that join with sealed tokens, pass `diag.nvml` acceptance and apply their NodeSpec. Whole-node
reservations then bind, and turn active when the daemons report `AllocationBound`.

| Placement rate         | Create p50 / p99 | Placed to active p50 / p99 | Gate               |
| ---------------------- | ---------------- | -------------------------- | ------------------ |
| 2/s (60)               | 143 / 179 ms     | 72 / 133 ms                | met                |
| 5/s (100)              | 174 / 353 ms     | 776 ms / 1.02 s            | met                |
| 10/s (150), first runs | 1.61 / 2.04 s    | 17.1 / 29.4 s              | missed (saturated) |

In the first runs each inline placement held the pool fence about 150 ms on PGlite, so one pool saturated between 5
and 10 placements/s; at saturation even the `singleton:capacity.reservations` lease lapsed. Later recorded runs
(`bench/realtime-spine/results`) held place-to-active p99 at 101 to 191 ms at 5/s with a burst of 100 reported at
p99 4.6 to 6.4 s, and 176 ms at 10/s, with no lease lapses. On an idle fleet a reservation is active about 70 ms
after `placed`.

**Joins.** Joining 1k nodes at 32 concurrent Joins took 75 to 99 s in the first spine runs and 62 to 65 s in later
ones, with Join RPC p50 about 1.8 to 3 s: every Join runs JoinService, the node CA and acceptance on the one PGlite.
The target set for join throughput, 1k joins in under 30 s on PGlite, is not met in these runs. In the multi-gateway
simulation, 1,000 nodes joined through JoinService in 9.8 s of wall time and 10,000 in 97 s.

A join-storm profile then cut the statements each join costs from 168 to 97 (200 joins on PGlite: 17.5 s to 10.7 s):
the gateway ingests the inventory sections of a whole group commit set-wise (one insert and one work-queue write per
batch, instead of a savepoint, an insert and an enqueue per section), inventory applies devices, labels, health
findings and conditions with one statement each, the node's authorization row is refreshed in one statement, and
JoinService no longer wakes the node controller at join when the pool runs acceptance checks (the first `Hello`
does).

**Commands.** `bench/realtime-fleet` times dispatch on 1k connected nodes. Gateway group commit (CommandUpdate and
event ingestion from every session in one I/O turn share one transaction, up to 500 per flush) moved ack p99 from
2.0 to 2.6 s down to 0.65 to 1.13 s across the recorded runs. Steady load on 1k nodes, 20 s per rate:

| Offered                         | Completed | Ack p50 | Ack p99 | Terminal p99 |
| ------------------------------- | --------- | ------- | ------- | ------------ |
| 25/s                            | 24.4/s    | 58 ms   | 94 ms   | 597 ms       |
| 50/s                            | 48.7/s    | 59 ms   | 114 ms  | 636 ms       |
| 100/s                           | 96.0/s    | 245 ms  | 378 ms  | 1.48 s       |
| 200/s asked, saturated at 117/s | 111.7/s   | 1.04 s  | 1.88 s  | 2.45 s       |

A full-fleet reconnect storm cost about what the first connect did: reconnect p99 10.3 s at 1k nodes and 28.1 s at
2k, since each handshake takes 10 to 20 ms serialized on PGlite.

### Gateway kill [#gateway-kill]

`bench/multi-gateway` runs N nodes joined through JoinService against 3 real gateways and the real control plane, in
simulated time, then kills one gateway without disconnect writes or a lease release.

| Nodes  | Moved | Re-home p50 | p90   | p99     | Violations | Heap per session (upper bound) |
| ------ | ----- | ----------- | ----- | ------- | ---------- | ------------------------------ |
| 200    | 67    | 0.5 s       | —     | 1.25 s  | 0          | 408 KiB                        |
| 1,000  | 334   | 0.5 s       | 1.0 s | 1.0 s   | 0          | 209 KiB                        |
| 10,000 | 3,334 | 3.25 s      | 8.5 s | 13.25 s | 0          | 102 KiB                        |

The gate (p99 below 60 s) holds at 10k. Re-homing grows with the fleet because a third of the nodes redial at once
into two gateways, whose handshake buckets and Hello transactions pace them. I1, I2, I5 and I8 held afterwards: no
command ran twice.

### Cold start [#cold-start]

`bench/cold-start` runs `runColdStart` with 3 real gateways and the real control plane on PGlite, in simulated time
(see [testing](/docs/operations/testing#cold-start)). Every node powers on within a 30 s spread; the fleet is ready
when every node is connected, adopted (after its `diag.nvml` acceptance) and schedulable. Then every gateway dies
for 10 s and every node reboots.

| Nodes                    | Connected p99 / all | Adopted p99 / all ready | Power cycle, all back | Unreachable | Violations | Wall  |
| ------------------------ | ------------------- | ----------------------- | --------------------- | ----------- | ---------- | ----- |
| 1,000 (`workerPasses` 1) | 29.8 s / 30.0 s     | 79 s / 80 s             | 40 s                  | 0           | 0          | 73 s  |
| 10,000                   | 29.7 s / 30.0 s     | 79 s / 80 s             | 40 s                  | 0           | 0          | 641 s |

The gate (under 3 minutes) holds at 10k with room to spare. Connection is bound by the power-on spread, and
adoption takes the same time at 1k and 10k (p50 55 s): it is paced by the acceptance command and the node pass,
not by throughput. Invariants I1 to I5 and I8 held.

A harsher variant starts all 10,000 nodes at the same instant (`bench/cold-start`, c3c7423). Before the retry
pushback and liveness counted from the join, every node was ready at 95 s but 23 were marked unreachable, all before
they had ever connected, and the connect tail was 65 s. With both, connection p99 is 16 s and the last node connects at
19 s, all are ready at 80 s, none is marked unreachable, a power cycle is back in 30 s, and there are no violations. Of the 10k run's wall time, 327 s went to the reconcilers and
46 s to the gateways' jobs, both on the one PGlite. The run does not measure Postgres CPU, the separate v0.4 gate.

## How each tier scales [#how-each-tier-scales]

<Mermaid
  chart="flowchart LR
  L7[&#x22;L7 balancer&#x22;] --> A1[&#x22;api&#x22;] & A2[&#x22;api&#x22;]
  L4[&#x22;L4 balancer, TLS passthrough&#x22;] --> G1[&#x22;gateway&#x22;] & G2[&#x22;gateway&#x22;] & G3[&#x22;gateway&#x22;]
  A1 & A2 & G1 & G2 & G3 --> PG[(&#x22;PostgreSQL + PgBouncer&#x22;)]
  W1[&#x22;worker: 128 shards&#x22;] & W2[&#x22;worker: 128 shards&#x22;] --> PG"
/>

**API processes** are stateless: authentication, admission transactions (and inline placement), the outbox. Add them
behind an L7 balancer as request volume grows. PLAN.md's HA region runs three.

**Gateways** hold sessions in memory and scale by streams; PLAN.md asks for at least three per region, behind an L4
balancer with TLS passthrough. The knobs that bound one gateway:

| Option                | Default | Effect                                                                                                                                                   |
| --------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `maxStreams`          | 5,000   | Sessions it admits; the next one is refused, told to retry in 5 to 30 s                                                                                  |
| `handshakesPerSecond` | 200     | Session opens and joins admitted per second (token bucket); a refusal is told to retry within the time the bucket needs for everyone refused that second |
| `maxPendingHellos`    | 64      | Hello transactions in flight; beyond it `RESOURCE_EXHAUSTED`, told to retry in 250 ms to 2 s                                                             |

Every busy refusal carries a jittered retry pushback (`grpc-retry-pushback-ms`, and on the WebSocket path also
`retry-after`), which bcd and the FakeDaemon follow instead of their own doubling back-off, so a power-on herd spreads
over the window the gateway can actually admit. See [admission and scale](/docs/daemon/gateway#admission-and-scale).
\| `gatewayLeaseTtlMs`, `leaseRenewMs` | 20 s, 5 s | Its `gateway:<id>` lease; losing it fences the gateway |
\| `dispatchIntervalMs`, `dispatchBatch` | 1 s, 1,000 | The dispatch loop, besides pokes (at most every `pokeWindowMs`, 10 ms) |
\| `livenessFlushMs` | 15 s | Liveness ages are flushed in batches, not per heartbeat |
\| `specSweepMs` | 10 s | How often sessions are checked for a newer NodeSpec |

A lost gateway loses open streams and nothing else: nodes redial through the balancer and re-home, and routing is a
join at dispatch time, so nothing needs rewriting. See [the gateway](/docs/daemon/gateway).

**Workers** scale by shards. Every sharded reconciler key maps to one of `SHARD_COUNT` = 256 shards (FNV-1a of the
key, `shardOf` in `@better-compute/core`). Workers heartbeat their `workers` row; rendezvous hashing over the live
rows names one owner per shard, and each worker holds the `shard:<n>` leases it owns (15 s TTL, renewed every 5 s,
epoch-fenced). Add a worker and shards move to it on the next membership pass; stop one and its shards move once its
leases expire. Singletons (the CommandSweeper, usage, retention, the IAM jobs) each run under one
`singleton:<name>` lease; outbox dispatch takes one lease per event sink (`singleton:outbox-dispatch:<sink id>`), so
several `sinks` workers share the sinks between them. The lease, epoch and work-queue schema is final, so scaling out needs no migration.

The work queue coalesces: enqueuing a dirty key bumps its generation instead of adding a
row, claims use `FOR UPDATE SKIP LOCKED` on the worker's shards, and a successful run deletes the key only if its
generation is unchanged. Owned keys are also re-enqueued every `resyncMs` (default 5 min, plus jitter) as a level
trigger. Worker tuning:

| `worker` option | Default  | Meaning                              |
| --------------- | -------- | ------------------------------------ |
| `batchSize`     | 32       | Keys a reconciler claims per pass    |
| `concurrency`   | 4        | Keys one reconciler runs at once     |
| `pollMs`        | 1,000 ms | Wait when there was nothing to do    |
| `claimTtlMs`    | 60 s     | How long a claimed key stays claimed |
| `resyncMs`      | 5 min    | Level-trigger resync period          |

`serve --roles` splits the roles over processes: for example two processes with `controllers,sinks,iam`. The
`scheduler` role runs reconcilers that declare `role: 'scheduler'` under a membership of its own, so placers scale
apart from the controllers and a scheduler-only process never takes node shards. The pool-lease placer
(`capacity.placer`) declares that role, so with `placer: 'pool-lease'` run at least one `scheduler` process; with
no sharded scheduler reconciler registered the role holds no shard leases. See
[deployment](/docs/operations/deployment).

## Choosing a placer [#choosing-a-placer]

The capacity module's placer is a deployment option: `betterCompute({ capacity: { placer, maxBatch } })`.

|                                    | `inline` (v0.1, default)                                                                                            | `pool-lease` (v0.4)                                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Where it runs                      | In the admission transaction, fenced by the pool row's `FOR UPDATE`                                                 | In the worker owning the pool's shard, as the `capacity.placer` reconciler (`shardBy: 'pool'`)                       |
| What `reservations.create` answers | `placed`, or `pending` with `Unschedulable`                                                                         | `pending` with a `Queued` condition until the batch runs, unless `wait: { timeoutMs }` holds the answer until it has |
| Throughput per pool                | One writer at a time; tens of placements/s on Postgres by PLAN.md's estimate, 35.7/s measured on PGlite at 1k nodes | Batches of up to `maxBatch` (at most 64, default 64) against one in-memory index                                     |
| Best for                           | Small pools, synchronous answers, simple tests                                                                      | Hot pools and placement storms                                                                                       |

Both commit with the same guarded statements, so a stale in-memory index costs a replan, never a double booking.
Gangs and placements without a reservation (VPS instances) always use the inline placer, as do repairs. The CLI's
`reserve` sends `wait` and polls while a reservation is still `Queued`; see
[waiting for placement](/docs/guides/capacity/reservations#waiting-for-placement) and
[placement](/docs/guides/capacity/placement).

## PGlite is not a benchmark database [#pglite-is-not-a-benchmark-database]

PGlite runs PostgreSQL in WebAssembly on one connection, with every transaction serialized behind a mutex, in the
same event loop as everything else. That shapes every number above:

* **Serialized writes.** Hello transactions ran at about 140/s, and handshakes cost 10 to 20 ms each, serialized. A
  gateway on Postgres overlaps them.
* **Shared event loop.** Load generators, 1k daemons, the gateway and the workers share one Node.js process, so
  event-loop delay shows up as latency.
* **Planner statistics.** PGlite has no autovacuum, so plans stay those of empty tables unless something runs
  `ANALYZE`. The `pglite()` executor runs it after migrations and every `analyzeEveryMs` (default 60 s), so dev, test
  and simulator plans match Postgres. Turn it off (`0`) only in tests that must issue no background statements.

Use PGlite for development, tests and the simulators. Run production and any capacity claim on PostgreSQL.

## What breaks first [#what-breaks-first]

PLAN.md section 8.6 estimates the load at 10k nodes and 100k GPUs: liveness writes for 10k rows every 15 s, about
170 session opens/s in a reconnect storm, about 50 commands/s steady (about 2k/s during mass provisioning), about 170
node events/s, about 33 reconciles/s of resync, and 200 or more placements/s. In order, the limits it expects and the
mitigation already built in:

1. **Liveness writes**: memory first, batched, transition-only.
2. **Telemetry**: goes straight to your TSDB, never through the control plane.
3. **Reconnect storms**: the handshake bucket, `maxPendingHellos`, retry pushback on busy refusals, full-jitter
   back-off, `Drain`, grace periods.
4. **Hot pools**: one writer per pool; split a pool above about 20k atoms; the pool-lease placer.
5. **Capacity reads**: rollups, never aggregates over devices.
6. **Inventory payloads**: section hashes; only changed sections travel.
7. **Connections**: PgBouncer in transaction mode. Leases are table rows, not session advisory locks, so pooling is
   safe.
8. **History**: `audit_events`, `inventory_snapshots` and `usage_windows` are partitioned by month
   (`core/0018_partitioned_history`), and the retention pass prunes rows and drops emptied months; see
   [retention](/docs/operations/retention#history-partitions).
9. **Gateway TLS CPU**: admission control; optional TLS offload is still to be verified.
10. **The IAM lane**: in embedded mode every IAM call takes a global lock. An admission costs at least 2
    acquisitions (3 on an authentication-cache miss), a list page 1 to 2 plus an audit append per denied item. The
    30 s authentication cache, the 10 s read-decision cache, one decision per mutation and SQL-scoped lists keep it
    down; watch `better_compute_iam_call_duration_seconds`.

For sizing PostgreSQL, PLAN.md gives only the reference machine (16 vCPU, 64 GiB, NVMe, PostgreSQL 17), an HA region
of three api processes, three or more gateways and two or more workers against Postgres HA behind PgBouncer, and the
v0.4 target of Postgres CPU below 50% at 10k nodes. See [database](/docs/operations/database).

  - [Testing](/docs/operations/testing): The benchmark suites and simulators.

  - [Observability](/docs/operations/observability): The series and findings to watch.

  - [Database](/docs/operations/database): PostgreSQL, PgBouncer and PGlite.

  - [Placement](/docs/guides/capacity/placement): How the placer picks atoms.


# Security (/docs/operations/security)

> The trust boundaries and threat model of Better Compute, the controls that enforce them, and a checklist for hardening a deployment.



Better Compute runs commands as root on your machines, so its security model starts from one question: who can make
a machine do something? The answer is meant to be only the control plane, only after Better IAM allowed the call,
and only in a form the machine can check itself. This page summarizes the model from the security section of PLAN.md, says which parts
are built, and ends with a hardening checklist.

## Trust boundaries [#trust-boundaries]

| Boundary                           | Protected by                                                                            |
| ---------------------------------- | --------------------------------------------------------------------------------------- |
| Caller to api                      | TLS at your load balancer; Better IAM authenticates every caller and decides every call |
| api to Better IAM                  | In-process (embedded) through the IAM bridge                                            |
| api and workers to gateway         | Mutual TLS from the internal CA                                                         |
| Gateway to daemon                  | Mutual TLS from the machine PKI: the node CA for clients, the serving CA for gateways   |
| Site agent to BMC                  | The management VLAN, reachable only from site agents                                    |
| Tenant to tenant                   | Tenant pre-checks, SQL tenant scoping, the authorization projection                     |
| Tenant-controlled node to platform | A `node-tenant` certificate that gets no secrets and no destructive platform commands   |

## Threat model [#threat-model]

| Threat                                 | Mitigation                                                                                                                                                                                                                                                                                                                                                                                                                                   | State                                                                                                                                                                                                                                                                                                        |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Cross-tenant access, forged attributes | Tenant pre-check before any database access; SQL tenant scoping; idempotency keys scoped to tenant and identity; attributes only from compute's own projection                                                                                                                                                                                                                                                                               | Built                                                                                                                                                                                                                                                                                                        |
| Stolen join token                      | Short TTL, `maxUses` (default 1), `expect` binding; tokens stored hashed, revocable, and sealed to JoinService in transit, bound to the CSR                                                                                                                                                                                                                                                                                                  | Built                                                                                                                                                                                                                                                                                                        |
| Compromised node                       | `nodeId` only from the certificate's SAN; 24 h certificates with pushed revocation; secrets bound to current work; reports only about its own allocations; its attested results speak only for itself                                                                                                                                                                                                                                        | Built                                                                                                                                                                                                                                                                                                        |
| Cloned node identity                   | A certificate alternating between two machines more than 3 times in 10 minutes is revoked (duplicate-identity quarantine); a `tpm` join admits only a node key the machine's TPM certifies it made and never lets out (`nodeKey: 'certified'`, the default), and its certificate names that TPM's AK (`urn:bc:tpm-ak:`), so every renewal must be certified by the same AK (`RenewRequest.key_attestation`; refused otherwise, no downgrade) | Quarantine built in the gateway; the TPM node-key proof built at join (cf24876) and through renewal (d94f1f3), for certified `tpm` joins only; tested with software TPMs only                                                                                                                                |
| Compromised gateway                    | Holds no key in its code; commands and NodeSpecs signed; Renew and `JoinResponse` verified end to end; secrets as JWE; results and `AllocationBound` reports signed by the node. It can delay, drop or read frames, not forge intent, identity, results, bindings or secrets                                                                                                                                                                 | Built, with the [process caveat](#gateway-processes)                                                                                                                                                                                                                                                         |
| Compromised api or worker              | Admissions still need IAM decisions; worker keys last 24 h; tickets are audited; fleet-wide destructive batches need k-of-n operator approval checked on the machine (the [destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10))                                                                                                                                                                                 | Built, the quorum included (v1.0): bcd and the FakeDaemon enforce it, the server stores, approves and holds batches, the gateway attaches approvals and proofs, the CLI reviews and signs, and the built-in workflows create their batches (f66e236, e594a4f). Open: metal's leaves do not bind payloads yet |
| Exploited node agent process           | With `bcd helper`, `bcd node` parses everything from the network as an unprivileged account; the root helper serves only that account (`SO_PEERCRED`) running an installed bcd binary, a few typed operations with fixed file paths, bcd's own nftables table and the host's own GPUs, and logs each one                                                                                                                                     | Built in bcd (28dc2ae, Linux), self-update included (2d67b25); opt-in with `daemon install-script --unprivileged` (169c459), while the default install still runs `bcd node` as root                                                                                                                         |
| Compromised driver plugin              | Pinned by SHA-256 and executed from the file that was hashed; run under its own unprivileged uid with `no_new_privs`, a seccomp filter (no namespaces, mounts or `io_uring`) and optional cgroup limits; only plugins named `privileged` run as root                                                                                                                                                                                         | Built in bcd (Linux)                                                                                                                                                                                                                                                                                         |
| Replay                                 | Command-id journal, `not_after`, epoch fencing, a server time mark against rewound `Welcome`s, ticket `jti` sets, a Renew replay cache                                                                                                                                                                                                                                                                                                       | Built in the control plane, the FakeDaemon and `bcd` (bcd keeps its ticket `jti` set in its journal, so a restart does not forget a spent ticket)                                                                                                                                                            |
| Agent overspend                        | Delegation scope, confirmations, per-agent quotas, budgets, `agents.suspendAll`                                                                                                                                                                                                                                                                                                                                                              | Through Better IAM                                                                                                                                                                                                                                                                                           |
| PXE and DHCP spoofing                  | One lease-holding boot server per site, isolated VLANs, signed images                                                                                                                                                                                                                                                                                                                                                                        | Partly built (v0.2)                                                                                                                                                                                                                                                                                          |
| Cross-tenant residue on bare metal     | BMC-driven teardown: attested erase, DCGM, a firmware baseline check, a new identity generation                                                                                                                                                                                                                                                                                                                                              | Built in `metal` (v0.2, v0.5)                                                                                                                                                                                                                                                                                |

> **What the real daemon checks today.** 
  `bcd` is built and tested but not released. It implements the journal, the pinned join and TLS, `JoinResponse`
  verification, signed commands and specs (once its trust bundle holds command-signing roots), node-signed attested
  results and allocation reports, the server time mark (a `Welcome` never moves its working clock back),
  exec-ticket re-verification before it opens a tunnel (a ticket that names an instance is dialed to that
  instance's own address, never the host's), the SSH
  `access` section (node-bound principals and a KRL; a spec whose CA keys or principals could inject an sshd line is
  refused whole, signed or not), and node-signed `SecretRequest` and `WorkloadTokenRequest` answered to a one-time
  key (the site agent sends them for BMC credentials and discovery tokens, the ramdisk and VPS hosts for user data,
  the metadata service for workload tokens). The site
  agent's boot server answers PXE only for prepared machines while it holds the boot lease, serves its image cache
  only under per-boot or per-firmware grants (b7ed8a2), bounds TFTP to 64 transfers and 4 per source (4d8b493) and
  request headers to 8 KiB, and keeps only well-formed fields of unknown PXE clients (0442561). PXE discovery never
  binds a MAC from that unauthenticated DHCP data alone (8de6e2f): a discovered MAC is a candidate until an operator
  confirms it with `metal.confirmMac`, unless the pool sets `policy.trustProvisioningNetwork`, in which case the
  ramdisk's UUID and NICs must match the machine (see
  [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines)). Driver plugins run
  sandboxed (below), and each `bmc.*` kind admits only its own BMC operations, so a `bmc.query`, which needs no batch,
  cannot carry a power or RAID change, and `site.httpProxy` never reaches a BMC. With approver keys in its trust bundle it enforces the destructive-batch quorum. Its node key is in the
  host's TPM when bcd can open one (non-exportable, but root on the node can use it), else a 0600 file. A `tpm` join
  proves it to the control plane: the TPM's AK certifies the key over the CSR's hash, the attestor refuses the join
  without that proof by default, and `node.joined` records `nodeKey: 'tpm'` (cf24876). The certificate then names
  the AK, and every renewal must carry that AK's certification of the next key, which bcd makes in the TPM or fails
  the renewal (d94f1f3). Other joins, and their renewals, prove nothing about where the key lives; see
  [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm). `bcd node` can run unprivileged beside the root `bcd helper` ([below](#privilege-separation-on-the-node)),
  when installed with `--unprivileged`, self-update included; by default it runs as root. See
  [the daemon](/docs/daemon#what-bcd-does-today).

## Gateway processes [#gateway-processes]

The gateway package holds no CA, signing or encryption key: it relays `Join`, `Renew`, `SecretRequest` and
`WorkloadTokenRequest` to `gatewayServices` and returns the answers verbatim. The design puts those services in the api process behind
internal mutual TLS.

In this release, `serve --roles gateway` builds `compute.gatewayServices` in the gateway's own process from the
shared configuration, so that process loads the PKI bundle and the deployment secret like any other. Treat gateway
hosts with the same care as api hosts until the services move out. `createGateway({ services })` takes any
implementation of `GatewayServices`, so a remote implementation can replace the in-process one without changing the
gateway.

## Controls [#controls]

### Mutual TLS, failing closed [#mutual-tls-failing-closed]

* Daemons connect to gateways over mutual TLS. `certificateAuthenticate` checks the chain against the node CA, the
  SPIFFE trust domain, revocation, role and expiry. `Join` alone accepts a client without a certificate; `Renew`,
  `Session` and `Tunnel` require one.
* The gateway role refuses to start without node-CA roots and a trust domain (`INVALID_CONFIG`). `serveGateway`
  refuses without `nodeRoots` or an explicit `authenticate`: there is no weaker default.
* `unsafeSanOnlyAuthenticate` and `insecure: true` (cleartext h2c) exist for tests only. `better-compute dev --sim`
  runs its gateway on loopback h2c with header authentication, for local development only.
* The api reaches gateways' internal control APIs (pokes, tunnels, revocations, closing an agent's streams) over
  mutual TLS from the internal CA. The client side is configured in `notifier: gatewayNotifier({ sql, tls })` and
  `host.tunnels.control.tls`.

### Allowlisted command kinds [#allowlisted-command-kinds]

Each certificate role accepts only its own command kinds. A certificate can be narrowed further with
`node_certs.allowed_kinds`, read at `Hello`: any other kind is cancelled `kind-not-allowed`, including in-flight
ones after a restricted re-join. A ramdisk admitted through discovery gets a certificate of at most 2 h limited to
the commissioning kinds. A `node-tenant` certificate gets no secrets, and destructive kinds sent to it are
cancelled `tenant-controlled`.

### Duplicate identities [#duplicate-identities]

A second live `Hello` for a node from another boot wins the session and publishes `security.node.duplicate-session`
(with both boot ids and the running count). A certificate that keeps alternating between two machines, the sign of a
cloned key, is quarantined once it does so more than 3 times within 10 minutes: its serial is revoked with reason
`duplicate-identity`, so every gateway's revocation feed drops the other clone too, `security.node.quarantined` is
published, and the `Hello` is refused `UNAUTHENTICATED` after the revocation commits. Re-join the real machine with a
new token. A session that already went quiet is superseded without counting. See
[sessions](/docs/daemon/sessions#duplicate-identity).

### Signed commands [#signed-commands]

With `pki.commandSigning`, every command is signed by a 24 h worker key that the command-signing root certifies,
and so is every NodeSpec generation (v0.4): the NodeSpec controller signs the exact `DesiredStateBody` bytes of each
new generation (purpose `desired-state`) and stores them in `nodes.spec_body` and `nodes.spec_signature`. The gateway
relays those signatures verbatim; it cannot produce them. A daemon checks any signature that is present, and both
bcd (0d42de8) and the FakeDaemon (12ccfa1) with command-signing roots require a valid one on every command, whatever
its kind: a kind table that exempts "harmless" kinds let a compromised gateway inject an unsigned `site.httpProxy`
that could reset a BMC or insert media around the quorum. bcd and metal also treat `site.httpProxy` as destructive (signed, and under the batch quorum like
`bmc.*`) and never proxies to a BMC: its site agent remembers the BMC hosts it drives (`data_dir/bmc-hosts.json`)
and refuses them by name and resolved address, and refuses any `/redfish` path and any address in its own
`[site] bmc_networks` (local configuration the control plane cannot loosen; doctor warns about site agents without
it, a019c85), so a BMC is reached only through the `bmc.*` kinds and their quorum. A request other than a `GET`
runs only under a batch leaf bound to its exact payload. A request about one machine also cannot reach another (a2a853d, 8391d74): metal lists
the site's agentless machines and their inspected addresses in the site agent's spec (`SiteSpec.proxy_targets`,
never a BMC host), and bcd refuses a URL host that is not one of the target node's listed addresses or names
(`forbidden-target`) once the list names any target. For desired state, the FakeDaemon with command-signing roots
requires a signature on every spec by default (`signedSpecs: 'always'`, v0.4), and so does bcd, since an unsigned
spec with a high generation would freeze all later ones; `signedSpecs: 'access'` requires one only on a spec that
changes the `access` section (v0.2). Because `Welcome.server_time` is unsigned, both daemons also refuse a `Welcome`
more than 5 minutes before the highest server time they have seen, so a gateway cannot replay an expired worker's
signed command by rewinding the daemon's clock. See
[PKI](/docs/operations/pki#signed-intent).

A worker key can still sign anything, so a compromised api or worker could wipe the fleet with valid signatures.
PLAN.md §13's v1.0 answer is the destructive-batch quorum: `pki.approvers` pins operator approver keys, held off the
control plane, in the join-key-signed trust bundle, and a destructive command then runs only as part of a batch that
enough of them approved, proven by a Merkle inclusion proof the machine checks itself; unbatched destructive commands
are refused by default. The chain is built end to end: bcd and the FakeDaemon enforce it, the server's `quorum`
module stores batches and collects approvals (`commandBatches.approve`), the command queue holds batched commands
until approval, the gateway sends each with its approval and inclusion proof, and the `better-compute batch`
commands let approvers review a batch against its root and sign it. A change of the approver set itself needs the
current approvers' signatures (`batch sign-approvers`), so the join key alone cannot lift or replace the quorum. The
built-in workflows produce their batches: metal puts each workflow under a one-node batch of its own, or under the
batch `metal.bulk` creates for one verb on many nodes (f66e236), and the health reconciler's reboots and GPU resets
run under a node-scoped batch (e594a4f). Metal's leaves leave the payload unbound (a leaf admits any payload of its
kind for its subject), which approvers see flagged; binding metal's payloads is open. See
[the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10).

### Writing into deployed images [#writing-into-deployed-images]

`deploy.image` writes the NoCloud seed and bcd's join files into the image it just streamed, and that image may be a
tenant's: a crafted filesystem attacks whatever kernel mounts it. For an ext2, ext3 or ext4 root filesystem, the
ramdisk never mounts it (1ae1789): e2fsprogs' `debugfs` writes the files in userspace, unlinking whatever the image
has at each path (a symlink, a device node) without following it, writing root-owned 0600 regular files, and reading
each back. Ignition's config goes onto the `boot` partition the same way (`debugfs` for ext, `mtools` for FAT), and
`boot.kexec` copies the installed kernel and initrd out of an ext root with `debugfs`, resolving symlinks inside the
image, instead of mounting it (d9a900f). Only xfs and btrfs, or a ramdisk without the tools, still fall back to a
`nodev,nosuid` mount (read-only for kexec) with every write confined by `os.Root`. See
[what the ramdisk does](/docs/guides/bare-metal/deploying#what-the-ramdisk-does).

### Privilege separation on the node [#privilege-separation-on-the-node]

As installed, `bcd node` runs as root under a hardened systemd unit (`ProtectSystem=strict`, `NoNewPrivileges`, a
bounded capability set; see [the unit](/docs/daemon/install#the-systemd-unit)), so a memory-safety or logic bug in
the code that parses the session, commands and specs would run as root. PLAN.md §6 (v0.4) answers that with a split,
and bcd has it since 28dc2ae:

* **The node agent is unprivileged.** With `[helper] socket` and `[helper] user`, `bcd node` runs as that account.
  The session, the journal, and every frame, command and spec from the gateway are handled there.

* **The root helper trusts nothing it is sent.** `bcd helper` serves only a caller whose `SO_PEERCRED` uid and gid
  are `[helper] user`'s (or root) and whose `/proc/<pid>/exe` is an installed bcd binary. Requests are versioned JSON,
  at most 4 MiB, decoded strictly (an unknown field or version is refused).

* **Typed operations, not paths.** Files are named by kind and id (`cdi`, `allocation-env`, `ssh-dropin`, `ssh-ca`,
  `ssh-krl`, `ssh-principals`); the helper builds the path from its own table, refuses a symlink or non-directory on
  the way down, caps a file at 1 MiB and writes it atomically without following a symlink at the target. A ruleset
  must stay inside `table bridge bcd_instances` and `table inet bcd_taps` (no other table or family, no `include`,
  `define` or `ruleset`); `gpu.reset`
  names only a GPU in the host's own inventory; a reboot reason is cut to 200 bytes.

* **Everything is logged.** Each operation and each refused caller is logged with the caller's pid.

* **Self-update without trusting the node** (2d67b25). The slots and the upgrade and release records live in root's
  `[update] slots_dir` (default `/usr/local/lib/bcd`, never inside `data_dir`), and the executable check trusts only
  slots there. The node only downloads, into `<data_dir>/staging`; the helper takes no path from it, verifies the
  signed manifest under its own pins, floor and clock, opens the staged file with `O_NOFOLLOW` and hashes it as it
  copies it into the slot, so what is checked is what runs. Its own watchdog swaps back an update no `Welcome`
  confirmed, whatever the node does.

* **No symlink race on the socket** (2d67b25, from a security review). Root chmod and chown follow symlinks, so with a
  node-writable socket directory the node could have made root set `0660 root:bcd` on any file. The socket's
  directory must now be root's and closed to group and other writes, and the socket is bound, chmodded and chowned
  in a private 0700 directory, then renamed into place.

* **Host units only from signed specs** (03c023d, 2d18327). The helper's `host.unit` applies a config unit as root,
  and every request carries the signed `DesiredState` that last set the host section and its worker signature (`bcd
  node` keeps the body in `data_dir/host-intent.json`). The command-signing roots come from `command_root_sha256` in
  root's `config.toml` (the SHA-256 of each root certificate, given at install, never taken from the control plane or
  the node), which decides alone when set, or else, since 71ccf71, from the trust bundle in root's own signed join
  record, which makes signed intent mandatory whatever the node sends; a newer `SignedTrustBundle` the node presents,
  checked against root's `join_signing_key_pin`, rotates the roots forward only, so a root rotation needs no new pin. The helper refuses unless the roots
  check out (pinned, or from a bundle no older than one it accepted), the signature verifies (worker role), the body
  names this node (since a966cd4 the node root's own signed join record names, `/var/lib/bcd-hostconfig/join-response.pb`,
  re-verified against `join_signing_key_pin`; a node joined before it keeps the first signed spec's binding in
  `/var/lib/bcd-hostconfig/helper-intent.json`), its generation is
  not older than one already applied, and the unit's settings are exactly the body's. Once a bundle with roots was
  accepted, a request without a signed spec is refused. A verified body stays good after its short-lived worker
  certificate expires, and a node and a helper of different versions fail closed. &#x2A;*The caveat:** nothing is checked
  on an install where neither `command_root_sha256` nor root's join record names command-signing roots (a deployment
  that signs nothing, or a node joined before the record) until the node itself presents a bundle with roots; the
  helper is then bounded by strict unit validation, the sysctl denylist, root writes only under root-only directories,
  and `k8s-join` only to root-pinned `allowed_clusters`, and `bcd doctor` fails on it (71ccf71). `daemon
  install-script` pins `command_root_sha256` from `--command-root-sha256` or the local PKI bundle (59985ae), which
  closes that window. The trust-on-first-use binding of the node id is closed by root's join record (a966cd4), except
  on nodes joined before it, which have no record. `bcd doctor` (as root, with `[helper]`) warns on such a node
  (f77c6f9); re-join it as root with `bcd join --reset --yes` (a1c1a46), since re-running the install script
  on a joined node writes no record. See [signed specs](/docs/daemon/host-config#signed-specs).

* **Root writes only under root's directories** (02485b3, f8e13ef). Whenever bcd writes as root for a config unit,
  every existing directory from `/` down to the file must be a real directory owned by root and writable by no one
  else, so `/tmp` and anything the node owns are refused, and the units' records live in `/var/lib/bcd-hostconfig`,
  never in the node's `data_dir`. The helper's file kinds follow the same rule.

* **libvirt through the helper** (c305a05, af09411, 8c0cf9e, fea4d2e, ed3c7d2, d418820). The `libvirt` group is
  root-equivalent, so an unprivileged node is not in it. The helper defines, starts, stops, resizes, saves and
  restores the node's libvirt domains and opens their consoles, only for domains bcd defined, after a default-deny
  check of each definition: disks only under bcd's directories, checked and re-checked as root, taps named for the
  instance on `[helper] instance_bridges`, hostdevs only of the host's GPUs, no QEMU command line, filesystems,
  graphics or foreign namespaces. libvirt runs with `dynamic_ownership = 0`, so it never chowns a path the node names.
  The node reads through libvirt's read-only socket. See
  [libvirt through the helper](/docs/daemon#libvirt-through-the-helper).

* **cloud-hypervisor** (7fe74a1). Each VMM runs as its own uid in a confined unit, and the node shares the instance's
  files with that uid through ACL entries. Root creates only the instance's own tap names on `[helper]
  instance_bridges`, binds only whole IOMMU groups of this host's idle GPUs to `vfio-pci`, and builds each VMM's
  unit itself from the instance record it checks, with its own binary, never a command line of the node's. See
  [cloud-hypervisor through the helper](/docs/daemon#cloud-hypervisor-through-the-helper).

* **The TPM** (8870b8d, security review 3). The kernel's TPM resource manager filters no command: whoever opens
  `/dev/tpmrm0` can change the hierarchies' authorization, clear the TPM through the default empty lockout
  authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets and attestation. So the
  node never opens it. Root's helper derives, signs with and certifies its node keys (`tpm.*`, keys named by their
  32-byte unique), builds every TPM command itself, and answers only public areas, signatures and certifications,
  which the node checks against the key's public half. See
  [node keys through the helper](/docs/daemon#node-keys-through-the-helper).

`daemon install-script --unprivileged` (169c459) sets the mode up: a `[helper]` section with the socket in the
helper's own root-owned `/run/bcd-helper`, the `bcd` system account, `bcd-helper.service`, a `bcd.service` drop-in
with `User=bcd`, and `/etc/bcd` readable by the `bcd` group only; on a libvirt host, `dynamic_ownership = 0` and QEMU's
user in the `bcd` group (d418820). It keeps the `bcd` user out of the `tss` group: scripts from 0ba2a79 until
8870b8d put it there, and an upgrade run takes it out and deletes their `/etc/udev/rules.d/60-bcd-tpm.rules`.

The limits are plain. A host installed with `--unprivileged` before d418820 keeps `bcd node` in the `libvirt` group
until the install script runs again (see
[upgrading an unprivileged libvirt host](/docs/daemon/install#upgrading-an-unprivileged-libvirt-host)); `bcd doctor`
fails it until then. Likewise a host installed between 0ba2a79 and 8870b8d keeps `bcd` in `tss` until the script runs
again; since e6c3f53 `bcd doctor` fails its `tpm` check there (`gpasswd -d bcd tss`, then restart `bcd.service`). The helper covers files, the firewall table, GPU resets, reboots, sshd
reloads, host units, self-update, libvirt, cloud-hypervisor (tested against fakes only) and the TPM's node keys; PLAN.md's `mig.apply` is not a
helper operation, and bcx plugins need a root node (`--unprivileged` refuses `--plugins`). The default install still runs `bcd node` as root. See
[privilege separation](/docs/daemon#privilege-separation) and
[running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged).

### Driver plugins on the host [#driver-plugins-on-the-host]

bcx driver plugins run next to bcd, which runs as root as installed, so bcd confines them (`[plugins]` in
`config.toml`):

* **What runs is what was pinned.** Each binary is hashed through an open file and, on Linux, executed from that
  file (`/proc/self/fd/3`), never by path again, so swapping the file between the check and the exec runs nothing
  else.
* **Not as root, and not as each other.** When bcd is root, each plugin runs under its own uid and gid, or all as
  `[plugins] user`; no supplementary groups. The uids come from a block bcd keeps for itself (`0x7FFE0000` to
  `0x7FFEFFFF`, clear of system users, systemd's `DynamicUser` range and container ranges), are derived from the
  plugin's name, skip any uid an account already has, and are assigned in name order, so they are stable across
  restarts and reordering the `pins` file changes nothing (6827df8). No plugin can read another's memory. `user = "root"` is refused: plugins that need root (a Kata runtime) are named in
  `[plugins] privileged` and run without the filter, and bcd logs each one.
* **A narrow kernel surface.** With `[plugins] sandbox` (default on), bcd's `__bcx-exec` shim sets `no_new_privs`
  and installs a seccomp filter on every thread (TSYNC) that refuses kexec, kernel modules, mounts, namespaces
  (`unshare`, `setns`, `clone` with `CLONE_NEW*`; `clone3` answers `ENOSYS`), swap, reboot, `bpf`, `io_uring`,
  `ptrace` and `process_vm_*`, the keyring, `perf_event_open`, `userfaultfd`, `open_by_handle_at`, raw I/O ports,
  clock changes, `acct`, `quotactl`, `lookup_dcookie` and `syslog` with `EPERM`, and kills a call from a foreign ABI.
* **Bounded and reaped.** `[plugins] cgroup` puts each plugin in its own child cgroup with `memory_max` and
  `pids_max`; every plugin has its own process group, dies with bcd and is killed as a group on close, and with a
  cgroup also through `cgroup.kill`, so a child that left the group with `setsid` dies too (933c51b). The handshake
  line is at most 64 KiB, each plugin message at most 16 MiB, and `Describe` is bounded by the handshake timeout, so
  a plugin that never answers does not hold up bcd's start.

The install script's `bcd.service` stays as tight as before on nodes without plugins. On a node whose `config.toml`
has a `[plugins]` section (`daemon install-script --plugins`), the script adds a drop-in,
`bcd.service.d/plugins.conf`, with only what the user switch and the cgroup limits need: `CAP_SETUID`, `CAP_SETGID`
and `CAP_KILL` in the bounding set, `Delegate=yes` with `DelegateSubgroup=main`, and `ProtectControlGroups=no`;
`RestrictNamespaces=no` is added only when `[plugins] privileged` names a plugin, such as a Kata runtime. See
[the plugin drop-in](/docs/daemon/install#the-plugin-drop-in) and
[the plugin sandbox](/docs/extending/drivers#the-plugin-sandbox).

### Attested results [#attested-results]

A wipe or diagnostics result counts as attested only when the node signed it: the gateway stores
`CommandUpdate.node_sig`, and metal verifies it against `node_certs`. A revoked certificate is refused; an expired
one is accepted for 7 more days, for results journaled offline. Before this fix a kind's `attested` flag was
trusted, so a gateway could have forged a wipe. A `node-tenant` certificate never attests.

GPU bindings get the same treatment. With `pki.commandSigning`, capacity's `attestedBindings` defaults to
`pkiAllocationBindingCheck()`: a reservation goes `active` only once the node's own signed `AllocationBound`
verifies, the signed CDI devices replace whatever the gateway said, and a rejected report publishes
`reservation.binding-rejected`. A forged re-report of an active binding marks it lost instead of replacing it. The
gateway's report handlers also ignore reports about allocations placed on another node. See
[attested bindings](/docs/guides/capacity/reservations#attested-bindings).

### Sealed secrets [#sealed-secrets]

* PostgreSQL holds only `secret://` references. The default store seals values in `sealed_secrets` with
  AES-256-GCM; the authenticated data binds each ciphertext to its reference and tenant.
* `secrets.put` is write-only: the API never reads a value back. Events and audit rows carry the reference and
  version, never the value. The CLI's `secrets put` takes the value only from `--value-file` or `--stdin`, never an
  argument, and `metal enroll` takes a BMC password only from `--bmc-password-file`; tokens the CLI creates
  (`join-tokens create`, `site-tokens create`, a webhook's signing secret) can go to a new 0600 file with `--out` or
  `--secret-out` instead of the output.
* The broker (`gatewayServices.secret`) releases a value only when the request is signed by the node's current,
  unrevoked, platform certificate, names the session's node, is bound to work that node is doing right now, and uses
  a fresh request id. The answer is a JWE to the request's one-time key, expiring within 5 minutes. A reference is
  bound when:
  1. a live command of the named workflow step, addressed to the node, carries it;
  2. a live allocation on the node references it (an instance's user data, or what a module's `allocationRefs`
     accepts);
  3. the node's own spec names it as a credential (e93bf9c, b7346a9): the request names workflow `host:<unit>` and
     step `<field>` (both signed), the pair is allowed, the node's current spec (`nodes.spec`, rewritten with every
     generation) has that unit or section naming exactly this reference in that field, and the reference is a
     `secret://` reference of the node's owner or its pool's owner. The node's certificate is checked again in the
     same statement that reads the spec, so a spec from before a policy edit cannot release an old reference.
     Allowed by default: the `k8s-join` config unit's `tokenRef` (the kubeadm bootstrap token; see
     [k8s-join](/docs/daemon/host-config#k8s-join)) and the `telemetry` section's `headersRef` (the OTLP exporter's
     headers; see [node metrics](/docs/operations/observability#node-metrics-from-bcd)). `createSecretBroker`'s
     `hostUnitRefs` and `specRefs` options set the lists, and `[]` turns either off; `betterCompute()` uses the
     defaults. The audit row's details carry `hostUnit`.
* A hosted VPS instance's user data is released only when its `secret://` reference belongs to the instance's own
  tenant, so one tenant's VM cannot boot with another tenant's secret. Since the guest reads its user data,
  `instances.create` accepts only a secret the tenant wrote with `secrets.put` (a `secret_refs` row; never a
  platform-minted one such as a join token, a BMC credential or another instance's sealed user data) and also
  decides `compute:secrets:manage` on the tenant with the caller's own credential (and on the fleet in the root
  tenant), so `compute:instances:create` alone cannot read a write-only secret. A fleet secret (`secret_refs.scope`
  `fleet`, such as a shared site's BMC credentials in the root tenant) is never user data: `instances.create` refuses
  it with `INVALID_INPUT` even for a holder of fleet `compute:secrets:manage` (aac73a1). Sandbox templates follow the
  same rule.
* A provider account's references are its tenant's. An account of a tenant other than the root may name only
  `secret://<its tenant>/…` and the `vault://` paths `vps({ tenantVaultPaths })` allows it, never `env://` (the control
  plane's environment) nor the platform's or another tenant's secrets; `providerAccounts.create` and `update` refuse
  anything else with `INVALID_CONFIG`. The client checks again on every resolve (`ACCESS_DENIED` for a row stored
  around the API) and resolves a tenant's references with the tenant as the context, so the sealed store refuses a
  row stored for another tenant. A Google Cloud key's `token_uri` must be Google's token endpoint: the control plane
  POSTs to it, and the key is the tenant's.
* `secrets.put` records the scope it was decided on (`secret_refs.scope`, `tenant` or `fleet`), and a tenant-scoped
  put or delete never changes a fleet secret (`INVALID_STATE`).
* Nothing is released for a tenant that is suspended or deleted: after the signature and binding checks, a reference
  whose tenant has a non-active `compute_tenant_states` row is refused (reason `tenant-suspended` or
  `tenant-deleted`, recorded as `security.secret.refused`), so a deleted tenant's instance cannot fetch its user data
  while teardown is still reaching it. It is released again once the tenant is active.
* Commands carry references, never values.

### Workload identity (v0.5) [#workload-identity-v05]

With `workloadIdentity` configured, a VPS host's metadata service can get a hosted instance's OIDC token. The
broker (`gatewayServices.workloadToken`) mints one only when the request is signed by the node's current platform
certificate of role `node` (never `node-tenant`, ramdisk, site or a kind-restricted certificate) naming the session's
node, for a live instance placed on that node with its live allocation there, in a pool with
`policy.workloadIdentity: true`, for an allowed audience, with a fresh request id and within the instance's rate.
The gateway refuses a tenant-controlled session without asking, since the lessee holds root there. The token is a
JWE to the request's one-time key, lives at most 15 minutes, and its audit row carries only the `jti`. Refusals
publish `security.workload.refused`. On the host, a session token binds each metadata request to one instance and
forwarded requests are refused. Since the metadata service and instance tunnels name an instance by its address, bcd
guards every instance tap with host nftables rules (only the NIC's own MAC, IP addresses, EUI-64 link-local address
and ARP; each guarded address bound to its own tap so no other bridge port can send from it or claim it; on every
tap, guarded or not, no DHCP server reply, router advertisement or redirect, and never the host's own bridge
addresses; no foreign neighbour advert), trusts only addresses those loaded rules bind to an instance, drops traffic from or to
a tap that is off its bridge (the `inet bcd_taps` backstop), counts the tables' rules every 5 seconds so a flushed
table or chain is caught and loaded again at once, and never serves an
address two instances claim or a loopback caller. See [workload identity](/docs/guides/vps/instances#workload-identity).

### Tunnels, tickets and break-glass SSH [#tunnels-tickets-and-break-glass-ssh]

* `tunnels.open` decides the caller with Better IAM, then issues a 60 s, single-use exec ticket for one node. The
  gateway and the daemon each verify it again, including its `jti`.
* While a tunnel is open, the caller is re-authorized every 60 s by the ticket's own action; after 2 minutes of IAM
  failures it closes. A ticket with an unexpected action fails closed. Tunnels last at most 12 h. A revoked
  delegation or suspended agent closes its tunnels on every gateway when `host.iam` is configured.
* Break-glass shells use `nodes.issueSshCert`: `compute:nodes:ssh-root` on the node and, by default, an MFA session.
  Certificates last at most 1 h and never beyond the session, name `<account>@<nodeId>`, and grant only
  `permit-pty`. Each issuance is recorded; `nodes.revokeSshCert` puts the serial in the node's key revocation list.
  With `host.iam`, a role deactivation, an identity's offboarding, expiry or deletion, and its session revocations
  revoke its certificates too, and the `tunnels.ssh-certificates-reauthorize` reconciler re-decides every live
  certificate for its issuing session each minute. Only a `false` answer, `NOT_FOUND` or `TENANT_SUSPENDED` revokes.
  When IAM is unreachable, or refuses compute's own credential (a thrown `ACCESS_DENIED` or `UNAUTHENTICATED`), a
  certificate stays valid until it expires, on purpose: break-glass is for outages, and an expired service token
  must not revoke every certificate mid-outage.
* Per identity and per api replica: 10 tunnels a minute and 20 certificates an hour, then `RATE_LIMITED`.
* `ssh-cert --connect HOST` accepts only a host name or an IP address, never something ssh would read as an option.
  `ssh-cert` checks the server's answer before using it: the account must be a Unix account name (it reaches ssh's
  `%r` and `%u`), and the certificate an OpenSSH user certificate for the caller's own key.

See [interactive access](/docs/guides/access).

### Sandboxes (v0.5) [#sandboxes-v05]

A sandbox is a tenant VM for code you do not trust, often written by an agent. The controls:

* **One way in.** `sandboxes.exec` needs a signed-in session (not an API key), decides `compute:sandboxes:exec` on the
  sandbox, and signs a certificate of at most 15 minutes, never past the session or the sandbox's expiry, for a
  principal that names one instance. It refuses a sandbox that expires within a minute. `tunnels.open` refuses a
  sandbox's instance and a warm one (`sandbox-managed`), so `compute:instances:exec` cannot bypass that decision.

* **Re-decided on the sandbox.** The ticket carries the sandbox (`sbx`); the gateway keeps it and every 60-second
  re-check decides `compute:sandboxes:exec` on the sandbox, never on the instance it drives.

* **Separate CAs.** The sandbox SSH CA is never the break-glass CA: the tunnels module refuses the same key for both
  (`INVALID_CONFIG`), so a sandbox certificate never opens a node.

* **Secrets sealed before storage.** Template `env` values and a `secret://` user data are resolved into user data
  that is sealed through the secret broker before the instance row is written; without a broker such a template is
  `INVALID_CONFIG`.

* **Secrets only the tenant put.** A template's `env` and `secret://` user data must be secrets the tenant wrote with
  `secrets.put`, never fleet-scoped ones, and naming them decides `compute:secrets:manage` (on the fleet too in the root tenant, whose store
  also holds fleet secrets). An update that changes what the guest runs is decided on the template's secrets even
  when it names none, and each new instance re-checks them.

* **A rotated secret leaves the warm pool.** Warm guests hold resolved secret values, so each records the
  `secret_refs.version` of every secret it got; after a rotation or delete, claims skip and retire the stale ones
  (falling back to a cold create) and the reconciler replaces them (2c7072b, review 13 L4).

* **Snapshots are the owner's.** A snapshot holds memory and `/etc/sandbox/env`, so it belongs to the sandbox's owner,
  whoever took it, and only that owner can restore it (`NOT_FOUND` for anyone else in the tenant); someone allowed
  to snapshot another user's sandbox cannot boot its memory as their own. An agent restores only snapshots of
  sandboxes it created itself, a restore runs only under its snapshot's own template (so memory captured under a
  `none` network never resumes under `egress`), and an owner who leaves without a successor has every snapshot
  dropped (7b01053, security review 13). A deleted snapshot's object is purged only once its upload settled and no
  live instance restores from it.

* **Each restore gets its own identity.** A memory restore resumes with its source's sshd principal and host key, so
  a certificate for one such guest would open the others (review 13 H2). vps seals each memory restore its own
  principals and, with `certifyHostKey`, a fresh certified host key (5346523); bcd hot-adds them to the running guest
  on a 1 MiB `bc-rebind` disk that it zeroes and deletes afterwards (ca08d5c), and the guest half vps installs at
  first boot (3e25c78) installs them and acknowledges. Only when the host reports the acknowledged reference does exec
  sign for the restore alone. Until then `sandboxes.exec` refuses with `INVALID_STATE` (`restore-shares-identity`)
  while another live guest of the tenant shares the identity (87ff2c5): a guest that never acknowledges (booted
  before the guest half, on a runtime that cannot rebind) stays refused, which fails closed. Disk-only restores are
  not affected. See [a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity).

* **SSH only from the host.** Every sandbox instance's spec sets `ssh_from_host_only`, and bcd drops tcp/22 into its
  taps from anything but the host's own bridge addresses (474d51f, review 13 L5). An exec certificate therefore works
  only through a tunnel, which is re-authorized every 60 seconds and closed on revocation, and one guest cannot reach
  another's sshd. A bcd from before 474d51f would ignore the field, so since 82f4f51 sandbox and warm instances are
  placed only on hosts labelled `bc.dev/cap.instance.ssh-host-only`, which a bcd that enforces it advertises.

* **Replays and races.** A sandbox create's idempotency hash covers the caller's agent and delegation, so an agent
  cannot replay its person's key (or the reverse) to take over a sandbox, and a template update decided on one set
  of secrets is refused (`CONFLICT`, `secrets-changed`) if the template names others by the time it writes
  (7b01053).

* **The CLI treats the guest as hostile.** `sandbox exec` puts `--` before the host, so a generated command is never
  an ssh option, and turns off agent and X11 forwarding, port forwardings, proxies, connection sharing and local
  commands for that connection. CLI output strips control characters from server text (table cells, string results,
  hints) and JSON escapes DEL and the C1 controls.

* **Network policy on the host.** A template's `network` (`private`, `none`) becomes host nftables rules on the
  instance's taps, whatever runs inside; bcd fails closed, stopping an instance whose rules cannot be expressed or
  loaded, and vps refuses `private` or `none` on a pool without an IP range. Such instances are placed only on hosts
  labelled `bc.dev/cap.instance.network-policy`, since an older bcd would drop the policy and give full egress.
  `private` spans the pool's whole IPAM subnet, other tenants' instances included on a shared pool.

* **Capability labels are bcd's alone** (c1fd430, 0ff3ded, 736d3e5; review L11). The `bc.dev/` labels above, and the
  `bc.dev/driver.*` and `bc.dev/driver-kind.*` labels, are what bcd advertises, written by the gateway with source
  `system`. An operator could otherwise forge one and land sandboxes, placed flavors or volumes on a bcd that does
  not enforce them. Since 409f783 the same goes for GPU reservations: capacity books host resources only on nodes
  labelled `bc.dev/cap.devices.limits` and binds `kubernetes` reservations only on `bc.dev/cap.binding.kubernetes`
  ones, since an older bcd would run the hold without its CPU and memory limits (see
  [capabilities](/docs/daemon/sessions#hello)). So [`nodes.label`](/docs/reference/api/nodes#label) refuses to set or remove any `bc.dev/` key,
  and every reader counts a `bc.dev/` label only with source `system`: placement (the node labels it reads and the
  exact shortlist), metal's driver kinds for a site's BMC drivers, provision's `when:` expressions and the host spec
  it builds, and the vps self-check that trusts `bc.dev/cap.instance.network-policy`. A `Hello` whose `bc.dev/`
  labels match an operator row left from before c1fd430 takes that row over as `system`.

* **The tenant's image is hostile to the host.** Both bcd runtimes read every `qcow2` base image's header in Go,
  without `qemu-img` or any other tool probing it, and refuse one that names a backing file or an external data
  file, sets an unknown incompatible feature, is encrypted or is not a qcow2; `raw` and `iso` images are opened with
  their format stated and never probed (security review 12, 933c51b; the Go check since its verification,
  b12c217). An instance's own disk must name exactly its recorded base before every start, snapshot and resize, since
  a VMM can rewrite its header, and a restored snapshot's disk must name none. On `cloud-hypervisor` each VMM runs in
  a confined `bcd-ch-<id>.service` unit as its instance's own uid, with no capabilities and no network (bcd hands it
  its taps as file descriptors), `/run` and `/var` hidden, only KVM and VFIO devices, only its own instance directory
  and base image visible, and limits from its flavor; a memory snapshot restores only into the same vCPUs and
  memory, with its configuration rebuilt from an allowlist. See [the runtimes](/docs/daemon/delivery#the-runtimes).

* **The guest is pinned too.** With `vps({ sandboxExec: { certifyHostKey } })`, each guest presents a host key the
  sandbox CA certified (its private key only in sealed user data), and `sandbox exec` checks it with
  `@cert-authority` and `StrictHostKeyChecking=yes`. `hostCertified` is reported only while the guest's certificate
  (30 days, never renewed) still holds, and a sandbox's lifetime, extensions and memory restores never run past it; a guest without a valid one is refused unless `--accept-host-key` asks for trust on first
  use for that connection. `sandbox exec` also runs ssh with `-F none`, so no user or system `ssh_config` applies.

See [sandboxes](/docs/guides/vps/sandboxes#run-code-inside).

### Tenant isolation [#tenant-isolation]

* **Pre-check first.** A caller outside the input's tenant and the root tenant gets `ACCESS_DENIED` before any
  database access or IAM decision, identical to an IAM denial, so there is no existence oracle and outsiders put no
  traffic on another tenant's audit chain.
* **No ambient tenant.** Tenant-scoped repository functions take `tenantId` as a required parameter.
* **Attributes from the projection.** IAM decides on attributes from compute's own
  authorization projection, never from the request.
* **Idempotency keys** are scoped to `(tenant, identity)`, so one tenant cannot replay another's request.
* **Provider accounts are their tenant's.** A provider pool runs only on an account of its owner tenant
  (`pools.create` refuses another's as `NOT_FOUND`, and vps never uses one, deletion included), an account's
  secret references are its tenant's (see [Sealed secrets](#sealed-secrets)), and only root accounts may use the
  host's ambient cloud credentials. Tenants a pool is offered to do not see its account. In 0.1.x a tenant could
  point a pool of its own at another tenant's account: see the CHANGELOG's upgrade note.
* **Platform-issued compute.** Only a root caller acting in another tenant makes someone other than itself the
  owner of an instance (`instances.create` and `attach` with `ownerIdentityId`), the owner must be a member of
  that tenant when the IAM port can say (`IamPort.isMember`), and label-driven auto-attach takes an owner label
  only on platform accounts, for members. A tenant's own account attaches only for itself, and
  `providerAccounts.machines` there does not name another tenant's instances. See
  [platform-issued compute](/docs/guides/vps/providers#platform-issued-compute).
* **Tenant-controlled machines.** Under a dedicated-root lease the node's identity is reissued as `node-tenant`,
  and its inventory becomes advisory.
* **Suspended and deleted tenants.** Once the IAM bridge's `tenant` reaction (`tenantStateHandler`) mirrors a
  tenant's state, the envelope refuses a suspended tenant's mutations and every call of a deleted one with
  `TENANT_SUSPENDED`, root callers exempt, within 5 seconds in every process. A deleted tenant is then torn down:
  its reservations end, offers to it are withdrawn, its pools archived, its workflows aborted, its private images
  soft deleted, its image signers revoked, its bare-metal machines retired (deployed ones wiped first) and its VPS
  instances deleted. The subtree is read from IAM by the `tenancy` reconciler, outside the inbox transaction; while
  IAM is down the read is retried, so a suspension is delayed, never dropped.
* **Revocations before reactions.** When an identity departs or an agent is suspended, `serve`'s host revokes the
  identity's SSH certificates, closes its tunnels and drops cached authentications first, and only then runs module
  reactions (such as withdrawing an agent's sandboxes), each under its own savepoint (b3f7a81, review 7 M3). A
  reaction that throws is undone alone, so it can never roll the revocation back; failures of identity reactions go
  to `host.iam.onError` and the `IamReconcile` sweep re-runs them, and a failing agent reaction retries its event
  (a3be14c). See [SSH](/docs/guides/access/ssh).

See [tenancy](/docs/guides/iam/tenancy).

### Nothing leaks [#nothing-leaks]

* Unexpected errors answer `500` with `INTERNAL_ERROR` and a generic message; the detail goes only to `log.error`.
* Join, renew and secret refusals are one generic `ACCESS_DENIED` to the agent; the reason goes to audit
  (`security.join.refused`, `security.renew.refused` since 0e42018, `security.secret.refused`), rate-limited per
  reason.
* A report a node sends about another node becomes `security.node-mismatch` and never reaches a handler, and a
  report about another node's allocation is ignored.
* `/.well-known/better-compute` publishes only public key members; a private JWK member never passes.
* The PKI bundle is sealed at rest, and `pki show` reads only its public half.
* `bcd` reads its join token from `BCD_JOIN_TOKEN` or standard input, never from arguments, and never echoes a
  positional argument.

### Refused renewals [#refused-renewals]

Until 0e42018 a refused `Renew` was only a `renew refused` line in the server log. JoinService now also publishes
`security.renew.refused` for every `ACCESS_DENIED` renewal, in its own short transaction after the renewal's has
rolled back, so the event survives the refusal. Who sees it depends on how far the caller got:

| The caller                                                                                                   | Tenant                                                                 | `subject`       | Reasons                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proved possession of the current key (the leaf is compute's own, current certificate and `pop_sig` verified) | The node's owner (`nodes.owner_tenant_id`)                             | `node/<nodeId>` | `renew:replay`, and a refused key attestation: `renew:key-attestation-missing`, `renew:key-attestation-ak`, `renew:node-key-*`, `renew:tpm-ak-uri`, `renew:key-statement-unknown`, `renew:key-statement-unsupported` |
| Anyone else                                                                                                  | The IAM root tenant (`rootTenantId`); nothing is published without one | `renew`         | `renew:chain`, `renew:client-time`, `renew:proof`, `renew:csr`, `renew:certificate`, `renew:revoked`, `renew:retired`, `renew:peer`                                                                                  |

The split is deliberate: a node certificate is public, so anyone can present one with a bad proof, and such a
refusal must not land in the owner's event stream or name its node. The payload carries reasons only:

<TypeTable
  type="{
  reason: { type: 'string', description: &#x22;'renew:<reason>' from JoinService ('renew:denied' when the error carried none).&#x22;, required: true },
  nodeId: { type: 'string', description: 'Only for a caller that proved the current key.' },
  gatewayId: { type: 'string', description: 'The gateway that relayed the call, when known.' },
  suppressed: { type: 'number', description: 'Refusals of the same reason dropped by the rate bound since the last event.', required: true },
}"
/>

The rate bound is the one join refusals use (`refusalEventsPerMinute`, default 60 a minute), keyed per renew reason.
A renewal that bcd never sends (its TPM unavailable, or another AK than the certificate names) produces no event;
bcd logs `renewal failed`. `tests/server-join-tpm-renew.test.ts` checks both cases: a missing key attestation is the
owner's event naming the node, and a proof by someone else's key over the node's certificate is the root tenant's,
subject `renew`, with no node.

### The HTTP API [#the-http-api]

* `POST` requests need `content-type: application/json` and `X-Better-Compute: 1`, else `CSRF_REJECTED`. A plain
  HTML form cannot set that header.
* The IAM session cookie is ignored unless `acceptIamCookie` is set. Even then a cookie request needs an `Origin`
  exactly equal to `baseURL`'s origin: a missing one is `CSRF_REJECTED`, another one `UNTRUSTED_ORIGIN`.
* Bodies are JSON objects of at most 64 KiB (`PAYLOAD_TOO_LARGE`). `TRACE`, `CONNECT` and `TRACK` are refused.
* Every answer carries `cache-control: no-store`, `x-content-type-options: nosniff` and `referrer-policy:
  no-referrer`.
* `X-Request-Id` is echoed only when it matches `^[A-Za-z0-9._:-]{1,128}$`, and IAM sees the same id.
* IAM sees the caller's address only as the host verified it: `nodeHandler` passes the socket's peer, a fetch host
  passes `compute.handler(request, { peer: { address } })`, and remote IAM gets that as `x-forwarded-for`. The
  client's own `x-forwarded-for`, `x-real-ip` and `forwarded` are never forwarded, so a caller cannot choose the
  `request.sourceIp` its policies see (e7092bf, d269738, review 14 L3; the MCP server's remote IAM likewise, 7890821).
  Behind a load balancer, `trustedProxies` (0873d20) lists the proxies' addresses or CIDRs: only when the socket peer
  is one of them is the caller read from `X-Forwarded-For`, right to left, skipping trusted hops; a trusted proxy that
  sends no usable header, or a malformed hop, leaves `sourceIp` unset, never the proxy's own address. Since 0eba0ad
  the HTTP handler also strips `X-Forwarded-For`, `Forwarded`, `X-Real-IP` and the other client-address headers from
  what an embedded IAM sees and sets `X-Forwarded-For` to the verified address; since 004579e that happens for
  every credential on every path in (`presentedCredential`, the in-process API included), not only over HTTP, so a
  caller that passes `headers` to `compute.api` cannot choose its address either. The IAM caches key on it: the
  authentication cache (30 s) and allowed read decisions (10 s) are per caller address, so a stolen token used from
  a disallowed address no longer rides its owner's cached entries. See
  [the caller address](/docs/operations/deployment#the-caller-address).

### Delegated approvals [#delegated-approvals]

An agent's held-back request reaches a person as text, so the text must say what the request does, and the approval
must admit only that (security review 14):

* **Approvers see what they approve.** `reservations.create`, `reservations.extend`, `instances.create`,
  `sandboxes.create` and `sandboxes.extend` describe themselves (`MethodAdmission.describe`, ec36c63): pool, devices,
  lifetime, image, SSH keys as fingerprints, user data by size and digest, the snapshot a sandbox restores. The text
  is deterministic, and caller strings are clipped to one line with control characters removed.
* **An approval admits one request.** The envelope records a digest of that text per confirmation, keyed by
  tenant, delegation, action and resource (`confirmations/0001_bindings`), and a delegated call whose text differs
  is refused with `ACCESS_DENIED` (`approval-mismatch`), even while Better IAM's approval for that action and
  resource is still open (06d7f04, M1).
* **An agent's batch reads as the agent's.** A destructive batch an agent proposed records the agent and its
  delegation (`quorum/0008_batch_agent`, f3c5c4e), and `batch approve`, `batch show` and the console say "proposed by
  agent X for Y" (bd80209, M3).
* **The MCP gate stays out of confirmations.** Build it with `createMcpGate({ confirmations: false })` (daf58eb, M2);
  its rules accept only well-formed ids, `create_instance` takes no SSH keys (7890821, H1), `doctor` flags an
  ungated `host.mcp` handler (`host.mcp.ungated`, b491aed, L6), and one caller has at most 4 tool calls in flight
  (48d41a4, L5).

See [delegated confirmation](/docs/guides/iam/delegated-confirmation) and [MCP](/docs/guides/clients/mcp).

### Rate limits and admission [#rate-limits-and-admission]

| Where                             | Limit                                                                                                                                    |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Gateway handshakes                | A token bucket of 200 a second                                                                                                           |
| Gateway sessions                  | `maxStreams` (default 5,000)                                                                                                             |
| Pending `Hello`s                  | 64; more are refused `RESOURCE_EXHAUSTED` so a reconnect storm cannot starve the lease                                                   |
| Duplicate `Hello`s                | More than 3 from a second machine within 10 minutes quarantine the certificate (`createGateway({ duplicateHellos: { max, windowMs } })`) |
| Tunnels, break-glass certificates | 10 a minute and 20 an hour per identity, per api replica                                                                                 |
| Better IAM                        | A 429 from IAM becomes `RATE_LIMITED`, with `Retry-After`                                                                                |
| Security events                   | Join, secret and workload-token refusal events: 60 a minute per reason by default; the rest are counted as suppressed                    |
| Workload tokens                   | 30 a minute per instance by default (`workloadIdentity.perInstancePerMinute`)                                                            |

### Tamper-evident audit [#tamper-evident-audit]

Every allowed mutation writes an `audit_events` row with its admission context in the same transaction (composite
methods write one per sub-call). So do node joins and reclaims, secret releases by the broker, and workload tokens
issued to instances (`workload.token.issued`). Commands, workflow steps and exec tickets are not audit rows today;
they are recorded in their own tables and events. With `auditModule`, an hourly job seals new rows into signed,
hash-chained Merkle checkpoints per tenant (RFC 6962), in commit order so a long transaction cannot land behind a
checkpoint. `better-compute audit verify --tenant ID` recomputes the chain and finds an edited, deleted or inserted
row, a removed checkpoint or a foreign signature; `audit prove` writes an inclusion proof a third party checks
offline with `audit check-proof`. The module's self-check warns when rows older than 3 hours are still unsealed. An
audit sink (`sinks.audit`) copies every committed row, in commit order, to a store the database's operators cannot
change; see [sinks](/docs/extending/sinks#audit-sinks).

## Hardening checklist [#hardening-checklist]

      ### Secrets and keys [#secrets-and-keys]

    * A deployment secret of at least 32 random characters, from a secret manager, identical in every process.
    * `pki init` with the secret (never `--plaintext` outside development) and `--offline-root`, with the root key
      kept off every server.
    * The bundle, the secret and the root key backed up, separately.
    * A bundle sealed in the former format resealed once: `pki reseal` prints its pins, you compare them with your
      records, then `pki reseal --accept-unauthenticated`. `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED` unset afterwards;
      the commands that write the bundle refuse it anyway. See [PKI](/docs/operations/pki#pki-reseal).
    * Machines installed with their pins, taken from the sealed bundle (`bootstrapPins(bundle)` into provision's user
      data and `vps({ bootstrap: { pins } })`): a host build of `bcd join --server` without `config.toml` requires all
      three and refuses a `/.well-known/better-compute` whose pins differ, since only the control plane's TLS
      certificate vouches for that document. Only fakehw builds join on the document's word (trust on first use). See
      [installing bcd](/docs/daemon/install#joining-from-the-well-known-document).
  
      ### Network [#network]

    * api behind an L7 load balancer that terminates TLS; `baseURL` is exactly the public origin.
    * Gateways behind an L4 load balancer with TLS passthrough. Machines need only outbound 443.
    * Internal control APIs (`host.gateway.internal`) reachable only from the control plane, over internal-CA mTLS.
    * `GET /metrics` only with a `bearerToken`, and not exposed publicly.
    * BMC networks reachable only from site agents.
    * `policy.trustProvisioningNetwork` only on pools whose provisioning network no untrusted host can reach; elsewhere
      confirm each discovered PXE MAC with `metal.confirmMac`.
  
      ### Configuration [#configuration]

    * Better IAM wired through the bridge; `doctor` reports `iam.not-configured` otherwise.
    * `acceptIamCookie` only for a console on the same origin.
    * `pki.commandSigning` set, so commands and NodeSpecs are signed and GPU bindings are attested.
    * `tunnels` only if you need interactive access, with the SSH CA's MFA requirement left on.
    * `auditModule` registered with the bundle's audit key.
    * Nodes that run no bcx plugins installed with `daemon install-script --unprivileged`, so `bcd node` is not root,
      with `command_root_sha256` pinned so `bcd helper` checks host units against signed specs from the first request.
    * `trustedProxies` set to your load balancers when IAM policies condition on `request.sourceIp`.
    * On TPM 2.0 fleets, `tpmAttestor` left at `nodeKey: 'certified'`, with `policy.joinAttestors: ['tpm']` on the pools
      that must take no other kind of join, so every node there starts with a node key its TPM holds (see
      [tpm](/docs/guides/nodes/attestation#tpm)).
  
      ### Operations [#operations]

    * `better-compute doctor --strict` in CI and after every deploy.
    * Gateway serving certificates reissued before their 30 days end, and CA expiry watched (`doctor` warns 30 days
      ahead).
    * Revocations checked in practice: `nodes.revokeIdentity` should close the node's session within seconds.
    * `audit verify` run on a schedule for the tenants that matter.
  
## Reporting a vulnerability [#reporting-a-vulnerability]

The repository has no security policy file or dedicated disclosure address yet. Until it does, report a
vulnerability privately to the maintainers rather than in a public issue, and include the version, the
configuration involved, and steps to reproduce.


# Testing (/docs/operations/testing)

> The testing kit - a control plane on PGlite, the FakeDaemon, faulty links, the fleet simulator and its invariants, real-time harnesses and benchmarks.



`@better-compute/testing` (also `better-compute/testing`) is the kit the repository's own tests run on, and the
one you use to test modules, plugins and policies. It gives you a migrated control plane on PGlite with a clock
you can move, a scripted IAM, the daemon side of the agent protocol (`FakeDaemon`), in-memory links with fault
injection, a deterministic fleet simulator with invariant checkers, and real-time harnesses for latency
measurements. No database server is needed.

| Layer                                  | Exports                                                                                                                          | Clock       |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| Control plane fixture                  | `computeFixture`, `closeFixtures`, `fakeIam`                                                                                     | sim or wall |
| Daemon side                            | `FakeDaemon`, `createDaemonHarness`, `fakeHardware`, `MemoryJournal` and the journal rules                                       | sim         |
| Links                                  | `createAgentLink`, `createFramePair`, `FrameConn`                                                                                | sim         |
| Fleet simulator                        | `createFleetSim`, `GatewayStub`, `checkI1`, `checkI2`, `checkI3`, `checkI4`, `checkI5`, `checkI7`, `checkI8`, `assertInvariants` | sim         |
| Real control plane under the simulator | `createControlPlaneSim`, `controlPlaneFacts`, `runNightly`                                                                       | sim         |
| Wall-clock harnesses                   | `createRealtimeHarness`, `createRealtimeFleet`, `probeLeases`, `createH2cClient`                                                 | wall        |
| Local stack                            | `createDevStack`, `attachDevFleet` (see [local development](/docs/operations/local-development#the-dev-stack-in-code))           | wall        |
| Plumbing                               | `createSimScheduler`, `createRealtimeScheduler`, `createRng`, `EventLog`                                                         | either      |

The kit needs the optional peer `@electric-sql/pglite` as a devDependency; without it `computeFixture()` fails
with `MISSING_DEPENDENCY`. The embedded Better IAM of `@better-compute/iam/testing` needs the optional
`@better-iam/adapter-sqlite` peer.

## Test a module with computeFixture [#test-a-module-with-computefixture]

`computeFixture(options?)` builds `betterCompute()` on an in-memory PGlite, runs `compute.initialize()` (the
registry check, the base schema and every module's migrations) and hands you the pieces. It does not start the
workers: your test drives reconcilers one pass at a time with `compute.jobs[name].tick()`.

```ts title="my-module.test.ts"
import { afterEach, expect, it } from 'vitest';
import { closeFixtures, computeFixture } from 'better-compute/testing';
import { myModule } from '../src/module';

afterEach(closeFixtures);

it('asks IAM before listing, and reconciles', async () => {
  const { request, iam, tenants, compute, advance } = await computeFixture({ modules: [myModule()] });
  iam.allow({ token: tenants.acme.token, action: 'compute:nodes:read' });

  const answer = await request('nodes', 'list', { tenantId: tenants.acme.id }, { token: tenants.acme.token });
  expect(answer.ok).toBe(true);
  expect(iam.calls.length).toBeGreaterThan(0); // authenticate, then the tenant decision

  await advance(60_000); // compute's Clock and compute_now() move together
  await compute.jobs['my.reconciler']!.tick(); // one deterministic pass
});
```

| Field                                      | What it is                                                                                                                                                                |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `compute`                                  | The initialized control plane, typed by `modules` and `plugins`                                                                                                           |
| `database`                                 | The migrated PGlite `SqlExecutor`                                                                                                                                         |
| `clock`, `advance(ms)`                     | The fixture clock                                                                                                                                                         |
| `iam`                                      | The `fakeIam()` port compute calls                                                                                                                                        |
| `tenants`                                  | `root`, `acme` and `other`, each `{ id, token, caller }`; ids `tenant-root`, `tenant-acme`, `tenant-other`                                                                |
| `request(group, method, input?, options?)` | `POST {basePath}/{group}/{method}` through `compute.handler`, with the CSRF headers; `options.plugin` routes to `plugins/{plugin}/…`, `options.headers` replaces defaults |
| `close()`                                  | Stops compute and closes the store; idempotent                                                                                                                            |

Options: `clock` (`'sim'`, the default, or `'system'`), `start` (default 2026-01-01T00:00:00Z), `iam` (extra
`fakeIam()` options), `modules`, `plugins` and `log`. `request()` answers `{ ok, status, headers, requestId }` plus
`data` or `error`.

**The sim clock.** Under `clock: 'sim'`, compute's `Clock` and SQL's `compute_now()`
read the same time (principle P9). `advance(ms)` and `clock.set(ms)` move the `compute_sim_clock` table first and the
TypeScript clock second, so leases, `not_before`, column defaults and admission contexts all move together. Time
never moves backwards. Under `'system'`, `advance` rejects with `INVALID_STATE`.

**`fakeIam(options?)`** is a scripted `IamPort`. Bearer tokens map to callers (an unknown token is
`UNAUTHENTICATED`). A decision is allowed only when an `allow` rule matches and no `deny` rule does; a rule matches
on any of `token`, `tenantId`, `action`, `type` and `id`, and `{}` matches everything. Every call is recorded in
`calls`; `count(method)` counts them. `billingCheck` allows, `recordUsage` records, `featureEnabled` is false and
`requestConfirmation` answers `'not-held-back'`.

Plugins, drivers and providers have **conformance suites**: functions that return `ConformanceCase[]`
(`{ name, run(subject) }`, from `@better-compute/core`), such as `providerConformanceCases()` from
`@better-compute/vps` and `storeConformanceCases()` from `@better-compute/postgres`. Run them from any runner:

```ts
for (const test of providerConformanceCases()) it(test.name, async () => test.run(await createSubject()));
```

## Real Better IAM in tests [#real-better-iam-in-tests]

`computeFixture()` still uses the scripted `fakeIam()`. When a test must exercise real policies, use
`createEmbeddedTestIam(options?)` from `better-compute/iam/testing`: a real better-iam on in-memory SQLite, carrying
compute's plugin, resolver and inbox, bootstrapped with a root administrator (TOTP MFA) and two organizations,
"Acme" and "Other". Pass its `port` to `betterCompute({ iam })` and call `bind(sql)` once the compute store is
migrated.

* Each organization (`acme`, `other`, or a new one from `tenant(name)`) offers `member(name)`, `signIn(name)`,
  `policy(document)`, `grant(identityId, ...documents)` and `share(type, id, relation, subject)`.
* Options: `now`, `sql`, `sessionJwts`, `runIamJobs`, `inbox` (`'store'` or `'memory'`), `iamOptions`, and the
  catalog's `modules`, `plugins`, `extra` and `typePrefix`.
* `mode`: `'embedded'` (the default, or `BETTER_COMPUTE_IAM_MODE`) or `'remote'`, which hands compute
  `bridge.remote()` over the same better-iam's HTTP handler, so the suite runs as it would against IAM in another
  process.
* The secret is fixed and public. Never use it outside tests.

`pnpm test:iam-remote` reruns the IAM-dependent suites in remote mode (`vitest.iam-remote.config.ts`, 13 files: the
example policies, the resolver timeout, confirmation, the testing IAM, streams, MCP confirmation, watch, the e2e
access and spine tests, the authz matrix, the budget check, webhook intake and `serve`'s compose file), and
`pnpm check` runs it after `pnpm test`, so every change stays green in both IAM modes (a v0.4 exit). The
`server-*`, `capacity-*` and `vps-*` suites use the scripted `fakeIam()` port, so an IAM mode does not apply to
them. `pnpm check:head` runs `scripts/head-check.mjs`, which checks the committed tree in a separate worktree.

`scripts/head-check.mjs` exists because several sessions edit the shared checkout at once, so a full `pnpm check`
there mostly reports other people's work in progress. It keeps a git worktree under the temporary directory
(`bc-head-check`), moves it to the commit (`--ref`, default this checkout's `HEAD`), links every `node_modules` of the
checkout into it without installing anything, builds with `tsc -b` unless `--no-build`, and runs `vitest run` with
the filters you give. Each concurrent run takes its own tree (`bc-head-check`, then `-2` up to `-8`) through a
`<tree>.lock` file holding its pid, taken over when that pid is dead and released on exit, so parallel runs never
move each other's tree mid-run. `--run <repo-relative .mjs> [args…]` runs that script in the held tree after the
build instead of vitest, and exits with its status; the docs reference is regenerated from committed code this way:

```bash
node scripts/head-check.mjs tests/vps-sandboxes.test.ts
node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD"
```

The authz matrix has two halves. `tests/generated/authz-matrix.test.ts`, generated by `scripts/authz-matrix.mjs`,
covers the envelope side, one test per registered method. `tests/iam-bridge-matrix.test.ts` covers the IAM side:
every distinct (action, resource type) pair of the PLAN.md §9.2 table, alternatives included, is decided through
compute's own `IamPort` against a real better-iam carrying the bridge's plugin and resolver. Each pair is denied
without a grant, allowed with one, and never allowed on a resource type it was not granted on. Under `pnpm test`
the port is `bridge.embedded`; under `pnpm test:iam-remote` it is `bridge.remote` over the same IAM's HTTP API, and
the test asserts that the decisions went over HTTP.

`spyIamPort(port)` from `better-compute/iam` wraps any `IamPort` and records every call. Code paths tag themselves with
`spy.path(name, fn)`; each call is attributed to the innermost path across awaits, so a test can assert that fleet
paths never reach IAM. That is how the v0.1 security negative is tested: `tests/sim/iam-negative.test.ts` runs ten
simulated minutes of heartbeats, inventory, commands and a reservation, and asserts that every IAM call came from an
API admission. See [policies](/docs/guides/iam/policies).

## The FakeDaemon [#the-fakedaemon]

`FakeDaemon` is the daemon side of the protocol in TypeScript, rule for rule with the Go packages in
`daemon/internal/{journal,dispatch,session,converge,agent,fakehw}`. The real bcd has its own tests (see
[the Go daemon](#the-go-daemon)); the FakeDaemon is what runs where features bcd lacks, or many daemons on simulated
time, are needed. `bcd sim` runs many real agents too, but in real time against a real gateway.
See [the daemon](/docs/daemon).

* **Journal.** `accepted` before any side effect; `running{boot_id}` right before a self-terminating one; done
  results kept 7 days; unacked events capped at 64 MiB and 10,000. A crash keeps the journal, a reboot also changes
  `boot_id` and clears `/run`, and `loseJournal()` starts a new `journal_id`.
* **Duplicate delivery.** Done: resend the stored result. Running: re-attach. Accepted but not running after a
  crash: re-execute if `retry_safe`, else `FAILED{interrupted}`. Self-terminating across a reboot:
  `SUCCEEDED{rebooted}`.
* **Acceptance order.** Another node's command or a disallowed kind is `UNSUPPORTED`; a `min_generation` above the
  applied spec defers with a `SpecRequest`; past `not_after` is `EXPIRED`; no free lane credit is `BUSY`.
* **Session.** Hello from the journal, Welcome, heartbeats, PING liveness, full-jitter reconnect, `Drain`,
  `RevocationNotice`, `Cancel`; a `Welcome` more than 5 minutes before the journal's server time mark is refused.
  The mark (`JournalMeta.serverTimeMs`) is set on every Welcome and heartbeat and only moves forward, so a gateway
  cannot rewind the unsigned `Welcome.server_time` to make a recorded signed command or an old worker certificate
  valid again; the session drops with `server time went backwards` on every redial. It is not clamped to the local
  clock, since sim and manual server clocks sit far from the host's.
* **Desired state.** The devices converger writes the CDI spec and env file in an in-memory file map, reports
  `AllocationBound`, `AllocationReleased` and `AllocationFailed`, and never drops a hold on its own.
* **Kinds.** `diag.nvml`, `inventory.refresh`, `logs.collect` and `node.reboot`, plus the opt-in `fakeGpuReset`
  (not retry-safe). `knobs` make them fail: `diag.nvmlInitFails`, `diag.failGpus`, `diag.error`,
  `logs.uploadFails`, `cdiWriteFailures`.
* **Beyond v0.1.** With `intent`, it verifies signed commands (`BAD_SIGNATURE`; with command-signing roots every
  command must carry a valid signature, whatever its kind, 12ccfa1, which `tests/testing-daemon-intent.test.ts`
  checks with an unsigned non-destructive command) and signed NodeSpecs:
  `intent.signedSpecs` is `'always'` (the default: with command-signing roots every spec must be signed, since an
  unsigned spec with a high generation would freeze all later ones) or `'access'` (only a spec that changes `access`
  must be signed, for a control plane that signs nothing else), and refused specs are listed in `specRejections`.
  `intent.quorum` (a `FakeQuorumTrust`: approver keys, threshold, tiers, `unbatchedPerHour`), or the quorum of a
  newer signed trust bundle, enforces the v1.0 destructive-batch quorum: destructive commands are rejected
  `BATCH_UNAPPROVED` or `UNBATCHED_LIMIT`, and a newer trust bundle that changes the quorum without the current
  approvers' signatures for the FakeDaemon's own trust domain is ignored (logged `trust.ignored`; 6a7f1b7, 4835f4f) (see
  [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)).
  A refused session whose close carries `retryAfterMs` (the gateway's retry pushback) reconnects after exactly that
  long. `setPressure(pressure)` puts a `HostPressure` into every status report. `setDiscovered(machines)` (537e672)
  plays a site agent's PXE discovery list: each `{ mac, productUuid?, arch?, firstSeen?, lastSeen?, count?, relay? }`
  (times in epoch milliseconds, default now; MAC and UUID lower-cased) goes into `StatusReport.discovered` of every
  report from then on, the current report is re-sent, and `undefined` stops it. With `tunnels`, it answers `OpenTunnel`; with `nodeKey`, attested
  results and allocation reports carry `node_sig`. With `instances` (a `FakeHypervisor` or its options) it plays a
  VPS host, advertising `instance.network-policy` in `Hello` and converging domains, overlay disks, NoCloud seeds and taps and fetching user data through the secret
  broker; as such a host, `handleImds(request, instanceId)` answers a running instance's metadata request (v0.5,
  a `FakeImds`), minting its workload token through a `WorkloadTokenRequest` on the session. With `selfUpdate: { badVersions?, swapMs? }` it converges `NodeSpec.daemon` by swapping and restarting,
  and a bad build falls back to its previous slot and reports the section failed. `fakeDriverInvoke({ drivers })`
  adds the `driver.invoke` kind, running in-process or bcx drivers. `createWsDialer()` dials the WebSocket fallback.

| Option                            | Default                          | Meaning                                                             |
| --------------------------------- | -------------------------------- | ------------------------------------------------------------------- |
| `nodeId`, `scheduler`             | required                         | The certificate's node id; the scheduler that runs its timers       |
| `dial`                            | none                             | Opens a Session stream (`FrameConn`); `undefined` means unreachable |
| `hardware`                        | `fakeHardware({ seed: nodeId })` | The board it reports                                                |
| `role`                            | node                             | `AgentRole`                                                         |
| `kinds`                           | the four defaults                | Merged over the defaults; `null` removes one                        |
| `workers`                         | control 4, provision 2, bulk 1   | Worker pool per lane                                                |
| `journal`                         | 64 MiB, 10,000 events            | Journal caps                                                        |
| `backoff`                         | 1 s to 60 s                      | Reconnect back-off with full jitter                                 |
| `pingIntervalMs`, `pingTimeoutMs` | 20 s each                        | HTTP/2 PING                                                         |
| `welcomeTimeoutMs`                | 30 s                             | A session without Welcome is dropped                                |
| `rebootDurationMs`                | 60 s                             | How long a reboot keeps the node down                               |
| `clockSkewMs`                     | 0                                | The daemon's clock minus the server's                               |

**Fake hardware.** `fakeHardware({ seed, variant, missingGpu? })` gives `hgx-h100` (8x H100 SXM5 80GB on NVSwitch,
field for field the Go `fakehw`: the same seed gives the same section hashes), `hgx-h100-gpu-missing` (7 visible, so
`GpuCountMismatch`), `l40s` (4x L40S 48GB on PCIe) and `cpu` (no `gpus` section). `encodeInventorySections()`
reproduces the Go golden bytes.

**Links.** `createAgentLink()` builds a daemon end and a gateway end whose frames cross as protobuf bytes, with
faults per direction from the link's seeded RNG: `latencyMs` (default 1), `jitterMs`, `drop` with `dropMode`
(`silent`, or `reset`, which breaks the stream as gRPC would), `duplicate`, and `reorder` within `reorderWindowMs`
(default 50). Controls: `stall`, `partition` and `heal`, `halfOpen`, `reset`, `dropNext`, `duplicateNext` and
`intercept`. `createDaemonHarness()` wires one daemon to scripted servers on a sim scheduler.

**Host volumes** (38405a0). A `FakeHypervisor` keeps [sandbox volumes](/docs/guides/vps/sandboxes#volumes) the way bcd
does, so a FakeDaemon VPS host can stand in for bcd in volume tests:

* `InstancesSection.volumes` are created, or grown in place, only when the store has room (available minus reserved
  minus outstanding); otherwise the volume keeps its size and the instances naming it fail. A volume is never shrunk,
  and one no longer listed is destroyed once nothing has it open. The files would be `volumePath(id)`,
  `/var/lib/bcd/volumes/<id>.raw`.
* An instance's `InstanceSpec.volumes` are opened at boot, and what counts is what running and paused domains really
  have open, not their specs: one writer, and read-only volumes shared only with other readers. A pass applies stops
  before starts, so a volume handed from one instance to another is closed before it is opened again.
* Volumes survive `powerLoss`; `wipe()` clears them. The hooks `writeVolume(instanceId, volumeId, content, bytes?)`
  (a content marker, 1 MiB written by default), `volumeContent(volumeId)` and `openBy(volumeId)` let a test check
  what a guest would find and who has a volume open.
* With `volumeStore: { totalBytes, reservedBytes? }` in its options, `volumeStorage()` feeds
  `StatusReport.volume_storage`, which the FakeDaemon sends in every status report: available is total minus what
  guests wrote, reserved defaults to bcd's 5% of the total with a 2 GiB floor, committed is the sum of the sizes,
  and outstanding the sum of size minus written.

**Host admission** (6922553). A start the `FakeHypervisor` has no free vCPU or memory for is no longer an instance
failure. Like [bcd's host admission](/docs/daemon/delivery#instances) (`agent/admission.go`), the domain stays
`stopped` with a message starting `ResourcesExhausted:` (the prefix is exported as `RESOURCES_EXHAUSTED`), and the
section is `PROGRESSING` with reason `Retrying` and the message `ResourcesExhausted: this host cannot start <ids>
now`, unless another instance failed in the same pass (then it is `FAILED`, `InstanceFailed`). The next pass tries
the start again. That is the report vps turns into `HostResourcesExhausted` and a re-placement (see
[hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance)).

`tests/testing-daemon-instances.test.ts` has the unit test.

**A VPS host under `bcd helper`** (1c9684a). The FakeDaemon plays the control-plane side of a bcd that runs
unprivileged under [`bcd helper`](/docs/daemon#privilege-separation), with `cloud-hypervisor` through root's `ch.*`
operations:

* **Runtimes in `Hello`.** A FakeDaemon with `instances` lists each of its `FakeHypervisor`'s runtimes as a driver
  manifest of kind `runtime` (`runsOn: NODE`, its daemon version), as bcd does for its built-ins, so the gateway labels
  the node `bc.dev/driver.<runtime>`. An explicit `drivers` option replaces the list.
* **`FakeHypervisorOptions.helper: { instanceBridges }`** plays `[helper] instance_bridges`: a NIC on any other
  bridge (an empty list refuses every NIC) fails its instance in the helper's words, `libvirt-kvm` before the domain
  exists (`domain: bridge "…" is not an instance network`) and `cloud-hypervisor` when it starts (`ch.taps: "…" is not
  an instance bridge of this host ([helper] instance_bridges)`). `helperVpsHost(options?)` is the preset:
  `{ runtimes: ['cloud-hypervisor'], helper: { instanceBridges: ['br0'] } }`, merged under `options`.
* **Live usage.** `FakeHypervisor.usage()` returns each domain's usage as bcd's `runtime.UsageReader`s read it:
  `cloud-hypervisor`'s managed domains that are running or paused, `libvirt-kvm`'s running ones, none for other
  runtimes. The default is fakehw's synthetic usage (a CPU second per boot, half the configured memory, zero I/O and
  network); `setUsage(instanceId, usage)` merges `cpuSeconds`, `memoryBytes`, `ioReadBytes`, `ioWriteBytes`,
  `netRxBytes` and `netTxBytes` over it (`FakeUsage`), and the domain's next boot clears it.
* **`FakeDaemonOptions.telemetry: { export(batch) }`** receives the OTLP export the `telemetry` section runs, never
  over the Session. While the section names an `otlp_endpoint`, the FakeDaemon exports once when the section applies,
  then every `export_interval_ms` (default 60 s, at least 5 s); an unchanged section keeps its export and interval, a
  changed one restarts it, and an empty endpoint stops it. Each `FakeTelemetryExport` has `nodeId`, `endpoint`, `at`,
  a `resource` (the section's `resource_attributes`, then `service.name=bcd`, `service.version` and `bcd.node.id`,
  which win) and `metrics` from `instanceMetrics()`: `bcd.instances`, `bcd.instance.vcpu`, `bcd.instance.memory` and
  the live `bcd.instance.cpu.time`, `memory.used`, `io.read`, `io.write`, `network.receive` and `network.transmit`
  (`FakeMetric`, `FakeMetricPoint`). `headers_ref` is not fetched, and the GPU and host series are not modelled.
  Without the option nothing is exported; the section converges either way.
* **`DevJoinOptions.daemon`** gives one `dev.join` its own FakeDaemon options over the stack's, such as
  `{ instances: helperVpsHost() }` for helper hosts beside root ones. A `FakeHypervisor` instance passed there would be
  shared by every machine of the join, so pass options.

`tests/testing-helper-vps-host.test.ts` covers these on a harness. `tests/e2e/helper-vps.test.ts` runs 12 sandboxes
(`E2E_HELPER_SANDBOXES`) on two helper hosts through the whole stack, with `nodeTelemetryModule` pointing the hosts at
a fake collector: both hosts carry the system label `bc.dev/driver.cloud-hypervisor`, every sandbox reaches `ready`,
each host's next export carries its sandboxes' usage, a guest's `setUsage` shows up in the export after, and deleting
them leaves no leak and no usage in the next export.

Since 0fca413 vps places every hosted instance, built-in runtimes included, only on a host carrying
`bc.dev/driver.<runtime>`, so these labels are what lets the sandboxes place at all. The PGlite vps suites seed the
labels a current bcd's `Hello` writes, and `tests/vps-sandboxes.test.ts` checks that a `cloud-hypervisor` sandbox
waits, `Unschedulable` naming the label, until a host advertises the runtime.

**Device capabilities.** A FakeDaemon with GPUs advertises `cdi`, `nvml`, `devices.limits` and `binding.kubernetes`
(409f783, 2782094), so capacity places host-resource and Kubernetes reservations on it. A skew profile or an explicit
`capabilities` list replaces the defaults, so an older-bcd profile gets neither, as it should (see
[capabilities](/docs/daemon/sessions#hello)).

### Volumes at scale [#volumes-at-scale]

`tests/e2e/volumes.test.ts` (38405a0) is the v0.5 zero-leak churn applied to volumes, the "agent memory" of PLAN.md
§3.8. It runs the whole stack, the real server runtime and the real gateway, with three FakeDaemon VPS hosts on the
`cloud-hypervisor` runtime, each reporting a 150 GiB volume store with 10 GiB reserved. It needs no Go and no WSL.

1. It waits until every host has reported its resources, its volume store (`host_volume_storage`) and the
   system-sourced `bc.dev/cap.instance.volumes` label.
2. It creates 12 volumes concurrently, then an over-asking round, some of which must fail `CAPACITY_UNAVAILABLE`.
3. It makes a read-write template per volume and shared read-only ones.
4. It churns `E2E_VOLUME_SANDBOXES` sandboxes (40 by default since df4b371; the nightly `guarded` job sets 240):
   created (some refused `CONFLICT` beside a live
   writer), deleted, expiring at 60 s, writing markers, volumes growing, two host reboots and two volumes retired.

Mid-churn and at the end it checks that:

* the v0.3 no-leak diff is clean (domains, overlays, taps, IP leases, bookings);
* the hosts' volumes and `sandbox_volumes` agree, at the booked size;
* a read-write volume never has two writers, on the hosts or in `volume_attachments`;
* every sandbox runs on its volumes' host, with them open;
* headroom never goes below zero and no host's store is over-promised;
* the markers are on the hosts, and a new sandbox of a volume's template finds the last writer's marker after the
  reboots.

Deleting everything leaves no host volume and no volume booking. It passed the first time, but at 240 sandboxes it
took about 9 minutes, much of it deletions waiting on periodic passes: a deleted
sandbox stayed `deleting` for up to 15 seconds, until the next `vps.sandboxes` pass, and a deleted volume up to 30
seconds, until the next `vps.volumes` pass. Since df4b371 both finish when their instance or host does: the instance
reconciler wakes `vps.sandboxes` when it deletes a sandbox's instance, and the volume-store status handler wakes
`vps.volumes` for each `deleting` volume whose host has applied a later spec, in the same transaction (see
[sandboxes](/docs/guides/vps/sandboxes#read-extend-and-delete)). The run now takes about 43 seconds at 240 sandboxes
and 11 at 40, so the default suite runs 40 and the nightly `guarded` job (`.github/workflows/nightly.yml`) sets
`E2E_VOLUME_SANDBOXES=240`. Its hosts are FakeDaemons; bcd's own volume handling is covered by its Go tests (see
[the Go daemon](#the-go-daemon)).

## The Go daemon [#the-go-daemon]

bcd has its own tests in `daemon/` (Go 1.26 or later) and one end-to-end suite against the real control plane.

**Unit and interop tests.** `node scripts/daemon-test.mjs` runs the module's tests with the tags `nonvml,fakehw` (no
cgo, no NVIDIA driver, the fake HGX node). On Linux it is `go test ./...`. On Windows, where bcd cannot run and
Application Control may refuse locally built executables, it cross-compiles each package's test binary for
linux/amd64 into `daemon/bin/tests/` and runs it inside WSL from the package directory, so `testdata` paths resolve.
The pki tests read `daemon/internal/pki/testdata/interop.json`, which `node scripts/daemon-fixtures.mjs` writes with
`@better-compute/pki` (a node certificate, a signed `JoinResponse`, a sealed JWE, signed commands, specs and a trust
bundle, an exec ticket, and secret and workload-token exchanges), so Go is checked against the TypeScript
implementation byte for byte. The tunnel tests run a shell on a real pseudo-terminal (a resize is seen by `stty`),
and the `access` tests check the generated KRL with `ssh-keygen -Q` when `ssh-keygen` is installed.

```bash
node scripts/daemon-test.mjs                          # every package
node scripts/daemon-test.mjs --run TestSweep -v ./internal/journal
node scripts/daemon-test.mjs --build-bcd              # also writes daemon/bin/bcd-linux-amd64 (fakehw)
```

| Flag or variable                  | Meaning                                            |
| --------------------------------- | -------------------------------------------------- |
| `--run REGEXP`                    | Only tests matching the pattern                    |
| `-v`                              | Verbose output                                     |
| `--distro NAME`, `BCD_WSL_DISTRO` | The WSL distribution on Windows (default `Ubuntu`) |
| `--build-bcd`                     | Also build the fakehw `bcd` for linux/amd64        |
| `packages...`                     | Go package patterns (default `./...`)              |

**bcd helper as root.** `node scripts/helper-e2e.mjs [--distro NAME] [-v]` (da8751e) is the one test that runs
`bcd helper` as real root. It builds bcd and `internal/helper/roote2e` with `-tags nonvml,helpere2e` (which gives bcd
an e2e-only `helper-call`, so the node's requests come from the installed binary as the `bcd` user), renders the
install script with `--unprivileged --instance-bridges bcbr0`, and runs `TestHelperAsRoot` as root under
`unshare --mount --net`, over overlays of `/etc`, `/usr/local` and `/var` and a fresh `/run`, so the host is never
changed (the test fails outside private namespaces). On Windows it runs in WSL as root; on Linux it needs root;
elsewhere, or without `unshare` and `useradd`, it prints why and exits 0. Its steps: the script's unprivileged mode;
the helper started as `bcd-helper.service` runs it; signed host units and their refusals; libvirt against a fake
`virsh`; `cloud-hypervisor` with real taps (b9667d5); and since 035fc41 self-update under the helper, with the
script's `[update]` pinning the two release keys of `daemon/internal/pki/testdata/release.json` (threshold 2,
sequence 5). That step checks that `update.prepare` refuses a tampered manifest, another digest and a missing one;
that `update.swap` refuses nothing staged, a symlink, a root-owned file and other bytes without moving `current`;
that a good swap puts the release in slot b behind the entrypoint (sequence 7 recorded in `upgrade.json`), after
which a second swap and confirming another version are refused; that `update.rollback` brings the installed binary
back and `update.settle` records the release as failed, so preparing it again is refused; and that the slots and
records are root's and not group- or world-writable, with the swap in the helper's log.

**Site agent, ramdisk and instances.** The site agent and ramdisk have Go tests only; no end-to-end or interop suite
runs `bcd site` or `bcd ramdisk` against the control plane yet (a VPS host has one, below). The agent tests drive a real `Agent` over a scripted
session (`newHarness`), answering its `SecretRequest`s with sealed JWEs:

| Test                                                                                                                             | Checks                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `internal/agent` `TestSiteBMCAndBootEntries`                                                                                     | A BMC that does not answer succeeds with `ok: false` and `unreachable`; the credential is fetched for the workflow step; a driver the agent does not host is `unsupported-driver`; boot entries persist, lapse and are removed by `boot.cleanUp`; the site kinds are refused on a node                                                                                                                                                                                                                                                                                                                                                                       |
| `TestHTTPProxyRefusesTargetsOutsideTheSite`                                                                                      | Loopback, the metadata address, a public address and `ftp://` fail `forbidden-target`; private, CGNAT and ULA addresses are site addresses                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `TestParseCredentialForms`                                                                                                       | JSON and bare-password BMC credentials, with the username fallback                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `TestBootServerRendersARamdiskScript`                                                                                            | 503 without the lease, 404 for an unknown MAC, then a script whose kernel line `bootparams.Parse` reads back, with the discovery token fetched once and an assertion signed by the agent's key at the lease epoch; 503 again once the gateway has been silent past the lease TTL                                                                                                                                                                                                                                                                                                                                                                             |
| `TestDiskWipeQuickVerifies`, `TestDeployImageStreamsAndChecksTheDigest`, `TestPickRootDiskAndDCGM`, `TestPartitionsAndBootFiles` | Quick wipes zero and verify both ends of sparse fake disks and are node-signed; a gzip image lands byte for byte with its user data; a wrong digest fails `checksum-mismatch` and leaves the disk's ends zeroed; root-disk choice, DCGM parsing, `lsblk` parsing and kernel lookup                                                                                                                                                                                                                                                                                                                                                                           |
| `TestSiteCacheWarmsAndServes`, `TestDeployImageFromTheSiteCache`                                                                 | The image cache pulls `cache_images`, verifies them by digest and serves them only under a grant, `/cache/t/<grant>/<sha256>` (a plain digest path and another image's grant are 404; b7ed8a2); `deploy.image` takes the image from the cache                                                                                                                                                                                                                                                                                                                                                                                                                |
| `TestExtWriteInUserspace` (Linux, skipped without `mkfs.ext4` and `debugfs`)                                                     | On a real ext4 image with a planted `/etc/bcd/config.toml -> /dev/sdb` symlink: the ramdisk's `debugfs` writes replace the symlink, never follow it, and read back as root-owned 0600 regular files; relative paths, `..` and shell metacharacters are refused (1ae1789)                                                                                                                                                                                                                                                                                                                                                                                     |
| `TestInstancesConverger`                                                                                                         | The `instances` converger over the fake runtime                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `internal/runtime` `TestLibvirtRuntime`                                                                                          | The `libvirt-kvm` runtime against a fake `virsh` and `qemu-img`: the domain XML and metadata, states, grow and shrink, snapshot and clone with memory, a CPU model mismatch, destroy, a mismatched image digest (362317c). No test runs it on a real KVM host                                                                                                                                                                                                                                                                                                                                                                                                |
| `internal/runtime` `TestBaseImageChecks`, `TestCheckOverlay`                                                                     | Tenant base images against real qcow2 headers read in Go (933c51b, b12c217): a qcow2 naming a backing file or an external data file, and a `qcow2` that is not one, are refused and not kept; good `raw` and `qcow2` images pass; an image cached before the check is checked again when used; an instance disk naming another backing file or format than its recorded base is refused                                                                                                                                                                                                                                                                      |
| `internal/runtime` `TestParseRebind`, `TestRebindImage`, `internal/agent` `TestIdentityRebind`                                   | The identity rebind (ca08d5c): bundle validation, the tar and acknowledgement region of the rebind disk, `cloud-hypervisor` and `libvirt-kvm` rebinds against an acknowledging and a silent fake guest, and the converger end to end (the secret fetch, `identity_rebound_ref` in status, persistence across a restart)                                                                                                                                                                                                                                                                                                                                      |
| `internal/runtime` `TestCloudHypervisorRuntime`                                                                                  | The `cloud-hypervisor` runtime against a fake VMM REST API and fake `systemd-run`, `systemctl`, `ip`, `qemu-img` and seed tools: define (explicit disk `image_type`s), the states, a memory snapshot, a restore into a clone with its paths rewritten, stop and destroy, and a directory that is not bcd's left alone (2598f0a); since 933c51b also the tap created and bridged before the VMM's unit starts, the unit's confinement and flavor limits, a memory snapshot of an 8 GiB VM refused in a 1 GiB flavor, a VMM that survives its stop failing `Destroy` with its directory kept, and the taps deleted on stop. No test runs it on a real KVM host |
| `internal/runtime` `TestRestoredConfig`                                                                                          | A memory snapshot's saved configuration is rebuilt from the allowlist: affinity, memory zones, `fs`, `vsock`, `pmem`, `debug_console`, landlock rules, unknown platform keys and the snapshot's disks are dropped, and a 2-vCPU snapshot is refused for a 4-vCPU instance (933c51b)                                                                                                                                                                                                                                                                                                                                                                          |
| `internal/agent` `TestBMCThroughAHostedDriver`                                                                                   | A `bmc.*` operation for a driver other than Redfish goes to the hosted bcx driver with the operation's concern and name, the address as config and the credentials as secrets; a driver the agent does not host, or one without the concern, is refused and nothing is invoked (51806ab); the BMC's driver options reach the plugin as `config.options`, and a Redfish request with options answers `invalid-config` (7753df0)                                                                                                                                                                                                                               |
| `cmd/bcx-ipmi` `TestIPMIOperations`, `TestIPMICipherSuiteOption`, `TestIPMIRefusals`                                             | The IPMI plugin against a fake `ipmitool`: the argv with cipher suite 17 (`-C 17`), or 3 from `options.cipherSuite` (7753df0), the password read from `-f /dev/fd/3` and absent from the environment, and refusals that run nothing, among them a BMC with a key pin, a cipher suite outside 17, 12, 8 and 3, an unknown option and a password over 20 bytes (51806ab, 933c51b, 7753df0)                                                                                                                                                                                                                                                                     |
| `cmd/bcx-ipmi` `TestIPMIInspect`                                                                                                 | `inspect.inspect` against a fake `ipmitool`: the Redfish driver's shape from `fru print 0`, `mc info`, `lan print` and `sdr list` (7818f19)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `internal/selfupdate` `TestManagerNeverSwapsBelowTheFloor`, `TestManagerStagedFileChecks`                                        | A lying `Settle` right after a swap and a swap below the floor under any claimed version are refused; a FIFO (no hang), a hard link, a foreign owner and a symlinked staging directory at the staged path are refused; the slots directory is 0755 (c139b9c)                                                                                                                                                                                                                                                                                                                                                                                                 |
| `cmd/bcd` `TestPCIAddress`, `TestDoctorToolChecks`                                                                               | NVML bus ids normalized to libvirt's PCI addresses for VFIO; `bcd doctor`'s VPS host tool checks, for a `libvirt-kvm` host (4a0165f) and a `cloud-hypervisor` one (2598f0a)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `internal/redfish` `TestRedfishOperations`                                                                                       | Every operation against a fake Redfish service over TLS pinned by SPKI; after a rotation the old password is `auth-failed`; an unknown operation is `unsupported`; a wrong pin and a plain-http BMC are refused                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `internal/pxe` `TestProxyDHCPAnswersPreparedMachines`, `TestTFTPServesFilesUnderItsRoot`                                         | On real loopback UDP sockets: PXE firmware of a prepared MAC gets `ipxe.efi` with the client's UUID and architecture read, iPXE gets the chain URL, an unprepared MAC gets no answer; TFTP serves a file from its root                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `internal/bootparams` `TestRenderParseRoundTrip`, `cmd/bcd` `TestRamdiskNeedsBootParameters`                                     | The kernel command line round-trips; `bcd ramdisk` without its parameters exits 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `cmd/bcd` `TestSimNeedsNodesAndAConfiguration`                                                                                   | `bcd sim --nodes 0` exits 2; `bcd sim` with a missing configuration file exits 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

**Host configuration, holds and Kubernetes (v1.1).** These run against fakes: `hostconfig.FakeSystem` (an in-memory
host whose files and command answers a test scripts), a fake root for a real `Agent`, and a fake kubelet on real unix
sockets. Only `TestRealKernel` touches a real kernel.

| Test                                                                                                                                                                                                                 | Checks                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/hostconfig` (22 tests)                                                                                                                                                                                     | Every config unit's plan and apply, drift, cleanup, refusals and reboot cases: sysctl, kernel modules before the sysctl they provide, huge pages with the mount unit, the GRUB and grubby fallbacks and per-NUMA pools, `cpu`, `iommu` and `gpu-settings`, the NVIDIA driver installed or asking for a reboot, the toolkit's CDI regeneration, Fabric Manager and `FabricManagerDown`, the MIG layout grammar and busy GPUs, MPS and `MPSDown`, unknown units and a unit waiting for a reboot not holding back the others (6c2343f, c21ef84). See [host configuration](/docs/daemon/host-config#what-the-tests-cover) |
| `internal/hostconfig` `TestRealKernel`                                                                                                                                                                               | As root with `BCD_HOSTCONFIG_REAL=1` on Linux: sysctl, the transparent huge page selector and 2 MiB huge pages against the real kernel, each put back, and root refusing to write under `/tmp`, a world-writable or a symlinked parent (60143bf, 02485b3). Passes on WSL Ubuntu                                                                                                                                                                                                                                                                                                                                       |
| `internal/agent` `TestHostSectionConverges`, `TestHostSectionReportsRebootRequired`, `TestHostDriftRepairedWithoutANewSpec`, `TestSignedHostSectionKeptForTheHelper`, `TestExternalHostAppliesNoUnit`, `TestGPUBusy` | The host section through a real `Agent` under a fake root, with `FabricManagerDown` and `RebootRequired` in the status report, drift repaired by the 10-minute pass (3c0fe02), the signed host section kept for the helper, `[host] units = false`; a GPU is busy when a hold's atom id names it (0f68c9a)                                                                                                                                                                                                                                                                                                            |
| `internal/helper` `TestHelperHostUnit`, `TestHostUnitNeedsTheSignedSpec`, `intent_bundle_test.go`                                                                                                                    | `host.unit` through a real helper socket: re-planned as root with the unit's own validation, refused keys, names and unknown units, cleans, busy GPUs, an unconfigured helper (e0fe5ec), and a helper whose own `config.toml` says `[host] units = false` refusing the unit with nothing written (3fa9021; `cmd/bcd` `TestRootOnlyFile` checks the root-owned config rule); the signed spec under pinned roots (03c023d) and under the trust bundle the join key signed, with a root rotation, a replayed bundle, another key's bundle and a rootless one (2d18327)                                                   |
| `internal/agent` `TestHoldsAndVMsShareTheHostsMemory`, `TestAHoldTheHostCannotFitFailsAsResourcesExhausted`                                                                                                          | Host admission of holds beside VMs, and `resources-exhausted` (31555bf)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `internal/agent`, `internal/runtime` volume tests                                                                                                                                                                    | Persistent volumes: sparse, grown and never shrunk, discarded; attached in order by serial; one writer, handoffs within a pass, what running VMs have open; grown volumes told to running guests (550da91, b5415a2, 448dc5f, 39458f9, 408e002)                                                                                                                                                                                                                                                                                                                                                                        |
| `internal/kube` `TestHostedManager`, `TestDRADriver`, `TestResourceSliceV1beta1`, `TestUnhealthyDevicesAreNeverOffered`                                                                                              | Hosted mode's device plugins and `bcd.dev/*` labels without a join (1e5f96b); the DRA driver's registration, slice, prepare and reaper (7d24836, fa8611e); an unhealthy GPU never offered by either API (6f5a99f)                                                                                                                                                                                                                                                                                                                                                                                                     |
| `cmd/bcd` `TestResourceChecks`, `TestHelperRuntimeChecks`                                                                                                                                                            | `bcd doctor`'s cgroup, huge page, `vfio-pci` and systemd checks (ab26e87), and the `libvirt` and `cloud-hypervisor` checks under the helper: the node out of the `libvirt` group, `dynamic_ownership = 0` and QEMU's user in the node's group (ed3c7d2), and `cloud-hypervisor` warning without `[helper] instance_bridges` (7fe74a1)                                                                                                                                                                                                                                                                                 |
| `cmd/bcd` `TestDoctorTSS`, `TestJoinReset`                                                                                                                                                                           | `bcd doctor`'s `tpm` check: silent without `[helper]`, a failure naming `gpasswd -d bcd tss` for a node user in `tss`, silent outside it or without a `tss` group (e6c3f53); `bcd join --reset`: nothing to move lets the join go on, an unconfirmed reset moves nothing and exits 2, a confirmed one renames `identity.json` and `node.key` to `.reset-` names, after which no identity loads (a1c1a46)                                                                                                                                                                                                              |
| `cmd/bcd` `TestAnUnprivilegedNodesRuntimes`, `internal/helper` `ch_test.go`, `internal/runtime` `TestCloudHypervisorUnderTheHelper`, `TestCHRecordRefusals`                                                          | `cloud-hypervisor` under the helper against fakes (d86fcbe, 7fe74a1, 94843ec): both runtimes kept for an unprivileged node, taps only of bcd's names on allowed bridges, a launch never taking another running unit's uid, VFIO only for whole groups of this host's GPUs and never a group another running VMM's user owns, root's checks of the instance record; since 1c9684a the running domain's live usage (CPU time, memory, tap counters) read from its cgroup and taps as the node, with no `ch.*` operation                                                                                                 |
| `internal/agent` `TestNodeKeyInTheTPM`, `internal/tpm` `TestNodeKey`                                                                                                                                                 | The node key in `tpmsim` (965c14d, 50e18f9, 0ba2a79): a join stores only the record, the key loads again, node statements verify under the certificate's key and `kid`, renewal makes the next key in the TPM (and moves a file-key node there), no TPM means no load; SHA-256 and -384 signatures, a CSR, `TPM2_Certify` by the AK. `cmd/bcd` `TestUserCanOpen` went with the check it tested (8870b8d)                                                                                                                                                                                                              |
| `internal/helper` `TestHelperTPMKeys`                                                                                                                                                                                | An unprivileged node's TPM node keys through the helper's `tpm.*` operations against `tpmsim` (8870b8d): the same key and AK as the TPM's own, signing, certifying, at most 8 keys loaded with the least recently used unloaded, a released key derived again, and refusals of bad shapes, unknown verbs and a helper without a TPM. `tests/cli-pki.test.ts` asserts the install script no longer adds `bcd` to `tss` and removes it on upgrade                                                                                                                                                                       |
| `internal/agent` `TestRenewCertifiedByTheAK`                                                                                                                                                                         | A certificate naming the TPM's AK (`urn:bc:tpm-ak:`) renews twice with the next key made in `tpmsim` and certified by that AK over `sha256(csr)` in `key_attestation`; with no TPM, or another TPM's AK, the renewal fails before any request, never with a file key (d94f1f3). `tests/server-join-tpm-renew.test.ts` is the control-plane side: the statement carried through renewal and every refusal. See [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key)                                                                                                                       |
| `internal/agent` `TestResolveAtomNamesParentsAndWhatTheDriverSees`, `TestHoldConflictsAcrossWholeGPUsPartsAndVMs`, `TestCDISpecOfPartsAndLimits`                                                                     | MIG and slot atoms resolved against the inventory, parent-aware conflicts and the VM passthrough guard, CDI names without `/`, the slice annotation and env, one `NVIDIA_VISIBLE_DEVICES` per hold (46a61db, d02ba4f)                                                                                                                                                                                                                                                                                                                                                                                                 |
| `internal/slice`, `internal/agent` slice tests                                                                                                                                                                       | The hold slice's systemd properties and hugetlb limits, the name and cpulist checks, removal only once empty (1d03c28, 260e33c)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `internal/kube` `TestJoinFromBareHost`, `TestJoinRefusesAnotherCluster`, `TestJoinSettingsAreCheckedAgain`, `TestJoinUnderHelper`, `TestContainerdImport`, `TestJoinConfigurationV1beta3`                            | The `k8s-join` unit against `FakeSystem` (88ecb07)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `internal/kube` `TestDevicePlugin`, `TestResourceOf`, `TestWireMatchesUpstream`                                                                                                                                      | The device plugin against a fake kubelet: registration, ListAndWatch with health and NUMA, Allocate and its refusal of an unbound device, preferred allocation, the reaper's grace period, re-registration after a kubelet restart; the hand-encoded messages against the upstream descriptors                                                                                                                                                                                                                                                                                                                        |
| `internal/agent` `TestHostAdmissionCountsCommittedMemoryNotUse`, `TestHostAdmissionTakesHugepagesFromThePlacementNodes`, `TestStartsOnlyWhatIsNotAlreadyRunning`                                                     | Host-side admission of instance starts (0dd122a)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `internal/agent` telemetry tests, `internal/telemetry`                                                                                                                                                               | The OTLP exporter against a fake collector (27c7ee2)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `internal/runtime` netconfig tests                                                                                                                                                                                   | The NoCloud `network-config` for leased NICs (0791fc9)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `cmd/bcd` `TestKubernetesDaemonSetContract`                                                                                                                                                                          | `@better-compute/provider-kubernetes`'s generated `config.toml` and command lines parse with bcd's own parser and flags (2c9f953)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

The FakeDaemon holds `BINDING_KUBERNETES` assignments like bcd since 4e61f59: its CDI spec carries
`bcd.dev/binding: kubernetes` and no `NVIDIA_VISIBLE_DEVICES`, while a plain hold's spec sets it once for the whole
hold (`tests/testing-fake-daemon.test.ts`).

**End to end.** `tests/e2e/bcd.test.ts` runs the real binary (linux/amd64, fakehw, `BCD_FAKEHW_SEED=41`) against
`createDevStack({ externalTls })`: the dev stack's mTLS listener with a dev serving CA, which the test pins in a
generated `config.toml` together with the join keys, `gateway_server_name = "gw.dev.local"` and a data directory
under `/tmp`. It checks that:

* `bcd join` with a sealed token on standard input succeeds and never prints the token;

* `bcd node` is adopted after its node-signed `diag.nvml`, and `bcd status --json` answers on the socket;

* `inventory.refresh` succeeds;

* a 2-GPU reservation turns active on a node-signed `AllocationBound` (checked by `pkiAllocationBindingCheck`), its
  CDI hold appears under the fake root, and it disappears on release;

* a node console opened with `tunnels.open` (op `console`) through the dev stack's tunnel relay and gateway reaches a
  real shell in bcd: `echo hi-$((6*7))` answers `hi-42`, and `exit 5` comes back as exit code 5;

* `node.reboot` completes `SUCCEEDED` from the next boot, with the new boot id recorded in `node_sessions`;

* a provisioned OS bootstraps with nothing but a token file, the control plane URL and the three pins (what
  provision's user data and the Robot bootstrap run): `bcd join --server` reads the real server's well-known document,
  joins through the real gateway and JoinService, removes the token, writes `config.toml` and `join-encryption.jwk`,
  succeeds again on a re-run without a token, and `bcd node` connects on the written configuration alone (472a424);

* `bcd sim --nodes 5 --seed 100`, given one five-use join token on standard input, joins five simulated machines that
  connect as five distinct nodes (five distinct hardware keys), and never prints the token (aa7501c; see
  [bcd sim](/docs/daemon#bcd-sim));

* self-update (fc0ca6a): a second fakehw build reporting 0.9.0 is released under a key pinned in the node's
  `[update]` section; the node restarts from an entrypoint it may replace, gets a `daemon.upgrade` enqueued as the
  rollout controller does, downloads and verifies the release, swaps to slot `b`, execs, and is confirmed by its
  `Welcome`: `node_sessions.daemon_version` is 0.9.0, `current` is `bcd-b`, and `upgrade.json` is gone.

* the WebSocket fallback (4b2d5aa): the TypeScript `serveGatewayWs` runs beside the stack's gateway with a serving
  certificate from the CA the node pins and audience `gw.dev.local`; the node restarts with `[websocket] url`, runs
  a signed `inventory.refresh` over the WebSocket session, and renews over `POST /v1/renew`, which the fakehw knob
  `BCD_RENEW_AFTER=3s` brings forward.

* node metrics (efdff1d): a test module sets `NodeSpec.telemetry` (http to the WSL host, a 5-second interval and an
  extra resource attribute), the node's spec is rendered again, and the fakehw bcd posts OTLP/JSON to a local
  receiver in the test: `bcd.gpu.utilization` for all 8 GPUs, memory, temperature, power and active XIDs, with
  `service.name=bcd`, `bcd.node.id` and the attribute as resource attributes. The receiver is the test's own, not a
  collector product.

`tests/e2e/bcd-host.test.ts` (91f5e81, same gating) runs `NodeSpec.host` end to end: a `metal-host` pool with
`policy.hugepages` (2 MiB, 64) and `policy.kernelModules` (`tun`, `vhost_net`), provision renders the units, the
gateway sends them, and the fakehw bcd applies them under its fake root. `tests/e2e/bcd-placement.test.ts`
(427787a, same gating) runs PLAN.md §3.8 against fakehw bcd nodes, in 6 tests: a pinned, NUMA-affine flavor
(`vcpupin` and a strict `numatune` in the libvirt domain the fake runtime plans, within one NUMA node and off the
host-reserved CPUs), a 2 MiB huge-page flavor on the pages provision's unit reserved (page backing, locked, no
balloon), an overcommitted pool's guest ballooned to its floor under simulated memory pressure and deflated again, a
sandbox volume attached by its serial that outlives the sandbox, an instance on a host without the `firecracker`
plugin label staying unschedulable, and a GPU hold with booked vCPUs and memory in its `bcd-alloc-<id>.slice`,
reported as `binding.cgroupParent` and removed on release. It found that a re-collected inventory waited up to ten
heartbeats for its hashes, fixed in the same commit. It relies on two fakehw features: the runtime writes the
libvirt XML it would define to `<data_dir>/fakehw-plans/<id>.xml` and keeps its balloons on disk (9a6aaf4), and the
inventory reads the huge pages the host units reserved back from the fake root (cd7a311); see
[local development](/docs/operations/local-development). `tests/e2e/kubernetes-bindings.test.ts`
(8a084ca) runs `binding: 'kubernetes'` on the dev stack's FakeDaemons: the hold marked `bcd.dev/binding=kubernetes`
without `NVIDIA_VISIBLE_DEVICES`, a plain hold on the same pool not, the booked limits in the assignment, release,
and the refusal on a pool without `policy.kubernetes`. `tests/vps-sandboxes.test.ts` checks that a sandbox lands only
on a host labelled `bc.dev/cap.instance.ssh-host-only` (3b9e81d), and `tests/vps-volumes.test.ts` covers sandbox
volumes: booking, quota, replay, pinning, serial mounts, one writer, read-only sharing, resize, deletion and metering
(2a09202, 8cbece0). `tests/metal-discovery-e2e.test.ts` (452d6bf, not gated: FakeDaemons only) runs
[PXE discovery](/docs/guides/bare-metal/sites#discovering-new-machines) through the real control plane and gateway,
with a FakeDaemon site agent carrying metal's site kinds: an IPMI machine enrolled with only its product UUID,
`metal.ensureRamdisk` setting PXE and power-cycling it through the agent, the agent reporting it with
`setDiscovered` in DHCP's mixed-endian UUID spelling, metal's module status handler (composed into the dev stack's
gateway) storing the report, and `metal.discovered` showing the MAC as the machine's candidate with
`discovered_mac` set, `macs` still empty and no boot entry sent for 2 seconds. Only after `metal.confirmMac` does the
agent get `boot.prepareRamdisk` for exactly that MAC, with a `secret://` discovery token reference and no sanboot
digest. The dev stack has no ramdisk to join, so it ends at the boot entry; `tests/metal-discovery.test.ts` covers
the rest on the sim.

The whole file runs on `createDevStack({ signedIntent: true })` (5071dbc), which puts the stack under a
command-signing root: joined daemons receive the roots in `JoinResponse.trust` and every command and spec bcd takes
is signed and verified. With `bcd-vps.test.ts` that is 9 of 9 bcd end-to-end tests.

`tests/e2e/bcd-vps.test.ts` runs the same fakehw binary as a VPS host: bcd joins a `vps-host` pool over the dev
stack's mTLS listener, `instances.create` places an instance on it, the `instances` converger defines and starts it
on the fakehw build's file-backed `libvirt-kvm` runtime (not the libvirt runtime of host builds),
`StatusReport.instances` carries it and the vps module moves it to
`running`, and `instances.delete` removes the domain from the host. It has the same gating.

`tests/e2e/bcd-tpm.test.ts` (db9bd46, 8a9af54, same gating) runs the `tpm` join end to end: a fakehw bcd whose
software TPM lives in `<data_dir>/fakehw-tpm.json` runs `bcd join --attestor tpm` with no token against the dev
stack's JoinService with `tpmAttestor` (`createDevStack({ attestors })` passes join attestors to `betterCompute()`).
Since 8a9af54 the test makes a manufacturer root, the fakehw TPM carries an EK certificate it issued
(`BCD_FAKEHW_TPM_EK_CA`), and the attestor trusts only that root (`ekRoots`, no `allowUncertifiedEk`, and a `resolve`
allow-list for the TPM pool); with `BCD_FAKEHW_TPM_KEYS=1` the node key is made in the same TPM. It checks the two
rounds, `resolve` seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in
the resolved pool and bound to `sha256(EK)` in `node_bindings`, `node.key` holding only a `BCD TPM NODE KEY`
record, the node key certified by the TPM's AK so that both joins' `node.joined` events say `nodeKey: 'tpm'` (cf24876;
the attestor keeps its default `nodeKey: 'certified'`), a rejoin as the same node after the identity is removed (a
reinstall), `already joined` on a further run,
and `bcd node`, whose session and statements are signed in the TPM, adopted on that identity. Since d94f1f3 it also
checks that the certificate names the TPM's AK (`urn:bc:tpm-ak:`), and that `bcd node` (with `BCD_RENEW_AFTER=3s`)
renews with the AK certifying the next key: one more `node_certs` row, still carrying the statement, and `node.key`
still a TPM record. Its wire format is pinned in both directions without a stack too: bcd's `TestTSContract`
and `tests/pki-attestors-tpm-bcd.test.ts` share the fixtures in `daemon/internal/tpm/testdata`, which
`scripts/daemon-tpm-fixture.mjs` regenerates (see [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm)).
Since b973e05 those fixtures include a test manufacturer root (`ek-root.der`) and the EK certificate it issued
(`ek-cert.der`) in the sim's NV, so that pair runs `tpmAttestor` with `ekRoots` and no `allowUncertifiedEk`, and
checks that another root is refused (`ek-certificate-chain`). Since cf24876 the fixture's answer also carries the AK's
certification of a node key in the sim, which the TypeScript side admits as `nodeKey: 'tpm'` and refuses for another
CSR key (`node-key-not-csr`); `tests/pki-attestors-tpm.test.ts` covers the same checks with the TypeScript software
TPM, and suites whose software TPM holds no node key opt into `nodeKey: 'any'`.
None of these touches a hardware TPM.

On Linux the suites run in `pnpm test` whenever Go is installed; they build the binary first. On Windows they are
opt-in: they need `BCD_E2E=1`, a WSL distribution, and Go at `C:\Program Files\Go\bin` (or an existing
`daemon/bin/bcd-linux-amd64`). bcd runs in WSL through `wsl -e`, never a shell, and reaches the gateway at WSL's
default route. `BCD_E2E_LOG=1` streams bcd's log to standard error.

```powershell
$env:BCD_E2E = '1'; pnpm vitest run tests/e2e/bcd.test.ts
```

## The fleet simulator [#the-fleet-simulator]

`createFleetSim(options)` runs N FakeDaemons, a fresh faulty link per dial, a seeded chaos schedule and a pluggable
`GatewayPort`. The default port is `GatewayStub`, a minimal in-memory control plane (Hello, Welcome, dispatch with
credits, result ingestion, liveness, the command sweeper, bookings under an exclusion check, usage flushes). The
stub is a test double for the daemon side, not a reference for the server.

```ts
import { assertInvariants, createFleetSim } from 'better-compute/testing';

const sim = createFleetSim({
  seed: 'fleet-100',
  nodes: 100,
  up: { latencyMs: 5, jitterMs: 20, drop: 0.0005, dropMode: 'reset' },
  down: { latencyMs: 5, jitterMs: 20, reorder: 0.002 },
  chaos: { everyMs: 5_000, crash: 0.3, reboot: 0.1, journalLoss: 0.05, partition: 0.3, gatewayKill: 0.02 },
});
await sim.start();
await sim.run(10 * 60_000); // ten simulated minutes
sim.quiesce();
await sim.run(16 * 60_000);
assertInvariants(sim.invariants());
console.log(sim.digest()); // the same seed gives the same event log, byte for byte
```

| Option          | Default              | Meaning                                                                                                                                                                          |
| --------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `seed`          | required             | Everything random derives from it                                                                                                                                                |
| `nodes`         | 10                   | A count, or one `{ nodeId?, hardware?, role?, daemon? }` per node                                                                                                                |
| `start`         | 2026-01-01T00:00:00Z | Sim start time                                                                                                                                                                   |
| `gateway`       | a `GatewayStub`      | A factory for another `GatewayPort`                                                                                                                                              |
| `up`, `down`    | none                 | Link faults in each direction                                                                                                                                                    |
| `daemon`        | none                 | FakeDaemon options for every node                                                                                                                                                |
| `tickMs`        | 1 s                  | Period of `GatewayPort.tick`                                                                                                                                                     |
| `startSpreadMs` | 5 s                  | Daemons start at random times within this window                                                                                                                                 |
| `logFrames`     | true                 | Log every frame delivery, drop and duplicate                                                                                                                                     |
| `chaos`         | none                 | `everyMs`, `forMs`, and per-draw probabilities `crash`, `reboot`, `journalLoss`, `partition`, `isolate`, `halfOpen`, `reset`, `stall`, `gatewayKill`, with their duration ranges |

The simulator also exposes each fault as a method (`crash`, `reboot`, `loseJournal`, `partition`, `isolate`,
`heal`, `halfOpen`, `resetLink`, `stall`, `killGateway`), `every(ms, fn)` for workloads, and `rng.fork(label)` for
independent seeded streams: adding a node never shifts another node's draws. `better-compute sim` is this simulator
with a built-in workload; see [local development](/docs/operations/local-development#the-fleet-simulator).

## Invariants [#invariants]

The checkers are pure functions over collected facts, so the same checks run over the fleet simulator, the real
control plane or a database snapshot.

| Id | Invariant                                                                                                                                                                                                                                         | Checker   |
| -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| I1 | No atom is booked by two allocations at one instant, and no node holds one device for two allocations at once (and only binds what its allocation booked)                                                                                         | `checkI1` |
| I2 | A non-retry-safe or self-terminating command starts its side effect at most once; a retry-safe one re-runs only after a crash interrupted it; after a journal loss, in-flight non-retry-safe commands fail rather than replay                     | `checkI2` |
| I3 | Every `DesiredState` a daemon received is reported observed (a status report at that generation or later) within `CONVERGENCE_MS` (60 s), judged only while a gateway heard from the node at least every 15 s and only once the window has passed | `checkI3` |
| I4 | No node is marked `unreachable` or `lost` within `LIVENESS_WINDOW_MS` (20 s) of a gateway receiving a frame from it; the times come from the transport on the simulator's clock, so gateway clock skew cannot hide a false positive               | `checkI4` |
| I5 | Every release has an allowed cause (end of booking, release, cancel, lapsed lease, failed bind); none is caused by a disconnect                                                                                                                   | `checkI5` |
| I7 | After a quiet period, no pending reservation would fit if planned alone now; the input is capacity's offline solver, `placeablePending(tx, { now })`                                                                                              | `checkI7` |
| I8 | Per allocation, usage windows do not overlap, stay inside the booking and add up to its duration times atoms, with no duplicate keys                                                                                                              | `checkI8` |

Two more I1 checkers (40b1ce8) cover PLAN.md §3.8's host resources and Kubernetes bindings:

| Checker                                                 | Checks                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `checkHostResources(samples)`                           | Over sampled `node_resources` rows (`HostResourceSample`: vCPU, memory, huge pages), `allocated` never exceeds `floor(allocatable x ratio)` and always equals what the node's unreleased allocations claim, so a release that subtracts twice or a re-book without a claim shows as drift                                                                                 |
| `checkKubeAdvertisements({ bookings, advertisements })` | No device is offered to a node's kubelet for one allocation while the control plane books it to another, since a kubelet can hand an offered device to a new pod at any moment. The offers (`KubeAdvertisement`) come from `FleetFacts.advertisements`: a FakeDaemon records `kube.advertised` when it offers a Kubernetes-bound hold and `kube.withdrawn` at its release |

With the knob `kubePodReleaseMs`, a FakeDaemon's pods keep a released Kubernetes hold's devices that long: it stops
offering them at once, keeps them claimed (a new hold on them fails with overlap) and reports `AllocationReleased`
only then, as bcd's fail-closed release does.

`assertInvariants(violations, limit?)` throws one error listing them. The fleet simulator's `invariants()` checks I1,
I2, I3, I4, I5 and I8 (I4, I5 and I8 read the gateway port's `facts()`, which the stub gateway and the control-plane
simulator provide). I7 needs a database: pass it to `runNightly({ invariants })`, as the CI-sized night does.
PLAN.md section 15 also names I6 (latency SLOs); the kit does not check it, and the real-time benchmarks measure I6-style latencies instead.

I4 is also tested under gateway clock skew, which section 15 calls for: `tests/sim/clock-skew.test.ts` runs 9 nodes on
3 real gateways whose own clocks are 90 s ahead, 90 s behind and on time (more than the 40 s liveness timeout), with
`compute_now()` and the daemons on simulated time. Every gateway holds sessions, `diag.nvml` settles through all of
them, and when the unskewed gateway dies its nodes re-home onto the skewed ones; connectivity sampled every simulated
second never shows `unreachable`, no `node.unreachable` is published, and I1 to I5 and I8 hold, because gateways
flush ages rather than timestamps.

## The control-plane simulator [#the-control-plane-simulator]

`createControlPlaneSim(options)` puts the real control plane and real gateways behind the fleet simulator:
`betterCompute` on the sim clock with `compute_now()` pinned to simulated time, `gateways` real gateways on manual
jobs (default 1), and nodes that join through JoinService with sealed tokens bound to their CSRs. It sets up a site
and an H100 pool with `diag.nvml` acceptance offered to `acme`; reconcilers and gateway jobs run on the simulator's
ticks.

| Option                                          | Meaning                                                                                                                                                               |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `seed`, `nodes` (default 4), `variant`, `start` | As for the fleet simulator                                                                                                                                            |
| `gateways`                                      | Gateway processes behind the simulator (default 1)                                                                                                                    |
| `iam`                                           | Wraps the fake IAM, for example `(fake) => spyIamPort(fake)`                                                                                                          |
| `trace`                                         | Runs each part of the loop under a name (`join`, `fleet`, `gateway`, `reconcilers`, `api`), for an IAM spy's `path`                                                   |
| `restricted`                                    | The last `count` nodes join with a certificate that allows only `allowedKinds`                                                                                        |
| `modules`                                       | Extra control-plane modules, ticked with the rest                                                                                                                     |
| `gateway`                                       | Extra gateway options, such as lease timings                                                                                                                          |
| `workerPasses`                                  | Batches each control-plane job may run per simulator tick while it keeps finding work, as a real worker loops without sleeping (default 1)                            |
| `gatewayClockSkewMs`                            | One number per gateway slot: that gateway's own clock offset from simulated time in ms (positive is ahead); the database clock and the daemons stay on simulated time |

The rig offers `run(ms)`, `api(fn)`, `killGateway(i)`, `reviveGateway(i)`, `stallGateway(i, stalled)`,
`pauseWorkers(paused)`, `revokeNode(nodeId)`, `homeOf(nodeId)`, `serialOf(nodeId)` and `invariants()`, which reads
the facts from compute's own tables (bookings, billed allocations and usage windows, command outcomes,
connectivity) and checks I1, I2, I3, I4, I5 and I8. `tests/sim/multi-gateway.test.ts` uses it for a gateway killed
mid-command and a zombie gateway fenced by its session epoch.

`tests/sim/kubernetes-host-resources-1k.test.ts` (246c95b) churns Kubernetes-bound and hold reservations with vCPU and
memory shares over a GPU pool of `SIM_K8S_NODES` hosts (default 1,000) under link faults, crashes, partitions and
resets, re-inventories hosts with less CPU and memory and then restores them, and has pods keep released Kubernetes
GPUs for 45 seconds. After every step it asserts `checkHostResources` and the fleet invariants; the recorded run at
1,000 nodes is green in about 4.5 minutes. A second, strict 12-node run of `checkKubeAdvertisements` checks that no
kubelet is offered a GPU booked elsewhere. It used to fail (capacity freed a Kubernetes-bound GPU at release and
re-booked it before pods let go, so the node rejected the new binding: 55 of 240 reservations at 12 nodes); since
e7d922c a Kubernetes booking ends at the node's `AllocationReleased`, and the run passes with no failed reservations.

### The nightly simulation [#the-nightly-simulation]

`runNightly({ seed, nodes?, gateways?, minutes?, chaos?, stepMs?, modules?, inspect?, invariants? })` runs one simulated night on
the control-plane simulator: three minutes of start-up, then `minutes` (default 30) of tenants reserving with every
placement feature (half and whole nodes, gangs of two, spot, calendar starts, early releases) and operators running
commands, under chaos. Chaos adds gateway kills and revivals, worker outages and, at `heavy`, one certificate
revocation per night. `invariants(rig)` adds checks run after the quiet period, such as I7 with capacity's
`placeablePending`. The report has the invariant violations, `kindLeaks` (commands queued outside a restricted
certificate's kinds, which must be 0), place-to-active and command latencies in simulated time, and two digests:
the workload (a function of the seed) and the id-free outcome. Compute mints random ids, so outcomes can differ
between runs of one seed.

```bash
pnpm sim:nightly
SIM_NIGHTLY_SEEDS=1,2,3,4 SIM_NIGHTLY_CHAOS=heavy SIM_NIGHTLY_NODES=32 SIM_NIGHTLY_MINUTES=120 pnpm sim:nightly
```

`pnpm sim:nightly` (`vitest.nightly.config.ts`, `tests/nightly/*.nightly.ts`) runs every seed times chaos level one
after another, with `auditModule` checkpointing every 10 simulated minutes and every tenant's audit chain verified at
the end. Defaults: seeds `1,2,3`, chaos `light,heavy`, 16 nodes, 3 gateways, 60 simulated minutes; reports land in
`bench/sim-nightly/results`. A CI-sized night (4 nodes, 2 gateways, 6 minutes) runs in `tests/sim/nightly.test.ts`;
both check I7 through `invariants`.
The v0.4 exit asks for every section 15 scenario green on three consecutive nights; the `nightly` CI workflow
counts them (see [CI](#ci-and-the-repository-checks)).

### Cold start [#cold-start]

`runColdStart({ seed, nodes?, gateways?, startSpreadMs?, limitMs?, settleMs?, restart?, restartDownMs?, stepMs?,
workerPasses? })` is the v0.4 "10k cold start in 3 minutes" scenario on the control-plane simulator. The fleet powers
on at random times within `startSpreadMs` (default 30 s) behind `gateways` (default 3) gateways; each node's first
session (`node.connected`) and its adoption (`node.adopted`) are timed from power-on, in simulated time, from the
outbox. The run is ready when every node is connected, adopted and schedulable, and it must be ready within
`limitMs` (default 3 minutes). With `restart` (default true) every gateway then goes down and every node
power-cycles, and the way back is timed the same way. The invariants I1 to I5 and I8 are checked at the end, and a
`node.unreachable` mark during boot fails the run. Defaults: 100 nodes and `workerPasses` 10. The report has
per-phase latency summaries, `unreachableMarks`, `schedulable`, the violations and `wall.loops`, the wall time
spent in each part of the loop.

| Where                                                      | Size                                                                                                                                                                                                                                                                      |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tests/sim/cold-start.test.ts`                             | 24 nodes, in CI: boot and power cycle each within 3 simulated minutes, nothing marked unreachable                                                                                                                                                                         |
| `tests/nightly/cold-start.nightly.ts` (`pnpm sim:nightly`) | `COLD_START_NODES` (default 10,000), `COLD_START_GATEWAYS` (3), `COLD_START_SPREAD_MS` (30,000), `COLD_START_SEED` (`1`); writes a report to `COLD_START_OUT` (default `bench/cold-start/results`). The recorded 10k run took 641 s of wall time on a 64-vCPU workstation |

The recorded 10k run (`bench/cold-start/README.md`) was ready at 80 simulated seconds: every node connected by
30 s (p99 29.7 s, bound by the power-on spread), adopted by 79 s and schedulable at 80 s, and every node was back
40 s after the power cycle, with 0 unreachable marks and 0 invariant violations. Run it alone with:

```bash
COLD_START_NODES=10000 node node_modules/vitest/vitest.mjs run --config vitest.nightly.config.ts tests/nightly/cold-start.nightly.ts
```

See [scaling](/docs/operations/scaling#cold-start) for the 1k and 10k results side by side.

## Real-time harnesses [#real-time-harnesses]

The fake clock never measures latency. The real-time harnesses run on the wall clock over cleartext HTTP/2
(`createH2cClient`), against PGlite by default or a Postgres you name with `postgresUrl` (it must be disposable: the
harness migrates it).

* **`createRealtimeHarness({ nodes?, postgresUrl?, gateway?, services?, daemon?, sharedConnection?, pollMs? })`**: a
  real gateway and database with FakeDaemons on pre-registered nodes (default 50). `connectAll()`,
  `dispatch({ kind, perNode })` (enqueue to acknowledgement and to terminal), `load({ ratePerSecond, durationMs })`
  (steady state) and `storm({ fraction, mode: 'disconnect' | 'crash' })` (reconnect storms).
* **`createRealtimeFleet({ postgresUrl?, variant?, pollMs?, worker?, gateway?, daemon?, joinConcurrency? })`**: the
  whole v0.1 spine, a started control plane with JoinService on real PKI and a gateway wired to its services. `join(n)`
  times each node from its Join call to adopted; `place(n, { gpus?, ratePerSecond?, concurrency? })` times
  reservations from placed to active.
* **`probeLeases(sql)`** samples the `leases` table: renewal gaps, the least headroom before expiry, lapses while
  owned, and takeovers.

Every result is a `LatencySummary` (`count`, `p50`, `p90`, `p99`, `max`). Small runs are in `tests/realtime/`; the
1k runs belong to the benchmarks.

## Postgres-only suites [#postgres-only-suites]

Everything about locking and fencing needs real PostgreSQL: PGlite is one serialized connection. These suites skip
unless `BETTER_COMPUTE_POSTGRES_URL` is set:

```bash
BETTER_COMPUTE_POSTGRES_URL=postgres://postgres:postgres@localhost:5432/compute_test pnpm test:postgres
```

`pnpm test:postgres` runs `tests/postgres-conformance.test.ts` (the full store conformance suite, concurrency cases
included, each case in a fresh schema; the database needs `btree_gist` or the right to create it) and
`tests/postgres-schema.test.ts`. CI runs them against a `postgres:17` service. The benchmarks read the same variable.

## Benchmarks [#benchmarks]

`pnpm bench:scale` runs `tests/bench/**/*.bench.ts` with `vitest.bench.config.ts` (the test aliases, a 30-minute
timeout), on demand only. `gateway-streams` sets its own timeout, `BENCH_GW_DURATION_MS` plus 20 minutes, so the
30-minute gate run with its ramp, storm and settle is not cut off. Each suite writes `bench/<suite>/results/<date>-<database>-<n>.json` with the commit, the
machine, the Node.js version and the configuration. Commit only runs on the reference hardware of
`bench/REFERENCE.md`, or runs labeled `referenceHardware: false`.

| Suite             | Measures                                                                                | Knobs                                                                                                                                                                                                                                            |
| ----------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `gateway-streams` | Sessions one gateway holds at 10 s heartbeats, dispatch latency, RSS, a reconnect storm | `BENCH_GW_SESSIONS` (5000), `BENCH_GW_DURATION_MS`, `BENCH_GW_COMMANDS_PER_S` (50), `BENCH_GW_STORM`, `BENCH_GW_TLS=1` (mTLS, one connection per daemon), `BENCH_GW_SPLIT=1` (the gateway and its store in their own process), `BENCH_GW_GATE=1` |
| `iam-lane`        | Embedded better-iam admissions on adapter-postgres, lock wait                           | `BETTER_COMPUTE_IAM_POSTGRES_URL`, `BENCH_IAM_CONCURRENCY` (`1,4,16,32`), `BENCH_IAM_DENY_SHARE`, `BENCH_IAM_GATE=1`                                                                                                                             |
| `placements`      | Placements/s and lock hold, inline and pool-lease                                       | `BENCH_PLACE_NODES` (1000), `BENCH_PLACE_COUNT` (300), `BENCH_PLACE_GPUS` (1), `BENCH_PLACE_CONCURRENCY`                                                                                                                                         |
| `realtime-fleet`  | Command dispatch to ack and terminal, steady load, reconnect storms                     | `BENCH_RT_NODES` (1000), `BENCH_RT_PER_NODE`, `BENCH_RT_LOAD_RATE`, `BENCH_RT_STORM`, `BENCH_RT_GATE=1`                                                                                                                                          |
| `realtime-spine`  | Join to adopted, place to active on the whole spine                                     | `BENCH_SPINE_NODES` (1000), `BENCH_SPINE_RATE` (5), `BENCH_SPINE_RESERVATIONS` (200), `BENCH_SPINE_BURST` (100), `BENCH_SPINE_GATE=1`                                                                                                            |
| `multi-gateway`   | Re-homing after a gateway kill, in simulated time                                       | `BENCH_MG_NODES` (1000), `BENCH_MG_GATEWAYS` (3), `BENCH_MG_GATE=1`                                                                                                                                                                              |
| `postgres-spine`  | The spine end to end on Postgres (skipped without a URL)                                | `BETTER_COMPUTE_POSTGRES_URL`                                                                                                                                                                                                                    |

The numbers measured so far, and what they mean, are on [scaling](/docs/operations/scaling).

## CI and the repository checks [#ci-and-the-repository-checks]

Four GitHub Actions workflows are defined under `.github/workflows/`. This page describes what they run, not whether
they are green.

| Workflow      | Runs on                                                                                                                                  | Does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ci.yml`      | every push to `main` and pull request                                                                                                    | On Ubuntu and Windows with Node 22 and 24: `pnpm check`, `pnpm proto:check`, `buf breaking` (Ubuntu, Node 24), `pack:all`, `check:packed-types`, `test:packed`. A job with a `postgres:17` service runs `pnpm test:postgres`. After both, on `main`, the publish job publishes a version that is not on npm yet with `pnpm publish -r --provenance`, then tags the release and dispatches `release.yml`                                                                                                                                                                                                                                                                                                            |
| `nightly.yml` | 02:17 UTC every day, or by hand                                                                                                          | `sim`: `pnpm sim:nightly` with the 10k cold start (`COLD_START_NODES=10000`), results uploaded. `guarded`: `pnpm test` and `pnpm test:iam-remote` with `BETTER_COMPUTE_IAM_GUARD=throw`, so any `IamPort` call inside a compute transaction fails. `postgres`: the whole `pnpm test` against `postgres:17`, so the Postgres-only cases run. `streak`: counts consecutive green scheduled nights and reports when the v0.4 exit's three are reached                                                                                                                                                                                                                                                                 |
| `release.yml` | a `v*` tag, or a dispatch with a tag                                                                                                     | bcd's release: goreleaser archives and binaries for linux amd64 and arm64 with SHA-256 files, an SPDX SBOM per archive (syft), keyless cosign signatures, and SLSA level 3 provenance (`bcd.intoto.jsonl`) from the SLSA generic generator; see [installing bcd](/docs/daemon/install)                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `daemon.yml`  | pushes to `main` and pull requests that touch `daemon/`, `proto/`, `packages/provider-kubernetes/` or `scripts/k8s-render-manifests.mjs` | On Ubuntu and Windows: `go vet` and `go test` with `nonvml,fakehw`; on Linux also `go vet` of the NVML build and `goreleaser check`. The `container` job (ac1ca4b) builds `daemon/Containerfile` for `linux/amd64` with buildx, runs `bcd --version` and `kubectl version --client` from the image, renders the DaemonSet manifests (`scripts/k8s-render-manifests.mjs`, placeholder pins and digest) and, on a kind cluster, applies the Namespace and server-side dry-runs the rest (`kubectl apply --dry-run=server`), so the API server checks the schema and Pod Security admission. Nothing is scheduled or pulled: it is not a run of bcd as a DaemonSet (see [bcd on Kubernetes](/docs/daemon/kubernetes)) |

`pnpm check` runs `typecheck`, `test`, `test:iam-remote`, `lint:repo` and `validate:packages`. `pnpm lint:repo` runs
the repository rules in order: `check-deps` (dependency rules), `check-sql-now` (the one-clock rule), `umbrella --check` (umbrella drift), `proto-gen --check` and `proto-gen --breaking`, then `apps/docs/scripts/generate.mjs --check` (the generated docs reference must match the code; 267f8f4), then the API surface freeze. The generator
check comes first because the freeze reads the committed reference: before it, a narrowed input that nobody
regenerated passed the freeze. Under `pnpm check` it runs after `tsc -b`, so it sees the built code.

If you change the registry (a method, its input schema, an error code, a command kind, an event or a metric),
regenerate the reference in the same change. On a failure the check lists each file `out of date` or `stale` and
prints the command to run, which regenerates from committed code only, so other sessions' uncommitted builds never
reach the reference:

```bash
node scripts/head-check.mjs --run apps/docs/scripts/sync-from-head.mjs "$PWD"
```

### The API surface freeze [#the-api-surface-freeze]

PLAN.md's v1.0 freezes the API. `scripts/api-surface.mjs` compares the frozen surface, `generated/api-surface.json`
(every method with its route, kind, IAM action, resource, alternatives, list flag, status, phase and input JSON
Schema, and every error code with its HTTP status), with the current one, which it reads from the docs generator's
committed `apps/docs/generated/api.json` and `errors.json`.

Additions pass. These fail the check:

* a served method removed, or planned again;
* its route, kind, IAM action or resource changed, an IAM alternative removed, or a list method no longer one;
* its input schema rejecting an input the frozen schema accepted: a new required field, a type, enum value or
  property that goes, a tighter length, range, pattern, format or item count, a closed object that was open, a lost
  `anyOf` branch, or a constraint on a value that was unconstrained;
* an error code removed, or its HTTP status changed.

A method that is only planned may still change or go. When you add or change something compatibly, regenerate the
docs and record it, so the addition is frozen too:

```bash
node apps/docs/scripts/generate.mjs   # api.json and errors.json follow the code
pnpm api:surface                      # node scripts/api-surface.mjs --update: records additions, refuses breaks
node scripts/api-surface.mjs --update --accept-breaking   # only for a deliberate, reviewed break
```

The plain check prints the number of additions not yet recorded without failing. `tests/scripts-api-surface.test.ts`
covers the rules and checks the committed snapshot against the generated docs.

  - [Local development](/docs/operations/local-development): `dev --sim` and `better-compute sim` .

  - [Scaling](/docs/operations/scaling): The section 16 gates and the measured numbers.

  - [Observability](/docs/operations/observability): Metrics, spans and doctor.


# Upgrades (/docs/operations/upgrades)

> The order an upgrade runs in, how migrations behave, the version skew allowed between the server, gateways and bcd, rolling bcd out with signed releases, and the API surface freeze that keeps clients working.



An upgrade of Better Compute touches four things that run different code: the database schema, the control-plane
processes (api and workers), the gateways, and `bcd` on every machine. They are upgraded one at a time, in an order
that keeps every pair that talks to each other compatible at every moment. This page is the operator's runbook for
that order. The detail of each piece lives on its own page and is linked rather than repeated.

> **What has and has not been exercised.** 
  The upgrade order is proven in the fleet simulator: `tests/sim/upgrade.test.ts` runs the real control plane with
  three gateways and simulated daemons through expand migrations, workers, gateways one at a time and a signed bcd
  rollout, under load, and checks that no allocation moves, no VM reboots and invariants I1, I2, I5 and I8 hold
  (80195df). Both "releases" in that test are the same build plus one extra migration and a FakeDaemon that reports a
  new version; it is not an upgrade between two published releases. Every package is at 0.2.0 and no bcd release has
  been cut, so no deployment has been upgraded from one real release to the next yet.

## The order [#the-order]

PLAN.md §7.6 fixes the order, and the code is built around it:

| Step | What                                               | Why this position                                                                              |
| ---- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| 1    | Expand migrations: `better-compute migrate`        | New code may read new columns and tables; old code ignores them                                |
| 2    | Workers and api                                    | They read and write the expanded schema; old gateways still work against it                    |
| 3    | Gateways, one at a time, each drained first        | A gateway's restart costs its nodes a reconnect; one at a time keeps the rest of the region up |
| 4    | bcd, in waves within each pool's disruption budget | The daemon is the last to move, so the control plane always speaks its protocol                |
| 5    | Contract migrations, in the following release      | Only once no process of the old release is left                                                |

Two rules follow from it. A newer daemon is never put in front of an older control plane, and a database is never
migrated backwards: a release whose migrations ran cannot be undone by deploying the previous code (see
[rolling back](#rolling-back)).

## Before you start [#before-you-start]

* Read the [changelog](/docs/reference/changelog) between the two versions for new migrations, changed options and
  new module requirements.
* Take a backup you know how to restore: [backup and recovery](/docs/operations/backup-and-recovery). It is the only
  way back from a migration.
* Run `better-compute doctor` and clear its errors first. A half-done command-signing root rotation
  (`pki.rotation`), an unsigned quorum change (`pki.quorum-unsigned`) or overdue work (`workers.queue`) is easier to
  finish before the upgrade than to untangle in the middle of it.
* Check that no rollout is running (`better-compute rollouts list --tenant root`); a node can be in only one running
  or paused rollout.

## Step 1: migrate [#step-1-migrate]

```bash
better-compute migrate
```

`migrate` runs `compute.initialize()`: it checks every method's IAM action against the merged catalog and the
normative method table (`INVALID_CONFIG` before touching the store if one disagrees), applies the base schema and
every module's migrations, anchors the process clock to `compute_now()`, settles the trust bundle version, and
starts the modules. It is idempotent: run it as a one-off job on every deploy. The ledger, the naming rules and the
full list of migrations are on [database](/docs/operations/database#migrations); what matters for an upgrade is how
it decides what to run and what to refuse.

**What runs.** Migrations run in registration order: the base schema (owner `core`), the built-in modules, your
`modules`, then your `plugins`. Each runs in its own transaction and is recorded in `compute_migrations` in the same
transaction, so a failed one leaves nothing behind and is retried on the next run. A migration that fails is
reported as `INTERNAL_ERROR` naming it (`Migration <name> failed: …`). Migrations run without the executor's
statement timeout.

**Concurrent migrators.** Every migration transaction first takes `pg_advisory_xact_lock(1650683248, 1)`, a
transaction-level lock that is safe behind PgBouncer in transaction mode. Two processes migrating at once serialize,
and the second skips what the first applied.

**What it refuses.** Before applying anything, `migrate` compares the ledger with what this build knows and stops
with `INVALID_CONFIG` ("Refusing to migrate: … Run the better-compute release that migrated this database, or a
newer one.") when:

| Ledger holds                                                           | Meaning                                                  |
| ---------------------------------------------------------------------- | -------------------------------------------------------- |
| A migration numbered above this build's last one for an owner it knows | A newer release migrated this database                   |
| An unknown name at or below an owner's last number                     | A released migration was renamed or removed              |
| `compute_schema_version` above this build's base schema version        | A newer release applied base migrations this build lacks |
| A server below `server_version_num` 160000                             | PostgreSQL older than 16                                 |

The error's `details` carry `newer`, `unknown`, `schemaVersion` and `knownSchemaVersion`. Migrations of owners this
build has no migrations for at all are ignored, so removing a module from the configuration does not block startup;
its tables stay.

**Verify.** After the job, `better-compute doctor` reports `migrations` as an error when any migration of this
release and these modules is not in the ledger ("Migrations not applied: …").

### Expand and contract [#expand-and-contract]

The migrations in the tree today add tables, columns and indexes, and widen checks (`core/0015_inventory_images` and
`core/0028_volume_allocations` drop a `CHECK` constraint and add a wider one). None drops a column or table that an
earlier release reads. The one heavy migration is `core/0018_partitioned_history`: it renames `audit_events`,
`inventory_snapshots` and `usage_windows`, creates partitioned tables under the same names, copies every row with
its values intact and drops the old tables, all in one transaction, so those tables are locked for as long as the
copy takes. Plan for that on a large database. PLAN.md puts destructive changes ("contract": dropping a column the
old release still reads) in the release after the one that stopped using them, so that during step 2 the old and new
processes can share the schema. None has been needed so far.

> **serve --no-migrate and worker roles.** 
  The deployment pages run `migrate` once and start processes with `serve --no-migrate`. In the current code that works
  for `--roles api` and `--roles gateway`, but not for worker roles: `serve` calls `compute.start({ roles })` for
  `controllers`, `scheduler`, `sinks` and `iam`, and `start()` refuses with `INVALID_STATE` ("Call initialize() before
  start()") unless `initialize()` ran in the same process (`packages/server/src/index.ts`, `packages/cli/src/commands/serve.ts`).
  No test starts a worker with `--no-migrate`. Until that changes, start worker processes without `--no-migrate`:
  `initialize()` is idempotent and serializes on the migration lock, so it costs one ledger read when the job already
  ran. The consequence for upgrades: a worker of the old release that restarts after step 1 runs `initialize()`, finds
  the newer migrations in the ledger and refuses to start, so replace old workers with new ones rather than restarting
  them.

## Step 2: workers and api [#step-2-workers-and-api]

Roll api processes behind their load balancer as you would any stateless service, using `GET /readyz` for readiness
(see [health checks](/docs/operations/deployment#health-checks)). Clients retry failed calls with their idempotency
keys.

Workers can be replaced in any order, and old and new workers may run side by side for a while: shard and singleton
leases make every job single-writer whatever release holds it. Stop old workers with SIGTERM so they release their
leases at once; a worker that is killed loses them only when they expire, after 15 seconds (see
[high availability](/docs/operations/high-availability#worker-failover)). When a reconciler is new in the release, it
runs only once a new worker holds the lease for its shard or singleton.

The new processes publish the trust bundle again. Its version follows its content, so an upgrade that changes
nothing in the bundle keeps the version, and one that does (new roots, a quorum) takes the next version once; see
[PKI](/docs/operations/pki#signed-intent).

## Step 3: gateways [#step-3-gateways]

Upgrade gateways one at a time. `serve` handles SIGTERM by draining the gateway first: every daemon is told to
reconnect elsewhere (`Drain`, with a random delay of up to 60 seconds), the listeners close with a 2-second grace and
the `gateway:<id>` lease is released (see [shutdown](/docs/operations/deployment#shutdown)). Daemons redial through the
L4 balancer and re-home onto the remaining gateways; the next `Hello` replays whatever was in flight, and the
daemon's journal deduplicates it, so nothing runs twice. Wait for the drained gateway's sessions to settle on the
others before stopping the next one: `doctor`'s `fleet.gateway-headroom` warns when a gateway holds 80% of its
`maxStreams`.

A gateway killed instead of drained costs the same reconnect, only all at once; see
[failure modes](/docs/daemon/failure-modes#gateway-crash).

## Step 4: bcd [#step-4-bcd]

### Version skew [#version-skew]

Each `Hello` carries the daemon's `proto_min`, `proto_max`, `daemon_version` and `capabilities`. The gateway picks the
highest protocol version inside both ranges (`negotiateProtocolVersion`); when the ranges do not overlap it sets
`Welcome.upgrade_required`, and that session accepts only the `daemon.upgrade` command kind: no desired state, no
other command. The full rules, the capability labels (`bc.dev/cap.<name>=1`) and the per-feature gating are on
[version skew](/docs/daemon/protocol#version-skew). In short:

| Item              | Today                                                                                                                                                                                                                                                                                                                                                     |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Protocol versions | Exactly 1 on both sides (`PROTOCOL_VERSION = { min: 1, max: 1 }`, `ProtoMin`/`ProtoMax` in Go). No version beyond 1 is defined                                                                                                                                                                                                                            |
| A gateway's range | `createGateway({ protocol: { min, max } })`, default `PROTOCOL_VERSION`; for staged bumps and the skew tests                                                                                                                                                                                                                                              |
| Skew policy       | Daemons may lag N-1 minor versions before 1.0 and N-2 from 1.0                                                                                                                                                                                                                                                                                            |
| Skew matrix       | `tests/skew-matrix.test.ts`: FakeDaemon profiles for N (0.5.0), N-1 (0.4.0) and N-2 (0.3.0) against real dev stacks, each welcomed and adopted, with the kinds it lacks rejected `UNSUPPORTED`; a below-minimum profile gets `upgrade_required`                                                                                                           |
| What is recorded  | `node_sessions.daemon_version` and `proto_max` (`core/0008_session_versions`); `capabilities` (`core/0024_session_capabilities`)                                                                                                                                                                                                                          |
| Feature gating    | Placement places instances that need `instance.network-policy`, `instance.ssh-host-only`, `instance.placement` or `instance.volumes`, and reservations that book host resources (`devices.limits`) or bind through Kubernetes (`binding.kubernetes`), only on nodes that advertise them; `tunnels.open` needs `tunnel.instance-dial` for instance tunnels |

The skew profiles are FakeDaemon behaviours, not old bcd binaries: no older bcd release exists to run. `doctor`
reports `fleet.daemon-skew` when connected daemons speak a protocol below the server's minimum.

`UPGRADE_REQUIRED` is also a stable API error code (HTTP 426, gRPC `FAILED_PRECONDITION`) in `@better-compute/core`;
nothing in the server raises it today. On the daemon path the signal is the `Welcome.upgrade_required` flag, not an
error.

### Rolling bcd out [#rolling-bcd-out]

From v0.4 the daemon is upgraded through desired state, not by hand. The full guide is
[rollouts](/docs/guides/rollouts); the sequence is:

      #### Sign the release offline [#sign-the-release-offline]

    `better-compute release sign` writes `manifest.json` over goreleaser's output with a `--sequence` that must grow with
    every release, and `release cosign` adds signatures up to your threshold. Nothing on a server can sign one.
  
      #### Serve the binaries [#serve-the-binaries]

    Put each release's bare binaries where `rolloutModule({ downloadsUrl })` points, or let the api serve them with
    `downloads: { dir }` (see [serving the binaries](/docs/daemon/install#serving-the-binaries)).
  
      #### Register and roll [#register-and-roll]

    `releases.register` verifies the manifest under the configured pins and refuses a lower sequence (`rollback`).
    `rollouts.create` moves the version through waves (1%, 10%, 50%, 100% by default) within each pool's disruption
    budget, pauses itself when more than 5% of a wave fails, and soaks 600 seconds between waves.
  
      #### The node swaps itself [#the-node-swaps-itself]

    bcd takes only a manifest signed by the keys in its own `[update] release_pins`, checks the sequence against its
    release floor and the binary's SHA-256, stages it in the other of two slots, swaps with a 120-second deadline and
    confirms on the new binary's first `Welcome`; without one it swaps back. See
    [self-update](/docs/daemon/install#self-update).
  
A node too old to share a protocol version with the gateway is in an upgrade-only session; the rollout controller
sees its `proto_max` below the minimum and enqueues `daemon.upgrade` with the same target, so it can still be
upgraded (see [daemon.upgrade for restricted sessions](/docs/guides/rollouts#daemonupgrade-for-restricted-sessions)).

Nodes installed without `--release-pins` refuse every self-update and must be upgraded by re-running the install
script; see [upgrades before v0.4](/docs/daemon/install#before-v04-rerun-the-script). Workloads are not touched by a
bcd swap: in the simulator's upgrade test every VM keeps running and is booted exactly as often as before.

> **Status of self-update.** 
  bcd's self-update is implemented (de9f380, c139b9c) and `tests/e2e/bcd.test.ts` runs a real `daemon.upgrade` to a
  second signed build, including through `bcd helper` on an unprivileged node (2d67b25). No bcd release has been
  cut, so no real host has self-updated in production.

### Kubernetes DaemonSets [#kubernetes-daemonsets]

bcd running as a DaemonSet is upgraded by Kubernetes, not by rollouts; see [bcd on Kubernetes](/docs/daemon/kubernetes#upgrades).

## Step 5: contract migrations [#step-5-contract-migrations]

A release that drops something the previous release used ships its contract migration one release later. Run
`better-compute migrate` again only after no process of the old release is left, workers and gateways included
(check the `gateways` directory rows' version and the `workers` heartbeats).

## The API surface [#the-api-surface]

Clients (the TypeScript client, the CLI, MCP, your own code) talk to the api through a method surface that PLAN.md
§16 freezes for v1.0. `scripts/api-surface.mjs` runs in `pnpm lint:repo` and fails CI when a served method
disappears, its route, kind, IAM action or resource changes, an IAM alternative goes, or its input schema rejects an
input the frozen schema accepted; error codes may not disappear or change HTTP status. Additions pass. The rules and
the update commands are on [the API surface freeze](/docs/operations/testing#the-api-surface-freeze).

For an operator, this means an upgrade within the frozen surface does not break a client built against the older
release. A client built against a newer release can call methods an older server does not have: the older api answers
those with `NOT_FOUND` ("Endpoint not found"), so upgrade servers before clients that need new methods.

## Rolling back [#rolling-back]

| What                                  | How                                                                                                                                                                                                                                                                                      |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| api and worker code, before migrating | Deploy the previous build                                                                                                                                                                                                                                                                |
| api and worker code, after migrating  | Not possible on the same database: the old build refuses to migrate it, and old workers refuse to start (see the callout above). Restore the database from the backup taken before step 1 together with the old build; see [restore](/docs/operations/backup-and-recovery#restore-order) |
| Gateways                              | Deploy the previous build, one at a time, as in step 3, with `--no-migrate`: a gateway-only process starts without `initialize()`, while one that runs it refuses the newer ledger                                                                                                       |
| bcd, not yet confirmed                | `rollouts.abort` with `rollback: true` sends nodes still upgrading back; bcd's own watchdog swaps back an update that never got a `Welcome`                                                                                                                                              |
| bcd, confirmed                        | Not possible: the release floor rose, and bcd refuses every older binary. Roll forward with a new release at a higher sequence; see [rolling forward](/docs/guides/rollouts#undo-a-bad-release-by-rolling-forward)                                                                       |

## Checklist [#checklist]

```bash
better-compute doctor                                    # clean before you start
# back up the database and the PKI bundle (backup-and-recovery)
better-compute migrate                                   # step 1, once
# step 2: replace api processes, then workers (SIGTERM releases their leases)
# step 3: one gateway at a time; SIGTERM drains it
better-compute doctor                                    # migrations, fleet.gateway-headroom
better-compute releases register dist/manifest.json --tenant root
better-compute rollouts create --tenant root --version 1.4.0 --key bcd-1.4.0   # step 4
better-compute rollouts get <rolloutId> --tenant root --query nodes
```

  - [Database](/docs/operations/database#migrations): The ledger, the order and every migration.

  - [Rollouts](/docs/guides/rollouts): Signed releases, waves and the failure gate.

  - [Version skew](/docs/daemon/protocol#version-skew): Negotiation, capabilities and the skew matrix.

  - [Backup and recovery](/docs/operations/backup-and-recovery): The way back from a migration.


# Use the docs with AI (/docs/reference/ai)

> Connect AI coding assistants to these docs through their MCP endpoint, or hand them the site as Markdown with llms.txt and per-page Markdown.



AI coding assistants write better Better Compute code when they can read the real reference instead of guessing
method names, IAM actions or CLI flags. This site offers the same content to tools as it does to people, in three
forms: an MCP server an assistant can query while it works, plain-text indexes for one-shot context, and a Markdown
version of every page.

> **Two different MCP servers.** 
  This page is about the **documentation's** MCP endpoint, which answers questions about Better Compute. To let an
  agent act on a deployment (list pools, reserve GPUs, create instances), use the `@better-compute/mcp` package
  instead; see [MCP](/docs/guides/clients/mcp).

## Connect an assistant with MCP [#connect-an-assistant-with-mcp]

The [Model Context Protocol](https://modelcontextprotocol.io) lets an assistant call tools while it works. The docs
serve an MCP endpoint at `/api/mcp` over Streamable HTTP. It needs no sign-in and only reads. The server introduces
itself as `better-compute-docs` and offers six tools:

| Tool                 | Input                      | What it returns                                                                                                                                                                                                                                                        |
| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `search`             | `query`, optional `locale` | Full-text search over every page, the same index as the search dialog, as JSON.                                                                                                                                                                                        |
| `list_pages`         | none                       | Every page with its URL and one-line description (the body of `/llms.txt`).                                                                                                                                                                                            |
| `get_page`           | `url`                      | One page as Markdown, by its path, for example `/docs/guides/capacity/reservations`.                                                                                                                                                                                   |
| `lookup_api_method`  | `method`                   | One API method: its HTTP route, the server call `compute.api.{group}.{method}(credential, input)`, the browser client call, its kind, the IAM action and resource it is decided on, its module, and whether this release serves it or it is planned for a later phase. |
| `lookup_error_code`  | `code`                     | What a `ComputeError` code means and its HTTP status, for example `CAPACITY_UNAVAILABLE` or `ACCESS_DENIED`.                                                                                                                                                           |
| `lookup_cli_command` | `command`                  | Usage, flags (with defaults and environment variables) and examples of one `better-compute` command, for example `dev` or `fleet apply`.                                                                                                                               |

The lookup tools are forgiving about how you name things. `lookup_api_method` accepts `reservations.create`,
`compute.api.reservations.create(...)`, `client.joinTokens.create`, a route such as
`/api/compute/reservations/create`, or a reference URL such as `/docs/reference/api/reservations#create`; an
unknown group or method answers with the list of valid ones. `lookup_cli_command` drops a leading
`better-compute` and any flags, and matches the longest command name the query starts with, so
`fleet plan fleet.json` finds `fleet plan`. Error codes are matched case-insensitively.

  **Claude Code:**

    ```bash
    claude mcp add --transport http better-compute-docs https://better-compute.com/api/mcp
    ```
  
  **Cursor:**

    ```json title=".cursor/mcp.json"
    {
      "mcpServers": {
        "better-compute-docs": { "url": "https://better-compute.com/api/mcp" }
      }
    }
    ```
  
  **VS Code:**

    ```json title=".vscode/mcp.json"
    {
      "servers": {
        "better-compute-docs": { "type": "http", "url": "https://better-compute.com/api/mcp" }
      }
    }
    ```
  
These docs are served at `https://better-compute.com`. When you run them locally with
`pnpm --filter @better-compute/docs dev`, use `http://localhost:4100` instead. The links the lookup tools print use
the site's canonical origin, `DOCS_SITE_URL`, which defaults to `https://better-compute.com` in a production build
and to the local address under `next dev`.

A session might look like this: the assistant searches for "extend a reservation", reads
`/docs/guides/capacity/reservations` with `get_page`, then calls `lookup_api_method` with `reservations.extend` to get
the exact route and IAM action before it writes the call.

## Give a model the whole site [#give-a-model-the-whole-site]

When a tool cannot call MCP, paste or fetch the site as plain text:

| URL               | Content                                                                                            | Use it for                                    |
| ----------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| `/llms.txt`       | An index: every page's title, URL and description, under a one-paragraph summary of Better Compute | Letting a model pick which pages to read      |
| `/llms-full.txt`  | Every page as Markdown in one file                                                                 | Loading everything into a long context window |
| `/docs/<page>.md` | One page as Markdown                                                                               | Quoting a single page precisely               |

Any docs URL also returns Markdown when the request asks for it with `Accept: text/markdown`, so agents that fetch
pages get Markdown without special URLs:

```bash
curl -H 'Accept: text/markdown' https://better-compute.com/docs/guides/quickstart
curl https://better-compute.com/docs/guides/quickstart.md          # the same answer
```

Both are rewritten to `/llms.mdx/docs/<page>/content.md`, which answers with a `Link: rel="canonical"` header that
points at the HTML page. Each page's **Copy Markdown** button and its view-options menu use the same Markdown.

The Markdown is not the raw MDX source. Each page starts with its title, URL and description, comments are dropped,
visual-only components are removed, the API endpoint boxes become a line of text with the route, authorization and
calls, and the [glossary](/docs/reference/glossary) becomes every term as a heading with its definition.
`/llms.txt` and `/llms-full.txt` carry `X-Robots-Tag: noindex`, so search engines index the pages themselves.

## Keep answers accurate [#keep-answers-accurate]

The reference pages, and the MCP lookup tools, come from the code: the generator builds a control plane with every
module and reads its method registry, command kinds, error codes and CLI manifest (see
[reference](/docs/reference)). They match the version of Better Compute these docs were built from, including
which methods that version serves and which PLAN.md lists for a later phase.

Two things to tell an assistant:

* **Planned is not served.** `lookup_api_method` says `Status: planned for <phase>` for methods this release does not
  route; the HTTP endpoint answers `NOT_FOUND` for them. Check the status before generating code against a method.
* **The daemon is not released.** `bcd` is a Go binary whose v0.1 node agent (with tunnels) is built and tested end to
  end, and whose site and ramdisk modes are written with Go unit tests, but no binary is published. Its VPS host
  runs instances on `libvirt-kvm` or `cloud-hypervisor` (both tested only against fake tools, never on real
  hardware) or, on fake-hardware builds, a file-backed runtime; its sim mode (many node agents in one process) runs
  only on fake-hardware builds; self-update and the WebSocket fallback are written, and GPUs are read through `nvidia-smi` rather than the NVML
  library (cb987e5). The
  TypeScript `FakeDaemon` in `@better-compute/testing` runs those paths. See
  [what bcd does today](/docs/daemon#what-bcd-does-today).

When an answer disagrees with your installed version, trust the TypeScript types in your editor: every server method
is typed end to end from the `betterCompute()` instance and the `@better-compute/client` client.


# Changelog (/docs/reference/changelog)

> Release notes for the better-compute packages and the documentation site, every notable change, newest first.



# Changelog [#changelog]

All notable changes to the better-compute packages are documented here. The format is based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html). Every package in `packages/` shares one version.

## Unreleased [#unreleased]

### Added [#added]

* **Managed Kubernetes clusters** (new package `@better-compute/kubernetes`; `core`, `capacity`, `events`).
  `kubernetes({ providers })` adds the groups `kubernetesClusters` (`create`, `list`, `get`, `update`, `delete`,
  `credentials`, `versions`) and `kubernetesNodePools` (`create`, `list`, `update`, `delete`) on the resource kind
  `kubernetes-cluster` (`compute:kubernetes:create|read|update|delete|connect`; tenant `compute:kubernetes:read`).
  Clusters (Standard with node pools, or Autopilot) live on a provider pool visible to the tenant, platform-hosted
  (a root pool offered to the tenant) or on the tenant's own account; the provider runs the control plane. The
  `kubernetes.clusters` reconciler converges them through `ProviderImpl.kubernetes` (a 30 s sweep, a 5 min refresh).
  `credentials` returns a kubeconfig whose token the provider mints for a per-identity, per-role subject (`edit`
  also needs `compute:kubernetes:update`, `admin` `compute:kubernetes:delete`); the token is never stored, logged or
  published. Events `kubernetes.cluster.*` and `kubernetes.nodepool.*`; meter `kubernetes.cluster.hours` (new in the
  meter catalog) as the new usage source kind `kubernetes` (migration `kubernetes/0001_clusters` extends the
  `usage_windows` source CHECK); `tenantDeleted` deletes a deleted tenant's clusters.
* **Managed Kubernetes on GKE** (`@better-compute/provider-gcp`). The GCP provider implements the optional
  `ProviderImpl.kubernetes` contract (capability `kubernetes`): Standard and Autopilot GKE clusters in the account's
  project, non-blocking `ensureCluster`/`deleteCluster` over GKE operations, in-place release channel, version and
  authorized-network changes (one update per call; downgrades and mode, location, network or private-node changes are
  `INVALID_STATE`), node pools (size and autoscaling in place), `versions` from `serverConfig`, and `credentials`: a
  service account per subject in `bc-access`, bound to `view`/`edit`/`cluster-admin`, with a bound TokenRequest
  token, minted over the cluster's Kubernetes API trusting only the cluster CA. New options `containerEndpoint` and
  `kubeTransport`. See the provider-gcp README, "Managed Kubernetes (GKE)", for the IAM roles.
* **Amazon Web Services** (new packages `@better-compute/provider-aws`, `@better-compute/aws-core` and
  `@better-compute/aws`, all left out of releases until the AWS program is ready; `core`, `vps`, `capacity`,
  `server`, `pki`, `inventory`, `gateway`, bcd; plan `docs/superpowers/plans/2026-10-18-aws-integration-program.md`).
  No AWS SDK: `aws-core` signs SigV4 requests and resolves endpoints and credentials.
  * **EC2 instances** through provider pools: offers with vCPU quota headroom, create/get/delete with `ClientToken`
    idempotency and eventual-consistency retries, power, reinstall by replace-root-volume (SSM images resolved to an
    AMI of a trusted owner, the architecture checked), discover/claim/release for `instances.attach`. Accounts take
    a platform-assigned ExternalId, or go keyless through web identity federation (`aws onboarding-template`,
    `providerAccounts.get` shows the onboarding; instances name the AWS role their workloads assume, and the workload
    broker mints `sts.amazonaws.com` tokens only for an allowed role). `aws iam-policy` renders tag-scoped (ABAC)
    policies per phase, and `aws trust-policy` the cross-account trust.
  * **Networks**: VPCs with zone-scoped subnets, a network range, private service access (`privateAccess`),
    firewall rules as security groups and network ACL denies.
  * **EBS volumes and snapshots** through the `providerVolumes` flow (types, IOPS, throughput, forced detach).
  * **Capacity markets**: Spot offers, launches and reclaims (metered as spot); On-Demand Capacity Reservations and
    Capacity Blocks as reservations and offers; `providerAccounts.reservationOfferings`, `purchaseReservation`,
    `purchases` and `purchaseProposals` buy capacity root-only, confirmed and exactly once across lost answers.
    Offer prices from the Price List and Spot price history; Local Zone offers stay unpriced.
  * **EKS**: `ProviderImpl.kubernetes` on EKS (managed node groups that follow control-plane upgrades, Neuron groups
    tainted, per-subject access entries) and bcd's `k8s-join` mode `eks`; `providerAccounts.clusters`.
  * **Provider events** (`ProviderImpl.pollEvents`, the `vps.provider-events` worker): EventBridge intake through SQS
    (`aws event-rules`) or scheduled-event polling; Spot interruption and rebalance notices reach capacity as
    `noticeNodeInterruption`, and bcd reports the IMDS notice as a node condition.
  * **EC2 hosts**: bcd's IMDSv2 client, `bcd join --attestor cloud-iid` (verified with `packages/pki`), the `cloud`
    inventory section (`cloud.*` and `efa.count` labels); instances never reach EC2's metadata service.
  * **Shared filesystems** (new `filesystems` group: `create`, `list`, `get`, `delete`; `ProviderFilesystems`): EFS
    and FSx for Lustre on a network, mounted by instances at boot, billed as `filesystem.<kind>.gib.hours`. EFA
    launches and placement groups; Capacity Blocks proposed 48 h before pending calendar reservations.
  * **FinOps**: month-to-date spend and a spend guard, list prices from the Price List, cost allocation tags, a tag
    policy, budget templates, `aws organization-accounts`, `aws quotas`, `aws quota-request` and
    `aws governance-policy`.
  * **Running better-compute on AWS** (`@better-compute/aws`): Secrets Manager, Parameter Store, KMS sealing, S3 blob
    and snapshot stores, EventBridge/SNS/SQS/Kinesis/Firehose sinks, ACM Private CA and KMS-key certificate
    authorities, RDS IAM auth tokens, a CloudWatch EMF exporter, and `aws deploy-templates` (ECS Fargate behind ALB
    with WAF and an NLB passthrough). See the docs page "Operating on AWS".
* **DigitalOcean** (`@better-compute/provider-digitalocean`, still left out of releases; new package
  `@better-compute/cloud-resources`, released as `better-compute/cloud-resources`; `core`, `vps`, `capacity`,
  `events`, `mcp`, `console`, `admin`, bcd; plan `docs/superpowers/plans/2026-10-18-digitalocean-program.md`).
  * **Provider volumes, snapshots and addresses** (new groups `providerVolumes`, `providerSnapshots`,
    `providerAddresses`; provider-neutral contracts with conformance cases): Block Storage volumes and snapshots,
    reserved IPv4 and IPv6 addresses, `deleteWithInstance`, metered by `vps.ancillary-usage`. NFS shares are shared
    provider volumes on many instances of a tenant network (`vps.share.gib.hours`).
  * **Images and machines**: catalog images resolved per tenant, Spot offers declared by the provider with
    `PoolPolicy.replacePreempted`, Bare Metal GPUs as metal hosts (`digitaloceanMetal`), Droplet metadata in bcd's
    inventory.
  * **DOKS**: `ProviderImpl.kubernetes` on DigitalOcean Kubernetes (CPU, GPU and Spot GPU node pools, autoscaling,
    in-place upgrades, per-subject credentials through the cluster's own API); DOKS worker Droplets are never
    discovered, claimed or deleted.
  * **Managed services** (`cloudResources` group: `kinds`, `catalog`, `signals`, `create`, `list`, `get`, `update`,
    `delete`, `discover`, `adopt`, `release`): one governed, audited, metered API for a provider's managed services
    (Managed Databases, App Platform, Functions, 1-Click apps, Partner Network Connect, Spaces, CDN, Container
    Registry, and the AI platform: dedicated inference booked against the GPU quota, inference keys behind a metered
    proxy, knowledge bases, guardrails, agents). IAM decides on `cloud-resource/{id}` by kind and family; secret
    outputs are sealed and returned as SecretRefs; `cloud.resource.hours` plus sampled kind meters.
  * **Edge** (new groups `certificates`, `loadBalancers`, `dnsZones`, `dnsRecords`; `networks.natGateway`,
    `networks.peer`, `networks.routes`): Let's Encrypt and custom certificates, regional load balancers
    (`vps.load-balancer.hours`), DNS zones and records (`instances.create({ dns })`), NAT gateways, VPC peerings,
    VPC routes and BYOIP addresses (preview).
  * **Operations**: OAuth account connection (`providerAccounts.connectSetup`, `connect`), `check` (with the
    credential's rate limit), `findings`, `billing` (the account's own bill) and `billing.leaks`; external host metrics
    of provider-pool VMs without bcd (`vps.external-metrics`); the admin panel's Provider accounts module.
* **Kubernetes fleet pools** (`@better-compute/kubernetes`, `fleet`, `provider-kubernetes`, `testing`, `server`;
  docs guide "Kubernetes fleet pools"). A GPU pool's `policy.substrate` names one of the owner's provider accounts
  and the node shapes it wants; the `kubernetes.substrates` reconciler creates (or adopts) a GKE, EKS or DOKS cluster
  and its node pools, runs bcd on the nodes as a DaemonSet, and the nodes join the pool, booked with
  `binding: 'kubernetes'` reservations. The substrate owns only the nodes its tokens admitted, matches them to their
  Kubernetes Node by name and uid, ends the bookings of a node that is gone, holds a version bump under
  `upgrades: 'drain'` until bookings end, and revokes its cluster-admin identity when an adopted cluster is torn down.
  New method `kubernetesSubstrates.get`. FleetConfig documents gain `kubernetesClusters` (plan, export and
  `fleet.apply` create clusters, node pools and substrate pools); the same document runs on GKE, EKS and DOKS.
  `kubernetesProviderConformanceCases` in `@better-compute/testing` check every provider, including what a
  credential's role allows.
* **Admission for every caller** (U-02; `@better-compute/capacity`, `vps`, `core`, `iam`, `postgres`,
  `provider-proxmox`, `provider-hetzner`). Entitlements and quotas are enforced inside the admission transaction
  of `instances.create`, `instances.attach`, `instances.resize` and `sandboxes.create`, whoever calls.
  * **Counted resources.** `instances` (one per created or attached instance, hosted or provider) and `sandboxes`
    (one per sandbox; a sandbox's instance is not an `instances`) are quota resources. A missing row limits
    nothing; each instance records what its admission took (`quota_charged`) and gives back exactly that.
    `chargeQuota` now returns the charge to record; `countQuotaResources`, `tenantQuotaKey`.
  * **Tenant-level quota.** Quota rows with `poolId: '*'` (`quotas.set`, `quotas.get`; `tenantQuotaScope`) cap
    a tenant's total over every pool: `vcpu`, `memoryGiB`, `diskGiB`, `gpu`, `gpu.<gpuModel>` (GPUs of
    pools of one model), `instances`, `sandboxes`. They move with every pool charge and refund; a new scalar row
    starts at the tenant's current total. Refusals are `QUOTA_EXCEEDED` with `details.scope` (`pool` or
    `tenant`). `pools.update` refuses a `gpuModel` change while such a tenant has GPUs charged on the pool.
  * **Flavor allowlists.** `pools.offer({ flavors })` and `pools.updateOffer({ flavors })` (`null` clears) list
    the flavors a tenant may use on a pool (`flavor-not-offered`); `quotas.set({ flavors })` lists the flavors
    that may consume one row (`flavor-not-allowed`), both `QUOTA_EXCEEDED`. Attach judges a machine by the
    pool's flavor whose offer is its type. Pools and quota rows show their `flavors`.
    `assertFlavorEntitled`, `parseFlavorAllowlist`.
  * **IAM attributes.** `compute-instance` gains `poolId`, `flavor`, `region` and `provider`, plus one
    `labels.<key>` attribute per instance label (in key order while the row's 64 attributes have room, else
    `labelsTruncated: true`), on create, attach, resize and sandbox instances.
  * **Admission policy hook.** `vps({ admissionPolicy })` (core `AdmissionPolicy`, `AdmissionPolicyRequest`)
    runs the host's rules in the admission transaction, after the flavor allowlists and before quota is charged.
  * **Inherited offers.** New read `pools.offers` lists the offers that reach a tenant: its own and its
    ancestors' made with `includeDescendants`, nearest first, by priority, each with the tenant's own
    quota rows on the pool (never an ancestor's). The tenant tree comes from
    `capacity({ tenantAncestors })` (`betterCompute({ capacity })`); `iamTenantAncestors` in
    `@better-compute/iam` reads it from better-iam.
  * **Address policy.** `ProviderInput.endpoints` declares the endpoints an account's config (or an API URL
    override) chooses; account validation refuses non-https, internal names and private, link-local (metadata),
    loopback, CGNAT, multicast, reserved, unique-local and cluster addresses (`INVALID_CONFIG`,
    `endpoint-address`), and clients call through `ProviderContext.fetch`, which checks again and pins the
    connection to the checked address. `vps({ addressPolicy })` adds cluster ranges or exemptions
    (`createAddressPolicy`, `pinnedFetch`). Proxmox declares its `url`; Hetzner Cloud and Robot their
    `endpoint` override. Compute Engine declares none: its hosts stay pinned by host options.
  * **Migrations:** core `0032_admission_flavors` (`pool_offers.flavors`, `quota_usage.flavors`); vps
    `0048_quota_counts` (existing `instances`/`sandboxes` rows start at what is live).
  * **Lock order.** Among quota rows, counted resources lock first, then the pool's row before the tenant's
    (`*`), so a tenant row created while a charge is in flight still counts it; a tenant's `gpu.<model>` row
    takes the pool's model once the pool's `gpu` row is locked, and `pools.update` locks those rows before
    changing `gpuModel`. A deleted instance refunds its recorded charge before its allocation is released.
  * `@better-compute/testing`: `computeFixture({ capacity })`.
* **Destructive CLI commands ask before they act** (`@better-compute/cli`). `metal retire`, `metal undeploy` and
  `metal power --action off|reset|cycle` now need `--yes`, as `nodes delete` did, or a yes at an interactive
  terminal (a prompt on stderr when stdin and stderr are terminals); without either they stop with a usage error
  before calling the API. Scripts that run them must add `--yes`. One helper decides for all four
  (`confirmDestructive` and `yesFlag`, exported for project commands; embedders answer the prompt through the new
  optional `CliIO.confirm`). `on`, `graceful-off`, `graceful-restart` and `nmi` are unchanged.

### Security [#security]

* **`metal.enroll` let a tenant read out another tenant's or the platform's secrets (`@better-compute/metal`).**
  `bmc.credentials` took any secret reference and `bmc.address` any address, both from the caller, and the site
  agent fetches the one and presents it to the other: a tenant with `compute:nodes:provision` on one of its pools
  could enroll `secret://<other-tenant>/…`, `env://…` or `vault://…` against a host it controls and receive the
  value as a BMC login. Now a `secret://` reference must be the enrolling tenant's own (`secret://<tenantId>/<id>`),
  `env://` and `vault://` references are taken from root callers only, and anything else is `ACCESS_DENIED`. A
  machine enrolled before this with another tenant's `secret://` is never sent a BMC command again (its workflows
  and `metal.power` fail with `ACCESS_DENIED`): enroll it again with its own secret.

* **The CLI sent a saved better-iam session to any `--url` (`@better-compute/cli`).** Compute's own saved profiles
  go only to the deployment they were saved for, but the session `better-iam login` saves was used for whatever
  `--url` (or `BETTER_COMPUTE_URL`) named: `nodes list --url https://staging…`, a typo or a repository's default
  URL received the production session as a bearer token. Now a better-iam session issued by a remote endpoint goes
  only to a `--url` of the same origin, and one issued in process only to an in-process deployment; otherwise it is
  passed over (compute's own profile for that deployment is tried next, and the `MISSING_ENV` hint says which
  credential was not used and why). Deployments that serve IAM and compute from different origins need
  `better-compute auth login --url …` or `BETTER_COMPUTE_TOKEN`.

* **Client: `paginate()` and `NETWORK_ERROR`** (`@better-compute/client`). `paginate(list, input, options?)` is an
  async iterator over every item of a list method, following each page's `cursor` and fetching a page only when the
  iteration reaches it (`for await (const node of paginate(client.nodes.list, { tenantId }))`); a page that does not
  advance ends it with `INVALID_RESPONSE`. A transport failure (DNS, a refused connection, TLS, a redirect) is a
  `ComputeClientError` with code `NETWORK_ERROR`, status 0, the request id and the original error as `cause`, where
  it used to be fetch's bare `TypeError`; the caller's own abort is passed through unchanged.

* **One address classifier for egress checks** (`@better-compute/core`, `vps`, `events`). core gains
  `parseAddress`, `addressFamily`, `classifyAddress`, `parseCidr`, `addressInRange`, `embeddedV4`,
  `nonPublicRanges` and `isInternalHostName` (pure, browser-safe): the classifier the provider endpoint policy had,
  now shared with the webhook URL check. `AddressCategory` moved to core; vps re-exports it.

* **Gateway limits and what they show** (`@better-compute/gateway`). New options `joinsPerSecond`, `helloTimeoutMs`
  (10 s) and `maxQueuedFrames` (4 096); `GatewayStatus.pendingStreams`; the handshake outcome `timeout`.
  `gatewayRenewHandler` takes `attemptsPerSecond` (50).

* **`EnqueueCommand.requestHash`** (`@better-compute/core`, `server`; migration `core/0061_command_request_hash`). A
  caller's fingerprint of its request, stored with the command: the same idempotency key with another fingerprint is
  `CONFLICT` (`details.reason: 'idempotency-key-reused'`). Optional; without it the kind is compared as before.

* **`ProviderResources.releasable(kind)`** (`@better-compute/core`). Optional and pure: a provider says which kinds
  `release` works for, so a release it cannot do is refused up front.

* **`WatchOptions.onError`** (`@better-compute/events&#x60;), &#x2A;*`RevocationFeedOptions.fullReloadMs`**
  (`@better-compute/pki&#x60;), &#x2A;*`AuthzProjectionOptions.maxStalenessMs`** (`@better-compute/iam&#x60;) and
  &#x2A;*`reapInstanceDrafts`*&#x2A;, &#x2A;*`upsertInstanceAuthz`*&#x2A;, &#x2A;*`lostMachineWindowMs`** (`@better-compute/vps`): see the
  fixes they belong to below.

### Fixed [#fixed]

* **Webhook URLs could reach internal addresses** (`@better-compute/events`). The SSRF check judged addresses as
  text: `https://[::ffff:10.0.0.5]/` becomes `::ffff:a00:5` once parsed, which the dotted-only pattern missed, as it
  did 6to4 (`2002::/16`) and NAT64 forms, so a tenant webhook could be pointed at private and metadata addresses.
  Delivery also resolved the host again after checking it (DNS rebinding). Addresses are now judged by value (core's
  `classifyAddress`), internal host names (single-label, `.local`, `.svc`, …) are refused when a webhook is saved,
  and the default sender connects to the address the check resolved (`postPinned`). A sender passed as `fetch` is
  called as before.

* **One slow webhook endpoint stalled every tenant's deliveries** (`@better-compute/events`). A webhook's claimed
  rows were sent one after another at up to the timeout each. Its first failure in a pass now ends its turn: the
  rest go back with their attempt and come due within a minute. Delivery outcomes are written only while the row is
  still that pass's claim, so a pass that outlived its lease cannot turn a delivered row back to pending.

* **Outbox retention deleted events the webhooks had not read** (`@better-compute/postgres`). `pruneBehindCursors`
  floored on `outbox_cursors` only; the webhook fan-out keeps its position in `event_cursors`. With a sink ahead of a
  fan-out lagging more than `minAgeSeconds`, the rows in between were deleted unread; with no sink the outbox was
  never pruned. The floor is the slowest cursor of both tables.

* **The watch stream showed database errors to tenants** (`@better-compute/events`). An unexpected failure was sent
  verbatim as the SSE error event (a connection failure names hosts and ports). The client now gets `UNAVAILABLE`
  with a fixed message and the host hears the error through `WatchOptions.onError`. A cursor whose numbers do not fit
  the columns is `INVALID_INPUT` before any query.

* **A reused idempotency key answered with the first request's result** (`@better-compute/server`).
  `commands.run` compared the kind only, so the same key with another payload or batch returned the first command;
  it is now `CONFLICT`. `secrets.put` answered the same key with another value at the old version, silently
  dropping a rotation; it is a retry only when the value equals what the current version holds.

* **`commandBatches.list`, `scope` and `members` paged with `next`** (`@better-compute/server`). Every other list
  returns `cursor`; a generic loop stopped after the first page. They return `cursor` now, with `next` kept as a
  deprecated alias of the same value.

* **Unauthenticated callers could lock nodes out of the gateway** (`@better-compute/gateway`). Join (no client
  certificate) shared the handshake bucket with session opens, so a flood of joins refused every reconnect; joins
  have their own bucket. `POST /v1/renew` charged the per-node hourly limit before verifying anything, keyed by the
  node id of an unverified certificate: six forged requests an hour kept a node from renewing until its certificate
  expired. Only renewals JoinService granted are counted, and one bucket bounds unverified attempts.

* **Streams that never said Hello, and streams that outran the gateway** (`@better-compute/gateway`). A stream
  admitted by `open()` that sent no Hello was never reaped and did not count toward `maxStreams`; it now holds a
  place and is closed `DEADLINE_EXCEEDED` after `helloTimeoutMs`. A stream's inbound frames queued without limit
  behind the database; past `maxQueuedFrames` it is closed `RESOURCE_EXHAUSTED`.

* **A close frame over 125 bytes** (`@better-compute/gateway`). `WsConnection.close` sized the frame from the whole
  reason but wrote 120 characters of it: a long reason made an illegal control frame the peer refused, losing the
  status. The reason is cut to 123 UTF-8 bytes on a character boundary.

* **The MCP handler buffered the whole body before checking its size** (`@better-compute/mcp`). It reads the stream
  with a byte limit now (it compared UTF-16 length), and refuses a declared `content-length` over it unread.

* **Deleting an instance could leave its cloud machine running** (`@better-compute/vps`; migration
  `vps/0124_instance_create_attempted`). A provider create runs outside any transaction, and its answer can be lost
  (a timeout after the provider accepted, a delete racing it). The row then had no machine id, and the delete
  finished at once, refunded quota and left the machine running, unbilled. A create whose row changed meanwhile now
  keeps the machine's id, and a delete with an unanswered create looks the machine up by its ownership label
  (`ProviderImpl.list` or `discover`) for `lostMachineWindowMs` (3 min) before it finishes. Such a delete can
  therefore stay `deleting` for up to three minutes where it used to finish immediately.

* **A hosted instance that could not get an address kept its booking** (`@better-compute/vps`). Placed by the
  reconciler and then refused (the pool's address range was full), it moved to `error` in the transaction that had
  just booked its allocation and quota, with no allocation id on the row. Scheduling runs under a savepoint now.

* **IAM judged instances by the labels they were created with** (`@better-compute/vps`). `instances.update` changed
  labels without touching the authz projection, and attach never projected labels. `upsertInstanceAuthz` writes the
  row from the instance as it is; update calls it when labels change, and attach passes its labels to the projection
  and to the admission policy.

* **Every provider poll bumped an instance's version** (`@better-compute/vps`). `instances.update` with
  `expectedVersion` from a form open longer than one poll was always `CONFLICT`. A poll that reads nothing new
  writes nothing.

* **Smaller vps fixes.** `LabelsNotApplied` stayed forever once the tenant put the labels back; the warm-pool pass
  read every warm instance a template ever had; drafts a refused create left behind blocked deleting their network
  or provider account (collected after a day by `reapInstanceDrafts`); the default `zoneRegion` made a Compute
  Engine AI zone (`us-south1-ai1b`) its own region.

* **A slow worker could delete a work row nobody had processed** (`@better-compute/server`). Work is settled by
  `(kind, key, gen)`, and `gen` starts again at 1 once a key's row is deleted: a worker whose claim had run out could
  report DONE against a fresh, unclaimed row and delete it, losing that wake-up until the next resync. Completion
  deletes only a claimed row.

* **A node could land in two active rollouts** (`@better-compute/rollout`). Two concurrent `rollouts.create` over the
  same pools both passed the "already in a rollout" check; creates of one tenant take turns under an advisory lock.

* **A late row could be destroyed with its history partition** (`@better-compute/postgres`).
  `dropEmptyHistoryPartitions` locks the partition before checking that it is empty.

* **A revocation committed late was never learned** (`@better-compute/pki`). The revocation feed tails a 60 s window
  by the revoking transaction's start time; a revocation committing further behind than that was missed until
  restart. The feed reads the whole unexpired set again every `fullReloadMs` (5 min).

* **The authz projection kept answering while its tail failed** (`@better-compute/iam`). After `maxStalenessMs`
  (30 s) of failed passes it is not ready (the resolver denies), and the first pass that works reloads every row.

* **Deleting a tenant no longer destroys what it adopted** (`@better-compute/cloud-resources`). The tenant teardown
  marked every live cloud resource for deletion, so a resource the tenant had adopted on its own provider account (a
  production database that existed before better-compute) was deleted at the provider. Adopted resources are now
  released (ownership labels removed, left running; `cloud.resource.releasing` with `reason: 'tenant-deleted'`);
  resources better-compute created are deleted as before, those created inside an adopted resource first.

* **A release that cannot complete no longer strands the resource** (`@better-compute/cloud-resources`,
  `@better-compute/provider-digitalocean`). `cloudResources.release` was accepted for kinds the provider cannot
  release (a DigitalOcean Functions namespace, a model access key): the row went to `error` still wanting to be
  released, and `cloudResources.delete` then did nothing, so it kept its name, its quota slot and its hourly fee for
  good. Release is now refused up front for such a kind (`UNSUPPORTED`, `details.reason: 'release'`; the provider
  says which kinds through `releasable(kind)`, which the DigitalOcean provider implements), and `delete` overrides a
  release that is still pending. A row already stranded recovers on the next sweep: the release of a resource
  better-compute created is withdrawn, an adopted one is let go without the provider call.

* **The DigitalOcean storage kinds are served** (`@better-compute/provider-digitalocean`). `do.spaces-bucket`,
  `do.spaces-key`, `do.cdn-endpoint`, `do.container-registry` and `do.registry-garbage-collection` were documented
  but never reached the provider's `resources`, so `cloudResources.kinds` did not list them and a create was
  `UNSUPPORTED`. `digitaloceanProvider()` now appends them to its long-tail kinds (same API client and request
  budget), and `spacesFetch`, `spacesEndpoint`, `spacesKeyFullAccess` and `spacesTenantBucketAdopt` are provider
  options (`DoStorageOptions` is now an alias of `DoProviderOptions`). The storage kinds use core's contract types
  (the `Do…` request, state and definition types are aliases), implement `find`, and get the request's
  `accountScope` and `secretOutputs`: a CDN endpoint's `certificateId`/`customDomain` are refused on a shared
  account (`accountWide`), and a Spaces key whose secret was never sealed answers `error` instead of `ready`.

* **A DNS zone whose marker write failed is no longer orphaned** (`@better-compute/provider-digitalocean`). When
  `POST /domains` succeeded and the marker TXT write then failed (a 429, say), the zone was left unmarked: every
  later `ensureZone` was `CONFLICT` (`zoneExists`) and `deleteZone` `INVALID_STATE` (`notManaged`). `ensureZone`
  now deletes the empty domain it just made when the marker cannot be written; if that fails too, the same client
  marks (or deletes) the zone on a later call while it still holds only DigitalOcean's apex NS and SOA records.

* **A failed indexing job no longer locks a knowledge base** (`@better-compute/provider-digitalocean`). After a
  `do.knowledge-base` indexing job ended failed, `ensure` returned `error` before it looked at the sources, so no
  source change was ever applied and no new job started. A failed job is still reported, but a source change (or a
  job still owed) is now applied and starts a new job; only a job still running, or a database that is not online,
  holds a change back.

* **Stuck deletes and releases back off** (`@better-compute/cloud-resources`). A cloud resource whose delete or
  release the provider refuses for good was queued by the 30 s sweep forever, one provider call per row per pass.
  Such a row is now asked again 30 s, 1 min, 2 min, … apart, up to once an hour (`resourceMaxBackoffMs`), and a busy
  provider's own `retryAfterMs` is honoured (30 s to an hour) instead of a fixed 30 s. New migration
  `cloud-resources/0006_retry_backoff` (`attempts`, `next_attempt_at` on `cloud_resources`); a change through the
  API still wakes the resource at once.

* **A model access key is not regenerated on every poll while its rename fails**
  (`@better-compute/provider-digitalocean`). A `do.inference-key` rotation regenerates the secret and then renames
  the key (the name records the rotation). When the rename kept failing, each 10 s pass saw the old rotation in the
  name and regenerated again, invalidating the key the tenant had just been given. The following passes now retry
  the rename alone while the caller holds that key sealed.

* **The wait of a DigitalOcean 429** (`@better-compute/provider-digitalocean`). `retryAfterMs` came only from
  `ratelimit-reset`, floored at 0: with clock skew that was a hot loop, and for a burst limit it could be most of an
  hour (the hourly window's reset). It now prefers `retry-after` (seconds or an HTTP date), falls back to
  `ratelimit-reset`, then to a minute, and is always between 1 second and 5 minutes (`minRetryAfterMs`,
  `maxRetryAfterMs`, `rateLimitWaitMs`).

* **A fleet pool says when its Nodes are not joining** (`@better-compute/kubernetes`). A substrate whose Nodes never
  ran bcd (the image does not pull, the gateway is unreachable from the cluster, a pin is wrong) stayed `ready` with
  0 joined, polled every 10 s and minted a new token every hour, with nothing to say why. `kubernetesSubstrates.get`
  now returns `waitingToJoin: { nodes, since }` while listed Nodes have no joined bcd (`since` starts over when one
  joins or more Nodes appear). After 15 minutes without a join the substrate, still `ready`, carries a `message`
  naming how many Nodes wait and what to check, `selfCheck` reports `kubernetes.substrate-nodes-not-joining`, and
  the substrate leaves the 10 s poll to the 2 minute refresh (the join token is kept for the Nodes). Migration
  `kubernetes/0007_substrate_join_wait` adds `kubernetes_pool_substrates.join_wait_since`.

* **bcd reports the Kubernetes Node it runs on** (`proto`, `@better-compute/protocol`, `bcd`,
  `@better-compute/kubernetes`). A fleet pool matches each pool node to its Kubernetes Node by the Node name bcd
  reports, but bcd never reported one: `NODE_NAME` only reached its Kubernetes manager, so matching fell back to the
  kernel hostname, which is not the Node's name on EKS AMIs with a short hostname. Such a node was never matched,
  so its Node going away was never noticed. `OsSection` gains `kubernetes_node_name` (field 9, additive): bcd's
  inventory collector fills it from `NODE_NAME` (the DaemonSet sets it from `spec.nodeName`) when the value can be
  a Node name, and reports nothing outside a pod, so other machines' `os` section hashes do not change. The
  substrate reads `os.kubernetesNodeName` first, then the hostname. The Node's uid is not reported (a pod cannot
  read it without the API); the substrate still records it from the Node list when it makes the match.

* **Provider Spot nodes are Spot to capacity** (`@better-compute/capacity`, `@better-compute/kubernetes`). A fleet
  pool's Spot flavor was marked only on the Kubernetes Node (`better-compute.dev/capacity=spot`), so a non-spot
  reservation could land on a machine the provider may take back, and a reclaim ended what the node held as
  `node-removed`. The substrate now writes the system label `bc.dev/capacity=spot` on every pool node whose Node is
  Spot, when it learns the Node (nodes learned earlier are marked on the next run). Capacity places only
  `spot: true` reservations on such a node (`capacity.preview` too; the request's `nodeLabels` carries
  `bc.dev/capacity: '!spot'`, and a `nodeLabels` value starting with `!` now means the node must not carry that
  value), and `endNodeReservations` ends whatever a Spot node held as `preempted`, whatever the reservation asked
  for. A node can still take a non-spot booking in the moment between its join and its first inventory, before it
  is matched to its Node; that booking ends as `preempted` as well.

* **A refused delete backs off** (`@better-compute/kubernetes`). A cluster whose delete the provider refuses for
  good was asked for again every 30 s forever (the fast sweep queued every row still to delete, the one in `error`
  included), and a substrate tearing down polled that cluster every 10 s. A cluster in `error` with a pending
  delete now leaves the 30 s sweep; the 5 minute refresh retries it 5, 10, 20, 40 minutes after the last try and so
  on, at most every 6 hours, counted from when it went to that error (migration `kubernetes/0006_delete_backoff`:
  `kubernetes_clusters.updated_at` and `retried_at`, `kubernetes_pool_substrates.retried_at`).
  `kubernetesClusters.delete` on such a cluster tries at once. The substrate of such a cluster goes to `error`
  (`cluster: …`, `desired_state` still `deleted`), stops polling, is looked at with the same backoff from 2 minutes,
  and is woken when the cluster is deleted, which now finishes its teardown.

* **`fleet.apply` without `prune` no longer deletes a cluster** (`@better-compute/fleet`). A document that dropped
  `policy.substrate` from a pool it kept planned as a plain `pool update (policy)`, and the substrate reconciler
  then tore the pool's cluster down. It is a delete now: with `prune` the plan carries a `delete` change on the pool
  (`fields: ['policy.substrate']`, with a note that the substrate is torn down); without `prune` the substrate
  stays in the policy that is sent, other policy changes are still made, and the pool's change notes
  `policy.substrate is kept`.

* **One cluster backs one pool, whoever reaches it** (`@better-compute/kubernetes`, `provider-kubernetes`,
  `provider-gcp`, `provider-aws`, `provider-digitalocean`). Adopting a cluster (`cluster: { adopt }`) was checked
  per provider account with an unlocked read, so two accounts on one project, or two pools at the same moment, both
  adopted it: the second added node pools, and its teardown revoked the shared `bc-substrate` identity under the
  first. Three things refuse it now. Migration `kubernetes/0005_cluster_claims` adds `kubernetes_clusters.claim_key`
  with a unique index over live rows: an adopted row claims its account and cluster id when it is inserted (the
  second pool's substrate goes to `error`: "already another pool's substrate"). The cluster reconciler then replaces
  that with the cluster's provider-global id, which the GKE, EKS and DOKS drivers now report as
  `outputs.globalId` (the selfLink, the ARN, `do:kubernetes:<uuid>`), before it makes any node pool; a second row
  for the same cluster is `CONFLICT` and its cluster goes to `error`. A cluster better-compute created records its
  global id too, so it cannot be adopted through another account. Last, the substrate claims the cluster itself:
  a ConfigMap `kube-system/better-compute-substrate` labelled with the pool id, created (never applied over) before
  anything else is listed, applied or joined, and before a newly adopted cluster gets its node pools
  (`kubernetes_clusters.claimed_at`). A cluster another pool or another deployment claimed is refused with nothing
  made in it, and tearing the refused substrate down deletes nothing there and does not revoke the identity. An
  adopted cluster's claim is deleted on teardown. `KubernetesApiClient` gains `create` (POST; false on 409), and
  `SubstrateClusterApi` an optional `create`. A third-party driver without `outputs.globalId` keeps the per-account
  check.

* **A fleet pool in error keeps tending its nodes** (`@better-compute/kubernetes`). A pool substrate whose cluster
  went to `error` (one flavor's node pool refused: quota, a stockout), or whose policy could not be converged (a
  flavor's shape changed, the account or cluster changed, the substrate no longer parsed), stopped before it looked
  at the cluster: gone or Spot-reclaimed nodes were never retired, their bookings and quota stayed held, and
  replacement Nodes got no join token. The substrate now stays in `error` with the reason as its message and still
  lists the Nodes, retires gone nodes, keeps the join token and re-applies bcd, as long as the cluster's control
  plane answered once and is not being deleted. A step that then fails is appended to the message
  (`cluster: …; credentials: …`).

* **Scripts on images without cloud-init** (`@better-compute/provider-gcp`). Compute Engine's Debian, Rocky,
  CentOS, RHEL, AlmaLinux and SUSE images do not run cloud-init, so a shell-script `userData` (`#!…`) never ran on
  them. Create now also writes it as `startup-script` metadata, wrapped to run once on the first boot as cloud-init
  would (`/var/lib/better-compute/user-data.done`). A `#cloud-config` user data, and every other image, is
  unchanged.

* **Bare metal and families that cannot live-migrate** (`@better-compute/provider-gcp`). `*-metal` machine types
  boot from Hyperdisk Balanced on an `IDPF` interface; they and the H3, H4D and X4 families are created with
  `onHostMaintenance: TERMINATE`, which Compute Engine requires of them.

* **A4X and G4 when the machine type cannot be read** (`@better-compute/provider-gcp`). The fallback that decides a
  GPU machine type (and so `TERMINATE`) when `machineTypes.get` fails now knows the `a4x-` and `g4-` families.

* **GB200 and RTX PRO 6000 GPUs** (`@better-compute/provider-gcp`). `nvidia-gb200` and `nvidia-rtx-pro-6000`
  accelerators report the `GB200` and `RTX-PRO-6000` GPU families (and meters) instead of `other`.

* **An EC2 instance another process just launched is never lost or leaked** (`@better-compute/provider-aws`).
  Whether EC2 may still not show an instance was decided from one process's memory of its own launches, so
  another replica (or the same one after a restart) read a just-launched instance as gone: `get` returned
  `undefined` (vps marked it `lost` and stopped billing it) and `delete` reported success without terminating it.
  A first miss by id now opens the consistency window for any instance: `get` answers `pending` and `delete` a
  retryable `UNAVAILABLE` (`eventual-consistency`) until EC2 shows the instance or it has been missing for the
  whole window. `TerminateInstances` not finding an instance a read just found is retried the same way. Volumes,
  networks and instances share one consistency helper.

* **A static AWS access key rotated in place is picked up without a restart** (`@better-compute/aws-core`). A
  long-term key is cached until it is invalidated, and only `ExpiredToken` invalidated it, so after an operator
  rotated the secret every call answered `AuthFailure` until the process restarted. `AuthFailure`,
  `InvalidClientTokenId` and `UnrecognizedClientException` (now `UNAUTHENTICATED`) re-read the credentials and
  retry once, and only when the source then gives a different key.

* **`RequestExpired` from expired temporary credentials recovers** (`@better-compute/aws-core`,
  `@better-compute/aws`). `RequestExpired` was handled only as clock skew: one retry with the same credentials,
  then `UNAUTHENTICATED` until the local clock reached the refresh point. When the measured skew is under a minute,
  or the clock-corrected retry is refused again, the credentials are now read again and the request retried once
  with the new ones. Each call makes at most one clock correction and one credential re-read. The platform
  transport of `@better-compute/aws` gets the same retries, including the one for a refused (rotated) key.

* **One caller giving up no longer fails the others waiting on the same AWS sign-in** (`@better-compute/aws-core`,
  `@better-compute/provider-aws`). The shared credential refresh (`cachedCredentials`) and the account identity
  check ran on the signal of the caller that started them, so when that caller aborted (a `listOffers` out of time)
  a healthy `create` or purchase waiting on the same refresh was rejected with someone else's abort. Both now run
  on their own signal and timeout (60 s; `cachedCredentials({ resolveTimeoutMs })`), each waiter leaves on its own
  signal only, and the work is cancelled once nobody waits. `aws-core` exports `sharedFlight`.

* **A capacity purchase is not held back by a failure that sent nothing** (`@better-compute/vps`,
  `@better-compute/provider-aws`). Any retryable failure marked a purchase attempt as one that may have bought,
  including failures raised before `PurchaseCapacityBlock` was sent (one throttled describe, an STS blip); every
  later attempt then answered `purchase-propagating` for 15 minutes and the block was bought late or not at all.
  provider-aws now marks retryable failures raised before the purchase request with `details.reason:
  'purchase-not-sent'` (the reason they had moves to `details.cause`), and vps leaves a purchase as uncertain as
  it was for those and for failures before the provider was asked. A failure of the purchase request itself, or
  after it, is uncertain as before: the double-buy guard is unchanged.

* **A retried EC2 create whose parameters drifted finds its instance** (`@better-compute/provider-aws`). When a
  `RunInstances` answer was lost and a parameter changed before the retry, EC2 answers
  `IdempotentParameterMismatch`; that surfaced as `CONFLICT`, which vps retried forever while the launched
  instance ran untracked. `create` now answers with the instance the key already launched (as it is: it reports
  its real zone and type), or a retryable `UNAVAILABLE` (`eventual-consistency`) until EC2 shows it.

* **EC2 resource quotas are `QUOTA_EXCEEDED`** (`@better-compute/aws-core`). `SecurityGroupLimitExceeded`,
  `CidrLimitExceeded`, `NetworkAclEntryLimitExceeded` and `TagLimitExceeded` were not in the error table, so a
  create that hit the security group quota failed as `PROVIDER_ERROR` and was retried forever.

* **Placement groups are re-made when they vanish and deleted when unused** (`@better-compute/provider-aws`). A
  placement group is trusted for an hour without a read, so a group deleted meanwhile failed every launch into it
  until the hour passed, and nothing ever deleted a group. A launch EC2 refuses with
  `InvalidPlacementGroup.Unknown` now forgets the group, ensures it again and launches once more
  (`placement-group-propagating` if EC2 still does not take it). Groups better-compute made (tagged
  `better-compute:managed=true`; never one found by name) are deleted once no instance uses them: the sweep runs
  with the account's event poll, at most every 10 minutes, and deletes a group only at the second look.
  `aws iam-policy --phases A10` adds `PlacementGroupDelete` (`ec2:DeletePlacementGroup` on tagged groups); an
  account with the older policy keeps its unused groups. Per-instance tag security groups were already swept
  (`ensureNetwork`, `ensureFirewall` and `setTags` delete tag groups unused at two looks).

* **A retried Secrets Manager `put` no longer leaves a second secret** (`@better-compute/aws`).
  `awsSecretsManagerStore().put` used a new name and a new `ClientRequestToken` on every call, so a `CreateSecret`
  whose answer was lost, followed by a retry, left a secret holding tenant plaintext that no ref named. `put` now
  takes `idempotencyKey` (`AwsSecretPutOptions`): the name and the token are derived from it and the tenant, so a
  retry with the same key answers with the same ref (the same key with another value is `CONFLICT`); a rotation
  with a key is one version however often it is retried. Without a key, a `CreateSecret` that may have created
  the secret (no answer, a 5xx) is followed by a best-effort `DeleteSecret` of that name. `SecretStore.put` in
  `core` does not carry a key yet, so callers pass it through the store's own type.

* **`@better-compute/aws` signs and redacts with `aws-core`** (`@better-compute/aws`). Its private copy of the SigV4
  signer had drifted from `aws-core`'s and is gone: requests and presigned URLs are signed by `aws-core`. Error
  messages are redacted with `aws-core`'s patterns (every access key id prefix, signed header lines, credential
  XML) plus `SecretString`, `SecretBinary` and `Plaintext`, and the transport no longer follows redirects (a
  redirect would have sent the signed headers to another host). RDS IAM tokens still sign the empty-body hash.

* **A Proxmox VE create that was never cloned** (`@better-compute/provider-proxmox`). Any PVE error containing
  "does not exist" was read as "no such VM", so an account whose `storage` or `pool` the cluster lacks had its clone
  refused and `create` still returned a `pending` machine. Only a read or a delete of a VM's own path now answers
  "missing"; a clone, a config write, a resize or a start that fails is a `PROVIDER_ERROR` carrying PVE's message, and
  a power call on a VM deleted meanwhile is still `NOT_FOUND`.

* **A Proxmox VE create that never returned** (`@better-compute/provider-proxmox`). When the key's VMID was held by
  something the VM listing does not show (an LXC container, a VM the token cannot audit), the clone was refused as
  "already exists" and retried against the same VMID until the call was aborted. Such a VMID now counts as taken and
  the create probes the next one, ending in `CAPACITY_UNAVAILABLE` when the range has no free VMID nearby.

* **Hetzner Robot calls on a server that is no longer claimed** (`@better-compute/provider-hetzner`, `vps`).
  `power`, `rescue`, `identify` and `bootstrap` never read the claim, and `get` and `delete` accepted any `bc-` name,
  so a call that landed after `delete` released a dedicated server, or after another key claimed it, hardware-reset
  it and ran installimage on it. Each of them now reads the server again and acts only while its name is the claim
  of the call's idempotency key (`robotClaimName(key)`); anything else is `NOT_FOUND` and untouched, and `bootstrap`
  checks again once rescue is up, before the join token is delivered. **Callers of the Robot provider outside vps
  must pass the key the server was claimed under on every call** (the reconciler and `providerAccounts.bootstrap`
  already do). `providerConformanceCases()` now calls `get`, `delete`, `power` and `reinstall` under the key the
  machine was created with.

* **Scripts and SSH keys on images whose reference does not say what they are** (`@better-compute/provider-gcp`).
  Whether an image runs cloud-init, and the user its SSH keys log in as, were guessed from words in the image
  reference. A Debian-based family named without the distribution (`common-cpu`, `pytorch-latest-gpu`),
  `oracle-linux-*` and custom families got no `startup-script`, so a `#!` user data never ran there; and a create by
  family could install the keys for another user than `images.resolve` reported for that family. Create now reads
  the image a family resolves to (the day's cached answer `resolve` gives; only when a key or a script needs it) and
  decides both from that image, so the login user is the one `resolve` reports. A `#!` user data is sent as
  `startup-script` to every image except Canonical's Ubuntu images, and the wrapper (`firstBootScript`) is safe on
  an image that does run cloud-init: it runs nothing where cloud-init has taken the script. `runsCloudInit(image)`
  now answers `true` only for an image known to run cloud-init (a `projects/ubuntu-os-*/global/images/…`
  reference); new `GcpImages.bootImage(image, signal)`.

* **Stopping and restarting VMs with Local SSD** (`@better-compute/provider-gcp`). `graceful-off` and
  `graceful-restart` called `instances.stop` without `discardLocalSsd`, which Compute Engine requires for a VM with
  Local SSD disks (every A3, A4 and G4), so those VMs could only be powered off hard. Every stop now sends
  `discardLocalSsd=false`, as `off` already did: the disks' data is kept.

* **AI zones** (`@better-compute/provider-gcp`). `regionOf` took only a one-letter suffix off a zone, so an AI zone
  such as `us-south1-ai1b` was its own region: its offers and machines reported the zone as their region, and a
  create with a subnet named `regions/us-south1-ai1b/subnetworks/…`, which Compute Engine refuses. The region is
  now the zone without its last segment (`us-south1`), and the provider answers `ProviderImpl.zoneRegion` with it, so
  vps places a tenant network's subnet in that region.

* **Conformance cases that checked nothing were reported as passed** (`@better-compute/core`, `vps`, `testing`). A
  case that returned early because the provider lacks a capability, or the subject names nothing to run it on, counted
  as passed: a provider offering only machines and power "passed" all of `providerConformanceCases()`, and the
  Kubernetes role and revoke cases passed without a `can` to check access with. A case now returns (or throws)
  `skip(reason)` from `@better-compute/core/conformance`, and `runConformance` reports it under the new
  `ConformanceReport.skipped` (`{ name, reason }`), never under `passed` (new exports `skip`, `ConformanceSkip`,
  `isConformanceSkip`; `ConformanceCase.run` may resolve to a skip). **A test that asserted
  `report.passed.length === cases.length` now counts fewer passes**: assert `report.failed` is empty and what
  `report.skipped` may hold. `providerConformanceCases()` skips with a reason wherever a capability or a subject
  option is absent, fails a case whose capability the definition declares but whose client lacks the method, and
  ends with a new case, "every capability the definition declares is implemented by its client".
  `kubernetesProviderConformanceCases()` skips the role, revoke, unknown-cluster and adopt cases the subject cannot
  run, and fails the revoke case when the subject sets `revoke` without `can`.

* **Kubernetes conformance left failed clusters running** (`@better-compute/testing`). The cases took a cluster's id
  only once it was ready, so a cluster that went to `error` (or never got ready) was not deleted at the end of the
  case and stayed running, and billed, on a live account. The id now comes from the first `ensureCluster` answer.

* **A stale list after "load more" during a refresh** (`@better-compute/console-kit`). `loadMore()` was documented
  as a no-op while loading, but during a refresh it aborted the refresh, dropped its result and appended a page of
  the list being replaced. It now does nothing while a refresh is in flight, and two calls share one load.

* **Events lost while a live console starts** (`@better-compute/console-kit`). `startLiveConsole` loaded every store
  and only then connected to `/watch`, so whatever happened in between was never shown. It now opens the stream
  first, loads once the stream is open, and holds events that arrive during a load until the load has finished. A
  stream that opens only after a load began, with no cursor to resume from (its first attempt failed, it took
  longer than the new `openWaitMs`, default 3 s, or it dropped before any event), makes the stores load once more.
  New options `openWaitMs` and `sleep`.

* **Aborting or timing out a workflow step cancels what it relayed** (`@better-compute/provision`, `metal`). Only
  agent steps recorded their command, so `metal.abort`, `workflows.abort` and a step timeout cancelled nothing a
  control-plane step had sent through `awaitCommand`: a `deploy.image` or `disk.wipe` still pending for a ramdisk
  that reconnected later ran anyway. Now a step that is aborted, fails or times out has every open command under
  its key (`wf:<workflow>:<step>…`) cancelled, pending ones at once and in-flight ones through the gateway, and a
  step run that outlives its step is refused any further command (`INVALID_STATE`).

* **`retries` only on retry-safe steps** (`@better-compute/provision`). A step's `retrySafe` flag was never read, so
  a profile's `retries: 2` on `deploy.deploy` or `raid.apply` sent it again under a new command key. Now
  `profiles.validate`, `profiles.render` and every workflow start report `retries` on a step that is not retry-safe
  (`steps[0].retries: deploy.deploy is not retry-safe…`), `defineStep` refuses `retries` with `retrySafe: false`,
  and the engine never retries such a step even when its row says so (a workflow started before the check).
  Driver operations written with `on:` (no registered step) keep their `retries`.

* **Commands held for a quorum approval no longer expire while they wait** (`@better-compute/provision`, `metal`).
  A destructive command of an unapproved batch kept the lifetime of an unheld one (a step's timeout, ten minutes
  for `awaitCommand`, five for `metal.power`, the request's for `site.httpProxy`), while approvers have the batch's
  24 hours: a slow approval found the command `expired` and the step failed. Now a held command's `notAfter` is its
  batch's (`heldBatchNotAfter`), and a workflow step does not time out while a command of its is held; its
  timeout counts from the release. A batch that expires still cancels its held commands.

* **A late `metal.confirmMac` no longer finds a failed node** (`@better-compute/metal`, `provision`). Confirm-first
  PXE discovery waited for the operator inside `metal.ensureRamdisk`'s 20-minute timeout: after that the node was
  `failed`, and `recover` power-cycled it into discovery again. Now, once there is a candidate (or a finding only an
  operator can settle), the step waits outside its timeout, for up to 7 days; the node keeps its state and is not
  power-cycled again, `confirmMac` wakes the workflow, and the 20 minutes count from the confirmation. New for step
  authors: `stepWaiting(until)` next to `stepNotReady(ms)`.

* **`metal.ensureRamdisk` no longer trusts a ramdisk that was just powered off** (`@better-compute/metal`). After a
  hard power-off the gateway shows the old session as connected until its ping times out (about 20 s); a workflow
  that started in that window took it for a running ramdisk and sent its next command to a machine that was off.
  Metal now records when it last cut a machine's power (migration `metal/0009_power_cut`,
  `metal_nodes.power_cut_at`, set by `power.setPower`, the step's own power cycles and `metal.power`), and a ramdisk
  session counts only if it connected since.

* **A one-time secret is no longer minted for a file that cannot be written** (`@better-compute/cli`).
  `webhooks rotate-secret --secret-out`, `webhooks create --secret-out`, `join-tokens create --out` and
  `site-tokens create --out` called the API first and opened the file afterwards: with a path that already existed
  the secret was rotated or the token minted, the command failed `ALREADY_EXISTS`, and the value was lost. They
  now reserve the file (created exclusively, mode 0600) before the call, refuse a taken or uncreatable path while
  nothing has changed (`ALREADY_EXISTS`, `FILE_NOT_WRITABLE`), and remove the file if the call fails.
  `usage export --out` reserves its file the same way.

* **An upgrade by the install script keeps the operator's top-level `config.toml` keys** (`@better-compute/cli`).
  A script from `daemon install-script` run on an installed host rewrote `/etc/bcd/config.toml` with its own keys,
  two pins and the sections of the old file; every other top-level key was dropped, so `gateway_server_name`,
  `data_dir` and `socket_path` silently went back to their defaults. The script now carries over every top-level
  key it does not set itself, as written, never a key twice. Regenerate your install scripts to get the fix.

* **`doctor` exits non-zero on an error finding** (`@better-compute/cli`). `better-compute doctor` printed its report
  and exited 0 whatever it found; only `--strict` failed, and then on warnings as much as on errors. A deploy step
  running `doctor` passed over an unmigrated or unreachable database. Now any `error` finding exits 1 with
  `DOCTOR_FINDINGS` (the report is still printed first), `--strict` adds warnings, and `info` findings never fail.
  Scripts that ran plain `doctor` on a deployment with error findings and expected status 0 need to handle it.

* **Unknown configuration keys are refused** (`@better-compute/cli`). The `host` export took any key: a typo such
  as `host.gatway` was dropped and `serve` started with no gateway role. The `cli` export likewise ignored a
  mistyped `defaults`, command name or flag. Both now answer `INVALID_CONFIG` with the nearest known name
  (`host.gatway is not a setting of host. Did you mean host.gateway?`), at every level of `host` except the objects
  passed through as they are (`gateway.tls`, `gateway.options`, `http.wellKnown`). A configuration that carried an
  unused key must drop it.

* **BMC credential rotation no longer deletes the enrollment secret** (`@better-compute/metal`).
  `metal.rotateBmcCredential` purged every superseded credential from the secret store, including the one the
  machine was enrolled with: two machines enrolled with one rack-wide `secret://` broke each other on the first
  rotation. Now only secrets rotation itself made are purged, and only when no open command and no other machine
  names them; the enrollment secret is the operator's to delete.

## 0.3.0 (2026-10-20) [#030-2026-10-20]

### Added [#added-1]

* **Platform compute billing** (`@better-compute/vps`, `capacity`, `iam`, `server`, `provider-gcp`; plan
  `docs/superpowers/plans/2026-10-18-platform-compute-billing.md`). The root tenant can bill its sub-tenants for the
  compute it issues to them. Compute meters and Better IAM rates and invoices, as before; what is new is what compute
  meters. See the docs guide "Billing and usage".
  * **Usage from typed sources.** `usage_windows` gains `source_kind` (`allocation`, `instance`, `grant`,
    `network`, `snapshot`, `cost`) and `source_id`; new sources are keyed `cmp:<kind>:<id>:<windowStartMs>:<meter>`,
    pass-through costs `costx:<accountId>:<rowId>`. Allocation windows keep their keys, rows and amounts. The one
    `iam.usage` flush records every source.
  * **Provider VMs are metered.** Created and attached cloud VMs bill from lifecycle intervals written with each
    observed state change (`instance_usage_intervals`; the `vps.provider-usage` reconciler, every 5 minutes, writes
    each hour `billing.meterGraceMs`, default 5 minutes, after it ends): `vm.vcpu.hours`, `vm.memory.gib.hours`,
    `vm.gpu.<model>[.spot].hours` and `vm.instance.hours`; `vm.reserved.*` for a VM admitted against a provider
    reservation; or one `flavor.<name>.hours` line for a flavor whose root-set `billing.meter` prices the whole
    shape. A stopped VM bills its disks only; a `lost` span bills nothing.
  * **Billing modes.** `PoolPolicy.billing.mode` (`metered`, `platform-fee`, `none`; root only) decides what a
    pool's VMs record. A provider pool on a tenant's own cloud account defaults to `platform-fee`
    (`vm.managed.hours` only), so a tenant is never charged compute it pays its provider for. `instances.attach`
    takes `billing: 'metered' | 'none'` (`none` is root only) and bills from the first observed `running` after the
    attach.
  * **Reservation grants** (new `reservationGrants` group: `create`, `get`, `list`, `end`). Root commits slots of a
    provider reservation to a tenant for a term. The `vps.grants` reconciler bills `vm.grant.slot.hours` for every
    committed slot, used or not; VMs inside the grant record only the informational `vm.grant.usage.hours`, never
    compute meters. Admission serves the grantee from its slots first and keeps them from everyone else. A grant
    that starts, or renews another, takes in the tenant's VMs already running on the reservation. Events
    `reservation-grant.*` (`created`, `active`, `ended`, `cancelled`). When the provider stops reporting the
    reservation, its active grants end and its scheduled ones are cancelled then (`reason: 'reservation-gone'` on
    the events), and its VMs bill on-demand from then on. A reservation whose `total` drops below its live grants'
    slots publishes the new event `reservation-grant.capacity-short` `{ accountId, reservationId, total,
    grantedSlots, grantIds }` once per change; the grants keep billing.
  * **Ancillary meters.** The `vps.ancillary-usage` reconciler bills `network.vpc.hours` per provisioned tenant
    network and `snapshot.gib.hours` per sandbox snapshot; a VM's intervals add `ip.public.hours` while it runs
    with a public address and `disk.<type>.gib.hours` per disk, running or stopped (`DiskTypeUnknown` for a type
    with no meter). provider-gcp reports each VM's disks and their types.
  * **The `billing` API group** (root only: `compute:billing:read`/`manage` on `compute-fleet`). `meters` and
    `syncMeters` (the catalog on the IAM root tenant), `prices` and `setPrices` (list and negotiated prices,
    `targetTenantId`), `proposePrices` (provider list prices plus a margin, never applied) and `margins` (Better
    IAM revenue minus provider cost by tenant, pool, account or meter, platform accounts only).
  * **Provider costs.** The `vps.costs` worker (`vps({ costs })`: `every`, default `6h`; `lookbackDays`, default 3;
    `leakToleranceHours`; `passBudgetMs`) reads the accounts whose provider implements `costs` into
    `provider_costs`, attributes each row by its `better-compute-instance` and `better-compute-tenant` labels (or
    to the tenant of a BYO account), and publishes `billing.cost-ingested`, `billing.unattributed` and
    `billing.cost-corrected`. A row in another currency is stored with `CurrencyMismatch` and left out.
  * **Pass-through.** `providerAccounts.create`/`update` take `billing.passThrough` (root accounts only): rules of
    `{ skuPattern, meter, multiplier }` bill costs compute does not meter, such as egress, on a `reported` meter at
    net cost times the multiplier (1 to 10), once per row. A pattern that matches a SKU compute meters itself is
    `INVALID_CONFIG` (`meteredSku`), and such rows never pass through. Whether a row may pass is decided when it
    is ingested (`provider_costs.passable`, with a partial index) and again for an account's rows within the
    lookback when its rules change; the pass reads only those rows, in batches within the run's budget. A row whose
    usage month has already closed (`vps({ billing: { closeGraceHours } })`, default 6, after the month ends) is a
    late charge in the first hour of the current month, with the same key; `usage.deadLetters` items carry the
    `cost` of a pass-through window.
  * **Leak detection.** After each pass over a platform account, an instance whose provider-billed compute hour
    slots exceed its metered running hours in a UTC day by more than `leakToleranceHours` (default 1) raises
    `billing.leak`.
  * **Estimates.** `flavors.list` items on provider pools carry `estimate` (`{ perHourMicros, currency, lines }`,
    priced with the tenant's negotiated prices through `IamPort.quote`), and the new `instances.estimate` answers it
    for one flavor before a create (a read of the pool's flavors: no draft, quota or audit row).
  * **Root usage views and corrections** (`compute:usage:read`/`manage` on `compute-fleet`, root only).
    `usage.platform` totals every tenant's windows by tenant, pool, meter, source and provider (JSON or CSV).
    `usage.deadLetters` lists what Better IAM refused, with a `reason` (`meter-missing`, `period-closed`,
    `identity`, `invalid`, `other`); `usage.retry` sends windows again (`period-closed` is refused: it needs a
    root invoice item); `usage.void` drops unsent or dead-lettered windows with a reason (a recorded window needs a
    credit note in Better IAM). `usage.list` takes `includeVoided`.
  * **Contract** (`@better-compute/core`, additive). `IamPort.ensureMeters?`, `listMeters?`, `setPrice?`,
    `listPrices?`, `quote?` and `spend?` (embedded with `bridge.embedded(iam, { serviceCredential })`, remote over
    HTTP); `UsageEvent.cost?`; `ProviderImpl.costs?` and `listPrices?` with `ProviderCostRecord` and
    `ProviderListPrice`; `ProviderDefinition.billing?` (`computeSku`, `meteredSku`); `ExternalMachine.disks` and
    `bootDiskType`; `PoolPolicy.billing?`, `Flavor.billing?`, `Offering.allowedRegions?` and `suspendedAt?`;
    `canonicalGpuModel`, `meterCatalog` and the meter helpers in `core/meters.ts`. The fleet kind gains the actions
    `billing:read`, `billing:manage`, `usage:read` and `usage:manage`.
  * **provider-gcp.** `billingExport` (`project.dataset.table` of the Cloud Billing detailed BigQuery export) and
    `billingExportLocation` on the account config; `costs` reads the export with parameterized queries;
    `listPrices` reads the Cloud Billing Catalog. Turning the export on is a manual Cloud Console step; the package
    README lists the IAM roles and the Cloud Billing API it needs.
  * **Migrations:** core `0029_usage_sources`, `0030_offer_suspension`, `0031_usage_window_claims`; vps
    `0030_instance_usage`, `0031_reservation_grants`, `0032_network_metering`, `0033_provider_costs`. All are
    additive and rewrite no `usage_windows` partition.

* **Offer budget policies over the API** (platform compute billing, D16). `pools.offer` takes `onBudgetExceeded`
  (`notify`, `stop-new`, `drain` and the new `suspend`) and `allowedRegions`; the new `pools.updateOffer` changes
  them, suspends an offer by hand and lifts a suspension. A spent budget on a `suspend` offer sets
  `pool_offers.suspended_at` (migration `core/0030_offer_suspension`), publishes `quota.budget-suspended`, stops the
  tenant's running instances in the pool (`BudgetSuspended`; the `vps.suspension` reconciler does it within a minute
  for every suspended offer, a hand suspension included, and `sandboxIamReactions` in `iamReactions` makes it
  immediate), drains the offer's sandbox warm pools and refuses all its new work with `SPEND_LIMIT_REACHED`
  (`details.reason: 'offer-suspended'`) until lifted, which restarts nothing. A budget check for a caller from
  another tenant (root issuing into a customer) names the target tenant only. `instances.attach` (when billed) and
  `networks.create` (on a metered pool) now run the same budget check as `instances.create`; reservation grants are
  not budget-checked.

* **`@better-compute/gcp-common`** (new, outside the umbrella). The Google Cloud token sources
  (`serviceAccountTokenSource`, `metadataTokenSource`, `impersonatedTokenSource`, `cachedTokenSource`) move out of
  `@better-compute/provider-gcp` into a package that depends only on `@better-compute/core`, so other Google
  providers can share them. `provider-gcp` re-exports the same functions; its API and behavior are unchanged.

* **Universe domains** (`@better-compute/gcp-common`, `@better-compute/provider-gcp`). `googleEndpoint(service, { universeDomain })`
  and `tokenEndpoint(universeDomain)` build Google endpoints for a host-configured universe (default `googleapis.com`).
  `serviceAccountTokenSource` and `impersonatedTokenSource` take `universeDomain`: another universe pins a key's
  `token_uri` to exactly `https://oauth2.<universe>/token`, and a key's own `universe_domain` must match it. The default
  universe is unchanged. `gcpProvider({ universeDomain })` derives the compute endpoint from it.

* **`providerAccounts.federationSetup`** (`@better-compute/vps`). A read method (`compute:providers:read`, no provider
  call) that returns the workload `issuer`, its `jwksUri`, the exact `subject` `providerAccounts.create` will give an
  account (`compute-account:<tenantId>/<provider>/<name>`), the CEL `attributeCondition` that trusts that subject
  alone, and an `allowedAudienceHint`. It is the first step of keyless Google Cloud accounts (`federationAudience`);
  the providers guide has the `gcloud` setup and what each `create` refusal means.

* **Keyless Google Cloud accounts** (workload identity federation, GCP WS1).
  * `@better-compute/pki` `workloadIssuer({ issuer, signingKey, nextKey?, previousKeys? })`: the control plane as an
    OIDC issuer (ES256, 60 to 600 s tokens, `nextKey` published before it signs). `pki.workloadIssuer` in
    `betterCompute()` serves it, unauthenticated and cached 5 minutes, at
    `GET {basePath}/workload/.well-known/openid-configuration` and `GET {basePath}/workload/jwks.json`, names it in
    `/.well-known/better-compute` as `workload`, and hands it to modules as `context.pki.workload`. The issuer must be
    `<baseURL's origin>{basePath}/workload`, or on the origin named by the new `pki.workloadIssuerOrigin`, and differ
    from `workloadIdentity.issuer`'s; anything else is `INVALID_CONFIG` (`reason: 'workloadIssuerUrl'`). The PKI
    operations page covers persistent keys and rollover.
  * `@better-compute/provider-gcp`: a tenant account may name `federationAudience` (its workload identity pool
    provider) instead of `credentials`. `providerAccounts.create`/`update` prove before storing it that the pool trusts
    exactly the account's subject and that the token reads the project, and refuse it with `INVALID_CONFIG` reasons
    `credentialsAndFederation`, `noWorkloadIssuer`, `federationAudience`, `federationTrustsAnySubject`,
    `federationRefused`, `impersonationRefused` or `projectAccess`; Google failures are retryable `UNAVAILABLE` or
    `RATE_LIMITED`. An audience already in use by any account is `CONFLICT`.
  * `@better-compute/gcp-common`: `federatedTokenSource` and `stsExchange` (Google STS token exchange).
  * `@better-compute/vps`: migration `vps/0027_account_exclusive_key` (a provider-declared key, the federation
    audience, unique per provider across tenants); provider checks run in admission with no transaction open.
  * `@better-compute/core` (additive, optional): `WorkloadIssuerPort`, `PkiPort.workload`, `ProviderContext.identity`,
    `ProviderInput.checkAccount` and `ProviderInput.accountKey`.

* **Pool membership of discovered provider machines** (`@better-compute/vps`). `pools.bindMachines` and
  `pools.unbindMachines` (`compute:pools:manage` on the pool; only the platform or the provider account's own tenant)
  bind up to 100 machines the `vps.discovery` worker saw to a `provider` pool, row-locked and idempotent, and return
  `{ bound | unbound, skipped: [{ externalId, reason }] }` with reasons `held`, `gone`, `deleting`, `other-region`
  (Compute Engine accounts, against the region of the pool's site), `bound-elsewhere` and `not-found`; they publish
  `pool.machines.bound` and `pool.machines.unbound`. `pools.machines` (`compute:pools:read`) lists a pool's bound
  machines (`filter`, `limit`, `cursor`) to its owner and the tenants it is offered to, without other tenants'
  instances and, for offered tenants, without the account id. `instances.attach` attaches a machine bound to another
  pool as `CONFLICT`, refuses a Compute Engine machine outside the pool's region (`INVALID_INPUT`), and lets a tenant
  the pool is offered to attach a machine bound to it; detach and delete keep the binding, so the machine returns to its
  pool. `providerAccounts.machines` shows `poolId`. Migration `vps/0042_pool_machines`.

* **Instance labels** (`@better-compute/vps`, `@better-compute/core`). `instances.create` takes `labels`
  (Compute Engine label rules, at most 62, keys starting with `better-compute-` or `bc-` reserved), stores them
  (`vps/0043_instance_labels`), shows them on `Instance.labels` (optional, additive) and passes them to
  `ProviderImpl.create`; the provider's own ownership labels still win.

* **`networks.adopt`** (`@better-compute/vps`). The platform (`compute:providers:manage` on the fleet) registers a
  VPC that already exists in a provider pool's account as a tenant network, so attached VMs whose NIC is on it are
  linked to it. better-compute never creates, changes or deletes the VPC; its firewall rules must name
  `targets.instanceIds` and always compile to instance network tags. `default` needs `sharedTagScoped: true`, which
  the network records. `providerResourceName(id, externalName?)` prefers the VPC's own name; migration
  `vps/0044_network_adopt` (`networks.external_name`, `networks.shared_tag_scoped`); event `network.adopted`.

* **Image safety and resolution** (U-03; `@better-compute/provider-gcp`, `@better-compute/vps`, `@better-compute/core`).
  * **Image allowlist.** A Compute Engine account allows images only from `imageProjects` (new; default
    `defaultImageProjects`, the public OS image projects including `ubuntu-os-accelerator-images` and
    `deeplearning-platform-release`), its `imageProject`, and its own project unless `ownProjectImages` (new) is
    `false`. `create` refuses any other project's image, and any malformed image reference, with `INVALID_INPUT`
    (`details.reason` `imageProjectNotAllowed` with `details.allowed`, or `imageRef`) before it reads or creates
    anything. A compute image URL must name the provider's own universe host.
  * **`images.resolve`** (new; `compute:flavors:read` on the pool, served by vps): `{ tenantId, poolId, families }`
    (1 to 50) returns `{ poolId, provider, items, missing }`. Each item is a family's newest image that is not deprecated,
    from `images.getFromFamily`, with `image` (the exact `projects/<p>/global/images/<name>` to pass as `imageId`),
    `architecture`, `minimumDiskGiB`, `guestOsFeatures`, `loginUser`, a scheduled `deprecation`, `createdAt` and
    `description`; `missing` names families with no image (`notFound`) or a deprecated one (`deprecated`). The
    provider caches each answer a day per account (a missing family an hour) and hides an image whose deprecation time
    passes while cached.
  * `@better-compute/core` (additive, optional): `ProviderImpl.images` (`ProviderImages.resolve`), `ProviderImage`,
    `ProviderImageMiss`. provider-gcp exports `GcpImages`, `sourceImageOf`, `parseImageRef`, `allowedImageProjects`,
    `defaultImageProjects`; `sshUserOf` moved to its images module and is still exported. The GCE test emulator
    serves images and families.

* **Instance identity and listing (U-04)** (`@better-compute/vps`, `core`, `server`, `provider-gcp`).
  * **Name, metadata and labels.** `instances.create`, `instances.attach` and `sandboxes.create` take `name` (a
    display name, 1 to 128 characters), `metadata` (up to 64 string entries, 16 KiB, kept by better-compute only)
    and `labels` (Compute Engine rules); hosted instances and sandboxes now take labels too. Keys starting with
    `better-compute-` or `bc-` are refused in labels and metadata. `Instance` and `Sandbox` show them, and
    `Instance.createdAt` (epoch ms). Migration `vps/0046_instance_identity`.
  * **`instances.update`** (`compute:instances:update&#x60;) and &#x2A;*`sandboxes.update`** (`compute:sandboxes:update`)
    rename and replace metadata and labels (`null` or `{}` clears; `expectedVersion` for instances), publishing
    `instance.updated` / `sandbox.updated` with `changed`. In provider pools the reconciler writes new labels on the
    VM and its boot disk through the new optional `ProviderImpl.setLabels(id, { labels, previous }, context,
    instanceId?)`; a refused write leaves a `LabelsNotApplied` condition and is retried.
  * **`issuedBy`**, the platform's provenance: `instances.create`, `instances.attach` and `instances.update` take
    it from root callers only (`ACCESS_DENIED`, `details.reason: 'issued-by-platform-only'` otherwise).
  * **`instances.list`** leaves deleted instances out in SQL, before the per-item checks (`state: 'deleted'` with
    `scope: 'mine'` lists them; with `scope: 'all'` it is `INVALID_INPUT`), and filters by `projectId` and
    `labelSelector` (`key=value`, `key!=value`, `key`, `!key`, `key in (a,b)`, `key notin (a,b)`) besides
    `state` and `poolId`, with its cursor. `sandboxes.list` takes `labelSelector` too.
  * **`instances.issued`** (root only: `compute:instances:read` on `compute-fleet`, a new fleet action): the
    instances on the root tenant's pools in other tenants, newest first and paged, filtered by `tenantId`,
    `poolId`, `state`, `issuedBy` and `labelSelector`.
  * **`system.workQueue`** (`compute:fleet:admin`): the work queue's backlog per kind (queued, due, overdue,
    claimed, retrying, `nextDueAt`) and the live workers. `workQueueStats` is exported by `@better-compute/server`.
  * **Disks.** `Instance.disks` shows the disks the provider last reported for a provider machine
    (`ExternalMachine.disks`: type, size, boot), what `disk.<type>.gib.hours` meters. provider-gcp exports its disk
    helpers (`bootDiskTypes`, `defaultBootDiskType`, `hyperdiskBootType`, `diskSizeRange`, `needsGvnic`,
    `isDiskTypeRefusal`) and the label helpers (`machineLabels`, `tenantLabels`, `changedLabels`,
    `isReservedLabelKey`, `labelBootDisk`, `diskPath`).
  * **provider-gcp labels.** A created VM's boot disk carries the VM's labels. Tenant label keys that are, or become
    once lower-cased, `better-compute-*` or `bc-*` are dropped, and the ownership labels are merged last.
    `GcpCompute.setLabels` changes a VM's and its boot disk's tenant labels under their label fingerprints, keeping
    every other label, and refuses a VM labelled for another instance (`INVALID_STATE`, `reason: 'notOwner'`).

### Changed [#changed]

* **A provider create refused as `INVALID_INPUT` fails the instance** (`@better-compute/vps`). Like a capacity or
  quota refusal, it moves the instance to `error` with a `ProvisionFailed` condition (reason `INVALID_INPUT`) and
  publishes `instance.failed`, instead of being retried every 30 seconds: the same request would be refused again.
* **One deployment currency: `vps({ billing: { currency } })`** (ISO 4217, default `USD`). Price proposals, tenant
  estimates, provider cost ingestion and margins all use it; name Better IAM's deployment currency there. It is the
  only currency option.
* **`Offering.onBudgetExceeded` is `'notify' | 'stop-new' | 'drain' | 'suspend'`** (`@better-compute/core`). The
  union is wider, so a consumer with an exhaustive `switch` over it needs a `suspend` case. The database `CHECK`
  accepts `suspend`.
* **Estimates are a separate read, `instances.estimate`.** The planned `instances.create({ dryRun: true })` was not
  built: `create`'s input and return type are unchanged, and its catalog row has no `flavors:read` alternative.
* **`reservationGrants.list` is a plain tenant read** (`compute:quotas:read` on the tenant; the SQL filters to the
  tenant's grants), like `webhooks.list`, not a per-item `list` recipe. Root lists every tenant's with `all: true`.
* **Provider accounts with live grants.** `providerAccounts.delete` and `unbindReservation` refuse with
  `INVALID_STATE` (`details.grantIds`) while a grant on the account is scheduled or active, and `delete` also until
  every committed slot-hour of its ended grants is metered. Deleting a tenant ends its active grants and cancels its
  scheduled ones.
* **Usage CSV and voided windows.** `usage.export` (and `usage.platform`'s CSV) prefix a cell a spreadsheet would
  read as a formula (`=`, `+`, `-`, `@`, a tab or a carriage return) with `'`. Voided windows are left out of
  `usage.list` and of `usage.export` rows and totals (`voided` counts them); tenants never see root's void reason.
* **`UsageWindow.allocationId` is optional** (`usage.list`/`export`): windows of other sources have none.
* **The usage flush claims windows.** `iam.usage` marks its batch `claimed_at` (migration
  `core/0031_usage_window_claims`) before sending it; `usage.void` refuses a window claimed in the last 10 minutes
  ("being sent; retry shortly"), and a dead sender's claims are taken over after 10 minutes (a resend is a
  `duplicate`). A `BILLING_PERIOD_CLOSED` refusal is dead-lettered once and never retried automatically. A send
  that throws keeps its claims (Better IAM may have recorded the batch), so its windows are sent again once the
  claims expire, and a pass only clears its own claim.
* **`iam.jobs.close-period` waits for the month's last windows** (`runIamJobs`). It runs hourly instead of daily
  and calls `billing.closePeriod()` only once the current UTC month is `closeGraceHours` old (an
  `iamJobReconcilers`/`runIamJobs` option, default 6, from 0 to 72); set `vps({ billing: { closeGraceHours } })`
  to the same value.
* **`iam setup --meters` and the iam module create the full `meterCatalog`.** `computeMeters` stays exported but is
  no longer what setup creates. The iam module also syncs the catalog when it starts if its port holds a root
  billing credential (`serviceCredential`); the doctor warns `iam.billing-port` when it does not.
* **Upgrading.** `core/0029_usage_sources` builds a unique index over every `usage_windows` partition, which locks
  `usage_windows` (reads included) while it builds: let retention prune old windows before upgrading a large
  deployment. Its two `CHECK`s are added `NOT VALID`, so no partition is scanned.
* **Releases publish an explicit allowlist** (`scripts/release-publish.mjs`, [RELEASING.md](RELEASING.md)). CI
  and manual releases publish only the packages on the release allowlist, in dependency order, never
  `pnpm publish -r`: every package of 0.2.1 plus `@better-compute/gcp-common`, which provider-gcp depends on.
  `@better-compute/aws-core` and `@better-compute/provider-digitalocean` are not released yet. A new package must
  be put on the allowlist or in `notReleased` before anything publishes. `--dry-run` runs everything but the
  upload.
* **The API references name their version and read the full registry.** `docs/api-reference.md` (`pnpm docs:api`)
  is generated from a control plane with every module registered, as the docs site's reference is, so methods a
  module serves (`instances.create`, `networks.*`, …) are no longer listed as planned; both references name
  `@better-compute/server`'s package version, and the packages page shows each package's version and whether a
  release publishes it. `pnpm lint:repo` checks `docs/api-reference.md` and the release allowlist.

### Fixed [#fixed-1]

* **Hosted volume, spot GPU and `gpu.other.hours` windows were dead-lettered.** `computeMeters` never created
  `vps.volume.gib.hours`, the `.spot` meters or `gpu.other.hours`, so Better IAM refused their usage with
  `NOT_FOUND`. The catalog now holds them: run `billing.syncMeters` (or `iam setup --meters`) after upgrading, then
  `usage.retry({ reason: 'meter-missing' })`.

## 0.2.1 (2026-10-18) [#021-2026-10-18]

Compute Engine creates that 0.2.0 sent with a boot disk or network interface the machine type refuses, and SSH keys
on images without cloud-init. Every package is at 0.2.1.

### Fixed [#fixed-2]

* **provider-gcp: newer machine types failed to create.** 0.2.0 always gave the boot disk `pd-balanced`, which the
  Hyperdisk-only families refuse (`pd-balanced disk type cannot be used by a3-ultragpu-8g machine type`). The boot
  disk type now follows the machine family: Hyperdisk Balanced for A3 Ultra, A4, A4X, C4, C4A, C4D, G4, H4D, M4, N4,
  N4A, N4D and X4, `pd-balanced` for the rest as before. Create reads the zone's disk type first: a zone that does
  not offer it, or a `diskGiB` outside the sizes it takes, is `INVALID_INPUT` (`details.reason` `diskTypeUnavailable`
  or `diskSizeOutOfRange`) before anything is created. A family GCE refuses Persistent Disk for although the table
  allows it is retried once with Hyperdisk Balanced under its own requestId, logged as `gcp.disk_type_fallback`, and
  remembered.
* **provider-gcp: gVNIC-only machine types.** A3, A4, A4X, C3, C3D, C4, C4A, C4D, G4, H3, H4D, M4, N4, N4A, N4D, X4
  and Z3 VMs get `nicType: GVNIC`; they do not take VirtIO-Net.
* **SSH keys on images without cloud-init.** Instance SSH keys only reached the VM through cloud-init user data,
  which Compute Engine's Debian and Rocky Linux images do not run. `ProviderImpl.create` takes `sshKeys` (vps passes
  the instance's), and provider-gcp also writes them as `ssh-keys` metadata for the image's default user (`debian`,
  `rocky`, `centos`, else `ubuntu`), which the guest agent of every Compute Engine image installs.

## 0.2.0 (2026-10-18) [#020-2026-10-18]

A Google Compute Engine provider (`@better-compute/provider-gcp`, new), discovery and attach of existing provider
machines, provider capacity reservations, tenant networks and firewall rules, `instances.reboot`, platform-issued
compute (`instances.create` with `ownerIdentityId`) with owner membership, and security fixes for provider accounts
and their secrets that apply to every 0.1.x deployment. Every package, `@better-compute/provider-gcp` included, is at
0.2.0.

### Security [#security-1]

* **A tenant could build a provider pool on another tenant's or the platform's provider account (0.1.x).**
  `pools.create` is tenant-callable (`compute:pools:create` on the tenant) and checked that the site was the
  caller's, but not who owns `providerAccountId`. A tenant that learned an account id (an offered pool's
  `pools.get` showed it) could create a pool on it, set its own `ownerQuota` and flavors, and have vps create
  machines with that account's credentials in its owner's cloud project: no offer, quota or flavor of the owner
  applied. This affects every provider, not only the new Google Compute Engine one. Now `pools.create` refuses an
  account that is not the pool owner's with `NOT_FOUND`, worded as for an unknown account; vps uses a pool's account
  only when the pool's owner owns it (instance creates and the provider reconciler, flavors, networks, attach,
  reservation bindings); and `pools.get`/`list` no longer show `providerAccountId` to tenants a pool is offered to.

  **Upgrading from 0.1.x.** Find pools that point at another tenant's account:

  ```sql
  SELECT p.id AS pool_id, p.owner_tenant_id, a.id AS account_id, a.tenant_id AS account_tenant_id
    FROM pools p JOIN provider_accounts a ON a.id = p.provider_account_id
   WHERE a.tenant_id <> p.owner_tenant_id;
  ```

  Instances of those pools stop reconciling after the upgrade: the provider reconciler never uses the account, so
  they are neither created, polled nor powered (it retries with an `INVALID_STATE` "has no provider account").
  To clean up, delete each instance (`instances.delete`, or `instances.detach` for an attached one): deletion now
  finishes without a provider call and refunds the tenant's quota. Then archive the pools (`pools.archive`, by their
  owner or a root caller). Finally, check the account owner's cloud project for the machines those instances
  created and remove them there: they are named after the instance id (Hetzner Cloud: the id as host name, with
  `-` for `_`; Google Compute Engine: `bc-` and the same, labelled `better-compute-instance=<instance id>`).
* **Provider-account secret references were not scoped to the account's tenant (0.1.x, every provider).** A
  tenant's account could name `env://` (the control plane's own environment), any `vault://` path, or the
  platform's or another tenant's `secret://` references, and its client resolved them. Now an account of a tenant
  other than the root may name only `secret://<its tenant>/…` and the `vault://` paths `vps({ tenantVaultPaths })`
  allows it: `providerAccounts.create` and `update` refuse anything else with `INVALID_CONFIG`
  (`details.reason: 'secretRefScope'`), the client refuses such a reference again when it resolves it
  (`ACCESS_DENIED`, for rows stored before the upgrade or around the API), and a tenant's references are resolved
  with its tenant as the context, so the sealed store refuses a row of another tenant. Platform accounts are
  unchanged. Check existing tenant accounts before upgrading: one that names such a reference stops working.
* **The Google Cloud key's `token_uri` is pinned.** The key JSON of a tenant's account is the tenant's, and the
  control plane POSTs the signed JWT to its `token_uri`. It must be absent, `https://oauth2.googleapis.com/token`
  or `https://accounts.google.com/o/oauth2/token` (exact match); anything else is `INVALID_CONFIG`
  (`details.reason: 'tokenUri'`) before anything is fetched. (`provider-gcp` is new in this release and never
  shipped without it.)
* **Tenant Google Cloud accounts cannot use the platform's ambient credentials.** Only an account of the root tenant
  may leave out `credentials` and run on the host's identity (the host `tokenSource` or the metadata server); a
  tenant's account without its own key, or with `impersonate` but no key, is `INVALID_CONFIG` at create and update
  and on every use. (Also new with `provider-gcp`.)

### Added [#added-2]

* **Google Compute Engine provider** (`@better-compute/provider-gcp`, new, outside the umbrella). Compute Engine VMs
  as VPS instances and offers, GPU machine types included, over plain `fetch` against the REST v1 API. Accounts take
  `{ projectId, zones, credentials?, impersonate?, … }`; credentials are a service-account key reference, or for
  root-tenant (platform) accounts only, the host's own `tokenSource` or the metadata server (see Security). Create
  is idempotent through GCE's `requestId`, and the VMs it owns carry `better-compute-managed` and
  `better-compute-instance` labels; it never deletes or powers a VM without them. It implements discovery,
  ownership labels, reservations and networks.
* **DigitalOcean provider** (`@better-compute/provider-digitalocean`, new, outside the umbrella). Droplets, GPU
  Droplets included, as VPS instances and offers over plain `fetch` against the v2 API. Accounts take
  `{ token, regions, vpc?, sshKeyFingerprint?, … }` with a personal access token as a secret reference (custom scopes
  are listed in the package README). DigitalOcean has no idempotency key, so a create carries a hashed
  `better-compute-key` tag, looks for it before and after the POST, and converges racing creates on the oldest Droplet.
  Droplets it owns carry `better-compute-managed`, `better-compute-instance` and `better-compute-deployment` tags;
  `delete`, `reinstall` and the duplicate reaper act only on managed Droplets of their own deployment, `power` needs
  an instance tag, and the tag-scoped bulk endpoints are never called. A create always sends an SSH key.
  It implements discovery and attach (autoscale-pool members are listed and never claimed), and tenant networks: a VPC
  per subnet, a base firewall that denies ingress and allows egress, and a Cloud Firewall per rule. DigitalOcean has no
  firewall priorities, so an allow that outranks an overlapping deny is refused (`UNSUPPORTED`, `firewallPriority`)
  while the deny always applies. Requests stay under DigitalOcean's per-token limits, and discovery defers before
  writes do. Reservations, Spot capacity, volumes and snapshots are not part of this release; the opt-in live suite
  (`BC_DO_LIVE`) has not been run yet.
* **Discovery and attach** (`@better-compute/vps`). The `vps.discovery` worker lists every machine of accounts whose
  provider can `discover` (`vps({ discovery: { every } })`, default 5 minutes) and publishes `provider.machine.*`.
  `providerAccounts.machines` reads what it saw (another tenant's instance on a tenant's own account is shown
  without its id and tenant, except to the platform) and `providerAccounts.discover` runs a pass now (`wait` holds
  the answer for at most 30 s). `instances.attach` turns a running VM into an instance of a customer tenant without
  recreating it, with `onDelete: 'detach'` (default) or `'delete'`; `instances.detach` releases one, and
  `instances.reboot` restarts a provider instance (graceful, or `hard`). Ownership is re-checked on every read: a
  VM whose label is lost or foreign fails its instance (`OwnershipLost`, `AttachConflict`) and is never acted on
  again. Opt-in label-driven auto-attach (`discovery.autoAttach`): on a tenant's own account the
  `better-compute-owner` label is ignored (`AUTO_ATTACH_OWNER_IGNORED`, the configured owner gets the instance); on a
  platform account a label owner must be a member of the labelled tenant (`AUTO_ATTACH_OWNER_REFUSED` otherwise).
* **Provider reservations.** The `vps.reservations` worker detects provider capacity reservations, on-demand and
  future (`vps({ reservations: { every } })`, default 5 minutes), and publishes `provider.reservation.*`.
  `providerAccounts.reservations` lists them with `free` and `pending`; `bindReservation` turns a `ready` one into
  a flavor of a provider pool, and `unbindReservation` removes it. `instances.create` on a bound flavor takes a free
  slot under a database lock, so concurrent creates never over-commit it (`CAPACITY_UNAVAILABLE` otherwise).
* **Tenant networks.** The `networks` and `firewallRules` groups let a tenant build VPCs, subnets and firewall rules
  on a provider account that supports them; the `vps.networks` reconciler converges the provider and publishes
  `network.*`. `instances.create` takes `networkId` and `subnetId` (the network must have been provisioned, the
  subnet `ready`; `INVALID_STATE` until then), rules target instances by a per-instance network tag, and attached VMs
  are linked to the tenant network their NIC is on. Network names are unique among live networks. Every tenant's
  networks on one account share its provider project, so a tenant has at most 10 live networks, and a network at
  most 16 subnets and 100 firewall rules (`QUOTA_EXCEEDED` with `details.limit`).
* **Platform-issued compute.** The root (platform) tenant adds a provider account, offers a provider pool on it to
  customer tenants, and issues machines to them: `instances.attach` of a running VM, or `instances.create` with
  `ownerIdentityId` (new) for a new one. Only a root caller acting in another tenant names an owner other than
  itself, on `create` and `attach` alike (`ACCESS_DENIED` otherwise, before any decision), so the instance is the
  customer's: in its owner's `scope: 'mine'` list, under owner-conditioned policies, on the customer's quota. See
  the providers guide, "Platform-issued compute", for what each side can and cannot do.
* **Owner membership** (`@better-compute/core`, additive). Optional `IamPort.isMember(tenantId, identityId)`; the
  embedded better-iam bridge implements it from IAM's store (an active identity of that tenant). `instances.create`
  and `attach` ask it for an owner other than the caller, outside any transaction, and refuse a non-member with
  `INVALID_INPUT` (a refused create leaves no draft). Ports without it, the remote bridge included, check the
  identity's shape only.
* **Tenant vault paths.** `vps({ tenantVaultPaths: { [tenantId]: paths } })` lists the `vault://` paths a tenant's
  provider accounts may name (an entry ending in `/` allows every path under it).
* **Provider contract** (`@better-compute/core`, additive). Optional `ProviderImpl` members `discover`, `claim`,
  `release`, `listReservations` and `networks`; optional `ExternalMachine` detail (`zone`, shape, `labels`,
  `networks`, `reservation`, `claimedBy`, …); `create` takes an optional `network` and `labels`;
  `ProviderContext.account` (`{ id, tenantId, platform }`) tells a provider whose account it builds a client for;
  `SecretResolver.resolve` takes an optional `{ tenantId }` context; `Instance` gains `origin`, `onDelete`,
  `networkId` and `subnetId`; the `network` resource kind (a child of `compute-tenant`, with `providerAccountId`,
  `poolId` and `projectId` attributes) and IAM actions `compute:instances:attach` and `compute:networks:*`.

### Changed [#changed-1]

* **Tenant network subnets also accept an account's `regions`** (`@better-compute/vps`). `networks.addSubnet` took a
  subnet region only from the zones of a Google Compute Engine account; it now also accepts the `regions` of an
  account whose provider has no zones (DigitalOcean). Accounts with `zones` behave as before.
* **Modules see the resolved root tenant.** `betterCompute()` resolves the root tenant from its own `rootTenantId`
  or the port's, and now hands modules a port whose `context.iam.rootTenantId` is that root, even when only
  `betterCompute({ rootTenantId })` names it. Before, such a host's root accounts were treated as a tenant's by vps
  (secret-reference scope, GCP host identity, discovery).
* **Pools hide their provider account from tenants they are offered to** (`pools.get`, `pools.list` and the view
  `pools.offer` returns). The owner still sees `providerAccountId`.
* **A provider create refused for capacity or quota ends the instance.** `CAPACITY_UNAVAILABLE` or `QUOTA_EXCEEDED`
  from `create`, or from `get` while the instance is `scheduling` or `provisioning`, now moves it to `error` with a
  `ProvisionFailed` condition and publishes `instance.failed` with that `reason`, instead of retrying forever. The
  provider is asked nothing more about it until it is deleted, `instances.start` on it is `INVALID_STATE`, and
  deleting it refunds its quota.

## 0.1.1 (2026-10-07) [#011-2026-10-07]

Better IAM 0.3 support. Every package now accepts Better IAM `>=0.2.0 <0.4.0` (dependencies and peers), so
`npm install better-compute better-iam` resolves one Better IAM, the current 0.3.x, instead of a 0.3 umbrella beside
the 0.2 packages 0.1.0 asked for.

### Changed [#changed-2]

* **Better IAM range.** `^0.2.0` becomes `>=0.2.0 <0.4.0` on every `@better-iam/*` and `better-iam` dependency and
  peer. The workspace builds and tests against 0.3.0 (its full suite and the IAM contract tests pass); 0.2.x keeps
  working. Better IAM 0.3's breaking changes do not touch Better Compute: it declares none of the newly reserved
  principal or resource attribute names, and its resource resolver returns no `classification`.

## 0.1.0 (2026-10-07) [#010-2026-10-07]

The first release on npm: the umbrella package `better-compute` and 26 `@better-compute/*` packages, all at 0.1.0.
It needs Node.js 22.12 or later, PostgreSQL 16 or later (PGlite for development and tests) and Better IAM `^0.2.0`.

### Added [#added-3]

* **Control plane.** `betterCompute()` runs the API, gateway and worker in one process or separately. PostgreSQL is
  the source of truth: bookings are guarded by `EXCLUDE` constraints and all SQL reads one clock, `compute_now()`.
  Every mutation makes one Better IAM decision; fleet paths (heartbeats, inventory, commands) make none. Stores for
  PostgreSQL and PGlite ship in `@better-compute/postgres`.
* **Capacity and reservations** (`@better-compute/capacity`). Pools, selectors, NVLink- and NUMA-aware placement,
  quotas, previews and reservations for whole GPUs, MIG partitions and shared GPU slots. Reservations also book host
  resources (`vcpu`, `memoryGiB`, hugepages) and hand cgroup limits and a NUMA-local cpuset to the node. Usage is metered
  in hourly windows.
* **Bare metal** (`@better-compute/metal`, `@better-compute/provision`). Enroll, inspect, commission, deploy,
  release, retire and rescue workflows over BMC and PXE, driven by a workflow engine with profiles and a signed image
  catalog. A machine returns to `ready` only after an attested wipe. Also dedicated-root leases, two-phase BMC
  password rotation, and GPU health remediation within a pool's disruption budget.
* **VPS, sandboxes and volumes** (`@better-compute/vps`). Instances on your own hosts (IPAM, flavors with NUMA,
  pinning and hugepage placement) or on providers (`@better-compute/provider-hetzner`,
  `@better-compute/provider-proxmox`). Sandboxes add templates, TTLs, warm pools, snapshots and SSH exec with
  short-lived certificates. Sandbox volumes and persistent host volumes attach to instances.
* **Kubernetes workers** (`@better-compute/provider-kubernetes`). bcd runs as a DaemonSet and adopts a cluster's
  nodes and GPUs into a pool; reservations with `binding: 'kubernetes'` hand devices to the kubelet through device
  plugins or a DRA driver.
* **Gateway and protocol** (`@better-compute/gateway`, `@better-compute/protocol`). Daemon sessions over gRPC with
  mutual TLS, liveness, command dispatch, and join, renew and secret forwarding. The generated
  `bettercompute.agent.v1` wire contracts are frozen.
* **PKI and the destructive-batch quorum** (`@better-compute/pki`). Local CAs, SPIFFE node certificates, sealed join
  tokens, attestors (join token, TPM, cloud instance identity, x509 proof of possession), command signing, a sealed
  secret store and broker, and trust-root rotation. Destructive command batches can require a quorum of approvers
  (`commandBatches`, `pki.approvers`). `@better-compute/audit` adds signed Merkle checkpoints over the audit log.
* **Better IAM integration** (`@better-compute/iam`). The IAM plugin and catalog, the authorization projection, an
  embedded or remote `IamPort`, and the usage sink, on Better IAM `^0.2.0` from npm.
* **CLI** (`@better-compute/cli`, the `better-compute` command). PKI setup and certificate issuance, the bcd install
  script, capacity shortcuts, `iam setup` and release signing.
* **MCP** (`@better-compute/mcp`). Model Context Protocol tools for pools, capacity, GPU reservations, instances and
  sandboxes, gated by Better IAM with confirmations for risky actions.
* **Clients and consoles.** `@better-compute/client` is a typed, browser-safe API client with live `$watch`.
  `@better-compute/console-kit` provides framework-free view models for consoles.
* **Testing kit** (`@better-compute/testing`). Fixtures, the simulated `FakeDaemon`, a sim clock, the fleet
  simulation and conformance runners.
* **More packages.** `@better-compute/events` (signed webhooks and an SSE stream), `@better-compute/fleet` (fleet
  configuration as code), `@better-compute/observability` (Prometheus metrics), `@better-compute/rollout` (signed bcd
  releases and wave rollouts) and `@better-compute/driver-sdk` (out-of-process drivers).
* **The Go daemon bcd** (`daemon/`). The node agent is built and tested end to end against a real gateway on the
  fake-hardware build; the site and ramdisk modes are covered by Go unit tests. The release pipeline (goreleaser,
  SBOMs, cosign signatures, SLSA3 provenance) is in place, but this release publishes no bcd binaries.

### Known limitations [#known-limitations]

* **No real-hardware runs.** Everything is tested against fakes and simulations: no real server has run bcd, and the
  hardware gates (a GPU host, a KVM host, the reference benchmark machines) have not run. Local test runs and the
  benchmark numbers use PGlite rather than a PostgreSQL server.
* **bcd binaries are not released.** The install script and self-update need a published bcd, so until a release
  carries binaries, build bcd from `daemon/` with Go.
* See the [roadmap](https://better-compute.com/docs/reference/roadmap) for the state of every phase.

## Development history (pre-0.1.0) [#development-history-pre-010]

The internal notes kept while the packages were built, before the first release. They cite sections of `PLAN.md`
and commit hashes, and are kept for reference; the 0.1.0 entry above summarizes them.

* Round 5, the complete node daemon (PLAN §3.8, v1.1):
  * Host resources on every host: `vcpu`, `memoryGiB`, `hugepages2Mi`, `hugepages1Gi` rows from inventory (`capacity.hosts`, `policy.hostReserve`). VPS hosts' memory now excludes hugepage pools.
  * Reservations take `resources` (`vcpu`, `memoryGiB`, `hugepages1Gi`) and default to the pool's per-GPU share (`policy.perGpu`, else the smallest host share). Host resources are not charged to quota. `capacity.preview` books the same.
  * `DeviceAssignment.limits` carry cgroup limits and a NUMA-local cpuset (without the host's reserved CPUs) to bcd.
  * `binding: 'kubernetes'` works on pools with `policy.kubernetes`. bcd hands those devices to the kubelet: device plugins or the DRA driver `gpu.bcd.dev` (`gpuApi`). Releasing a bound one keeps its GPUs and host share booked, and billed, until the node reports `AllocationReleased` (pods let go), a lost node, or the booking's end. The placer used to re-book them under running pods: 55/240 failed bindings in the 12-node sim. Meanwhile the reservation carries the `DevicesHeld` condition. A booking that pods still hold at its end is extended in 5-minute steps while the devices are free.
  * bcd advertises the Hello capabilities `devices.limits` and `binding.kubernetes`. Reservations that book host resources, or bind to Kubernetes, place only on nodes that advertise them. An older bcd would drop the limits or the binding.
  * Flavors take `placement` (`numaAffine`, `pinned`, `hugepages`). Hugepage guests book hugepages and are charged as `memoryGiB`. Pinned guests take isolated CPUs first. Guests that don't get their placement carry `PlacementDegraded`. Guests of pools that overcommit memory get a balloon.
  * Guests get their IPAM gateway and DNS (`setIpPool({ dns })`, `InstanceNic.gateway/nameservers`).
  * Agent memory: `sandboxVolumes`, persistent host volumes (`InstancesSection.volumes`, `InstanceSpec.volumes`). Placement takes `onlyNodes`.
  * Extension kind `pool-policy`: provision refuses host policies nodes can't take, at `pools.create`/`update`.
  * Telemetry specs carry `headersRef` and `resourceAttributes` (`nodeTelemetryModule`, `PoolPolicy.telemetry`).
  * Security:
    * `trustedProxies`: behind a load balancer, IAM sees the caller, and fails closed.
    * Hello capabilities `instance.ssh-host-only`, `instance.placement` and `instance.volumes` gate where sandboxes, placed flavors and volumes land.
    * Hosted-Kubernetes join tokens land nodes `tenant-controlled`.
    * Slots per shared GPU are capped at 48.
  * MCP `reserve_gpus` takes `vcpu`, `memoryGiB` and `kubernetes`.
  * **Breaking (IAM):** `sandboxVolumes.create` is decided on `compute-sandbox-volume/<draft id>`, whose projection carries the requested `sizeGiB`, instead of the tenant (6cb30e9). Policies that granted it on `compute-tenant` still reach it through the pool parent; policies that bound volume size now see it. `sandboxVolumes.resize` stages the new size the same way.
  * **Breaking (API):** `sandboxVolumes.list` defaults to the caller's own volumes; pass `scope: 'all'` for everyone's (77bf9f5).
  * **Breaking (metrics):** bcd's `bcd.instance.memory.balloon` is now `bcd.instance.memory.balloon.reclaimed` (9bdd43a). It reports flavor memory minus current guest memory, only for guests with a balloon. Dashboards and alerts on the old name need updating.
  * Hosted-Kubernetes bcd uses the kubelet's credentials (e7d0765; NodeRestriction confines it to its own Node), never a service account.
  * CI builds the DaemonSet image (daemon/Containerfile) and server-side dry-runs the rendered manifests on a kind cluster.

* Round 4, server/core/mcp/client:
  * Security: admissions decide on the committed row (`IamPort.refreshResource`; §12.4 "draft in memory before require" was unimplemented, so `extend` decided on stale projection rows: BoundedExtend allowed 72 h + 1 h). A node reports only its own allocations. With `pki.commandSigning`, a reservation goes active only on the node's own signed `AllocationBound` (`pkiAllocationBindingCheck`). The secret broker releases an instance's user data only when it names a secret of the instance's own tenant. Certificate roles live in the Subject OU `bc-role:<role>` (no private OIDs; Go cannot parse the 2.25 arc).
  * Envelope: async-with-wait (`MethodDefinition.poll`, `wait: { timeoutMs }`), `MethodAdmission.check` (budget checks outside transactions, no `denied`), W3C `traceparent` onto enqueued commands.
  * Server: audit sinks (`sinks.audit`, module extensions), module-listed event sinks and scheduler plugins; `sinks.usage` refused. `GET {basePath}/downloads/bcd/{version}/linux-{arch}`; `supportBundles` presigned `logs.collect` uploads; `projects.archive`, `commands.cancel`, `usage.export`; selfCheck §14 operations findings; fewer statements per join; the node pass wakes started reservations on connectivity/schedulability changes; `site` NodeSpec sections merge per field (`NodeSpecPart`); `vps-host` nodes may be declared lost.
  * Core/proto (additive): PoolPolicy runtime/singleTenant/workloadIdentity, InstanceSpec allocationId/nics, CapacityPort.reserve admission, ProviderImpl.identify, Reconciler.role, PlacementRequest.nodeLabels, SiteSpec.cacheImages; `driver.invoke` payloads, WorkloadToken frames (sealed JWE, purpose `workload-token`), StatusReport.pressure; catalog rows providerAccounts.bootstrap, workflows.pause/resume, metal.unrescue.
  * MCP: `iamConfirmations` (per-request polling), `remoteMcpIam` (the gate over HTTP IAM), reservation and instance tools. Client: `$watch`.
  * Later:
    * Sandboxes are a product feature (D24, user decision); the design is PLAN §3.7.
    * vps `instances.create` waits (`wait: { timeoutMs }`) and applies the pool's budget policy (`SPEND_LIMIT_REACHED`).
    * A node dedicated to one tenant wakes pending work (`mayBeSchedulableSql`).
    * Ramdisk certificates allow the commissioning kinds under their real names (`inventory.refresh`, `disk.wipe`, `logs.collect`); before this, commissioning could not inspect or wipe.
    * Proto `Hello.trust_version` / `StatusReport.trust_version` for the command-root switch gate.
    * `iamGuard` / `BETTER_COMPUTE_IAM_GUARD=warn|throw` reports or refuses an IamPort call made inside a compute transaction (§8.1); `computeFixture` defaults to `throw`.
    * Sandboxes seams: paused instances, `restore_from`, `instance.snapshot` messages, runtime `pause`/`resume`/`snapshot`.
    * Workload tokens are refused for suspended or deleted tenants.
    * MCP `create_instance` waits for the instance to start; MCP sandbox tools; FleetConfig `sandboxTemplates`.
    * Sandbox seams: `InstanceSpec.network_policy` (egress/private/none).
    * CI:
      * nightly workflow: simulation plus 10k cold start, the IAM-guarded suite, the whole suite on Postgres,
        and a green-night streak;
      * the bcd release pipeline: SBOMs, keyless cosign, SLSA3 provenance;
      * the API surface freeze check in `lint:repo`.
    * The §16 gateway gate is met on a workstation: 5k mTLS sessions, 0 drops, dispatch p99 26 ms.
    * v1.0 destructive-batch quorum, server side: the `quorum` module (`commandBatches` get/list/scope/approve/cancel,
      batch expiry) and `pki.approvers` in the trust bundle.
    * Quorum security review #6:
      * Leaves bind a payload digest, and BMC subjects are site-qualified.
      * Approvals sign the trust domain.
      * Tiers must be ordered.
      * The trust bundle version follows its content; `pki.trustVersion` is now a floor.
      * `pki.approvers.setSignatures` are published for the version they sign, and an unsigned quorum change is
        logged.
    * Envelope: `MethodAdmission.abandoned` cleans up after any failure following transaction 1 (not the IAM
      refusal, which runs `denied`).
    * Quorum API:
      * `commandBatches.create` (`batches:create`, idempotent), and `commands.run({ batchId })`; CommandInfo gains
        `batchId` and `awaitingApproval`.
      * Leaves may carry their payload, which approvers see in `scope`; its type must be the kind's, at most
        8 MiB per batch.
      * Batch subjects must be the tenant's own nodes.
      * MCP `propose_command_batch` (there is no approve tool).
    * The approver set has its own epoch (`pki.approvers.epoch`, TrustBundle field 8). A stale replica refuses to
      start rather than republish an old trust bundle.
    * `SiteSpec.proxy_targets` bounds `site.httpProxy` to the listed machines.
    * Rollout sends the signed release manifest with `NodeSpec.daemon` and `daemon.upgrade`, for bcd's self-update.
    * `lint:repo` checks the generated docs reference before the API freeze.
  * Tests: the §16 v0.3 MCP confirmation acceptance and §12.6 example policies end to end, §5.6 walkthrough 2 (acme-cloud), the authz matrix known-key replay step; `pnpm check` runs the remote-IAM suites, `proto-gen --check` and `buf breaking`. Tools: `scripts/private-commit.mjs` (compare-and-swap commits), `scripts/head-check.mjs` (tests the committed tree in a worktree).

* Round 4, capacity and sandboxes:
  * Capacity:
    * The pool-lease placer reads a lazy `PoolIndex`: per-class counts from `capacity_device_counts` (trigger-kept), booked devices only, and full devices and links per node on first view. It loads the index before taking the pool lock. At 10k nodes on PGlite, `placeQueued` goes from \~500 to \~1,000 placements/s and lock hold p90 from 875 to 60 ms.
    * Also: single-tenant offers, `dedicated=<tenant>` taints, node label constraints (`nodeLabels`, `'*'` any value), the MIG layout gate, attested `AllocationBound`, admit hooks for node leases, the budget admission check, reservation `operation`/`poll`, active conditions, `onPlacement` metrics, the I7 solver, fast-check planning properties, `capacityTenantDeleted`, and JSON Schemas for every method.
  * Sandboxes (§3.7, D24), a `sandboxes` + `sandboxTemplates` group on `vps()`:
    * Each sandbox drives one hosted instance: admission, placement, quota, IPAM, sealed user data, metering and deletion are the instance's. Instance item methods refuse a sandbox's instance.
    * Templates are checked against a hosted pool once. They carry TTL bounds, `warm: { min, max }`, `env` (tenant `secret://` refs written to `/etc/sandbox/env` inside broker-sealed user data) and `network` (`egress`/`private`/`none` → `InstanceSpec.networkPolicy`).
    * TTL expiry is an ordinary instance delete (`sandbox.expired`, then `deleted`). `extend` is decided with the new lifetime.
    * Warm pools: `paused` when the runtime can pause, otherwise `stopped`. A warm instance is claimed with `FOR UPDATE SKIP LOCKED` only after the create's own allow, and never from an older template shape. Warm capacity is metered as `vps.sandbox.warm` until claimed.
    * Snapshots: `sandboxes.snapshot` sends `instance.snapshot` through `vps({ snapshots: SnapshotStore })` into a tenant-private `sandbox_snapshots` table. `create({ fromSnapshot })` restores on the same runtime and `cpu.model`, and FakeHypervisor carries guest memory across.
    * Exec without an in-guest agent: tunnels' `sandboxes.exec` returns an `ssh` ticket (`act`, `inst`, `sbx`) plus a ≤15 min certificate from the `sandboxSsh` CA for `<account>@<instanceId>`, which vps renders into the guest's trust.
    * Also: `readyAt` stamped at first run, selfCheck findings, and `vps().tenantDeleted`, which deletes every instance of a deleted tenant (hosted instances' inline allocations were never released before).
    * Later the same day:
      * Guest host keys certified by the sandbox CA (exec pins guests). A sandbox never outlives its certificate: claims take only covering warm instances, warm pools recycle short ones, a fresh sandbox's expiry is capped, and restores or extends past it are refused. `sandboxes.exec` answers `hostCertificate: valid|expired|none`.
      * Secrets: templates use only secrets the tenant put, never fleet-scoped ones, and writing one decides `compute:secrets:manage`.
      * `private`/`none` networks place only on hosts whose bcd enforces them. Warm instances on a host that stops doing so are replaced, and running ones are a doctor error (`vps.sandboxes.network`).
      * §12.8 IAM reactions: `withdrawAgentSandboxes`, `reassignSandboxes` (`sandboxIamReactions` for the host) and `sandboxHolders` for the IamReconcile sweep.
      * Budgets: `extend` runs the budget check. Warm pools stop refilling on a spent budget, and a `drain` offer empties them.
      * `sandbox.expiring` (vps/0013), and `sandboxes.snapshot` is `UNSUPPORTED` up front without `RuntimeTraits.snapshot`.
  * `daemon/README.md` rewritten for the v0.1 daemon.

* Metal attestation and dedicated-root leases:
  * Security fix: a wipe or diagnostics result counts as attested only when the node signed it. The gateway stores `CommandUpdate.node_sig` (`commands.result_sig*`, `core/0012_v02_evidence`), and metal's default `nodeSignedResults()` verifies it with pki against `node_certs`. Before this, the kind's `attested` flag was trusted, so a gateway could forge a wipe. A lessee's `node-tenant` certificate never attests.
  * v0.5 `binding: 'node'` (metal side): a dedicated-root lease deploys the lessee's profile under `tenant-controlled` trust with a revoked platform identity, and marks the allocation bound. Its release runs the §13 teardown (fence, attested secure erase, DCGM, firmware baseline check with quarantine on drift, new identity generation).
  * `core/0012` also adds `node_bindings`, which `nodes.delete` and retirement clear. Provision exports `provisionedTargetResolver()` and `discoveryWorkflowNodeResolver()` for pki's attestors.
  * Two-phase BMC password rotation (`metal.rotateBmcCredential`, `bmc_credential_versions`): the password is generated and sealed, sent as refs only, and becomes current only on a site-signed login check. The old password stays current on failure. The tenant-root teardown rotates, and a failed rotation quarantines the machine.
  * The ramdisk's discovery token is single-use and MAC-bound. It reaches the site agent only as `ProxyDhcpEntry.discovery_token_ref`, and `boot.cleanUp` deletes it.
  * Firmware drift can be reflashed from the pool's baseline images (`policy.firmwareRemediation: 'reflash'`) before quarantine.
  * `tests/metal-capacity-lease.test.ts` runs capacity and metal together for `binding: 'node'`. The dev fleet gains `joinSite()` for site agents, and FakeDaemon signs attested results.

* v0.2 server groups and schema:
  * `joinRequests` list/approve/reject: the approval queue behind deferred joins (`discovery`, placeholder DMI). Decisions are idempotent, and an approval admits one join (the retry with the recorded CSR consumes it in the join transaction).
  * `siteTokens` create/revoke (role `site`), and the SiteBoot lease `site:<id>:boot`: one connected site agent holds it, and the standby is promoted only after it expires, under a new epoch.
  * `secrets` put/delete: write-only and sealed in the SecretStore, with a version per put. Events and audit rows carry only ref, name and version.
  * Also: `drivers.list`; `sites.update`/`sites.delete` (tombstoned once referenced); `nodes.delete` (retired or never-adopted nodes only; capacity rows released in lock order, certs revoked, hardware key freed, tombstone).
  * `core/0010_v02_server`: `join_requests`, `node_certs.allowed_kinds`, `nodes.deleted_at`, `secret_refs`.
  * `core/0011_v02_security`: `sites.deleted_at`, `attestation_uses` + `pruneAttestationUses`.
  * PGlite now runs `ANALYZE` after migrations and periodically, so dev and sim plans match Postgres.

* v0.2/v0.3 metal walkthroughs, chaos and runtime schema:
  * Talos (§5.6 #3): agentless `agent: 'none'` pools run out-of-band profiles. `httpViaSite` reaches a machine's API through its site agent (`site.httpProxy`), and it refuses before sending any command: other hosts, names, plain http off the provisioning network, redirects, oversize responses, and secret headers passed as values.
  * Ubuntu autoinstall (§5.6 #1): a third-party plugin renders a per-node ISO with a single-use `provisioned` token, and `media.insert` takes `fromStep`. The §16 v0.2 acceptance runs in sim form: 20 unattended enroll → ready → deployed → released → ready loops, and every `ready` follows a verified wipe.
  * `tests/metal-chaos.test.ts`: a worker or site agent dies mid-deploy, and no destructive command is sent twice.
  * `core/0009_runtime`: `outbox_cursors`, `work_queue.attempts`, `commands.owner`, tenant indexes, and retention via `pruneBehindCursors`, which deletes only rows every cursor has passed.

* v0.1 W5a server runtime: `betterCompute()` now runs the control plane. Table leases with epoch fencing (§8.5), the coalescing work queue, 256 sharded reconcilers with rendezvous assignment plus `Reconciler.periodic` keys and deterministic `compute.jobs.*.tick()`, module `start`/`stop`, the command queue (protobuf `CommandBody`, per-node `seq`, idempotency keys, cancel), `CommandSweeper`, the xmin-ordered outbox with sinks and `compute.events.subscribe`, the authz projection writer, and a DB-anchored clock with `ClockSkew`. The envelope runs request-as-resource admission (drafts, idempotent replay, delegated `CONFIRMATION_REQUIRED`), 30 s authentication and 10 s read-decision caches (`compute.caches`), and the list recipe. v0.1 groups: projects, sites, joinTokens, nodes, commands, usage, events. JoinService admits nodes through `@better-compute/pki` (sealed tokens, `hardware_key` from inventory, signed `JoinResponse`, Renew with proof of possession) behind `gatewayServices`; the NodeLifecycle, NodeSpec, Liveness and drain controllers; `capacityModule` is built in and backs `ctx.capacity`; `gatewayReportHandlers` for `createGateway({ reports })`. `tests/server-spine-v01.test.ts` runs join → adopt → reserve → bind → liveness end to end with zero IAM calls on fleet paths; the generated authz matrix covers all 42 v0.1 rows.

* v0.1 W3a capacity: `@better-compute/capacity`. Selectors (`matchExpressions` + CLI shorthand), schedulability with its SQL twin, the placement pipeline (filters, NVLink/NUMA-aware atom pick, `binpack`/`topology`/`fragmentation`/`spread` scorers, top-N pick) and the savepoint-guarded `inlinePlacer` (≤3 replans, then `pending`); all-or-nothing quotas; release/extend decided by the `EXCLUDE` constraint; rollups (deltas + rebuild); `capacity/0001_device_links` + `syncDeviceLinks`. `capacityModule` serves `pools`, `capacity` (summary, byPool, preview, timeline), `reservations` (admission drafts, placement in tx 2, lifecycle reconciler with bind-timeout retries, leases, `maxPending`, delegated confirmation) and `quotas`; `markAllocationBound/Failed/Released` for the gateway; `capacity.usage` writes hourly `usage_windows` (`gpu.<model>.hours`, reserved time from active, D14). The torture exit test (1,000 ops, every §10.4 invariant) fixed a quota/rollup lock-order deadlock.

* v0.3 VPS: `vps()` serves `instances` (admission drafts; hosted pools placed through capacity with IPAM leases and converged from `reportInstances`; provider pools driven through `ProviderImpl` outside transactions; lost-host recreate), `flavors` and `providerAccounts` (manifest-validated, SecretRef-only credentials); `NodeSpec.instances`; `vps.usage` meters vCPU/memory/GPU hours; `providerConformanceCases()`; a 200-instance churn test proves no leaks.

* v0.3 `@better-compute/provider-hetzner` (outside the umbrella): `hetznerCloud` (label-idempotent create, paginated offers with prices and availability, power/rebuild/list, rate-limit mapping) and `hetznerRobotProvider` (claims unnamed servers instead of ordering, resets, rescue, installimage bootstrap delivering the join token over SSH stdin only); both pass provider conformance against API emulators.

* v0.2 Health and remediation: `device_health` quarantines (`core/0007_device_health`) take one GPU out of placement without blocking its node. XID policies (48 quarantine, 63 drain+reset, 64 RMA, 79/95 drain+reboot) and inventory health findings (row remap, uncorrectable ECC, degraded NVLink) feed `recordXid`/`applyInventory`. The provision `health` reconciler runs `remediate-reboot`/`remediate-gpu-reset` after the node drains, within the pool's disruption budget (`policy.disruptionBudget`, default 5%), and stops for a pool with >10% unhealthy nodes in 10 min (circuit breaker); a failed, cancelled or exhausted remediation escalates to maintenance. v0.5 fractional GPUs (inventory side): `gpu.slot` atoms `<uuid>/s<k>` for timeslice/MPS pools, `partitionPending` freezes a GPU during a MIG mode change, and `gpuAtomFamily` feeds the atom rule. `userdata.render` renders cloud-init with a single-use `provisioned` token into a secret, which metal's deploy sends as `userDataRef`. Metal command payloads are now the proto3 JSON of each kind's message.

* v0.2 metal: `metal()` serves the `metal` group. `enroll` keeps inline BMC credentials only as a secret reference, and driver bindings are inherited from the pool's hardware profile. The `metal` reconciler runs each provision state's workflow (`metal-enroll`, `metal-inspect`, `metal-commission`, the tenant's deploy profile, `metal-release`, `metal-retire`, `metal-rescue`) and applies `complete`/`fail`. A release returns to `ready` only after an attested wipe. Control-plane steps relay BMC and boot work to the site agent and disk, diagnostics and image work to the ramdisk; `boot.kexec` completes on rejoin. There are site and ramdisk command kinds, and `metal/0001_nodes`.

* v0.2 provision additions: the image catalog and `images` group (`provision/0002_images`: digest plus Ed25519/P-256 signatures against signer allowlists, unsigned images private, platform-signed ramdisks, promotion with a wildcard authz row, site caches with LRU eviction). Also site-step routing to the boot-lease holder, workflow `owner` wake-ups and profile snapshots (`provision/0003_workflow_owner`), `awaitCommand` for relaying steps, `defineStep({ relays })` and `provisionModule({ modules })`. Migrations `core/0005_audit_commit_order` and `core/0006_pool_archive`.

* v0.2 provision: the workflow engine. `provisionModule({ steps, profiles })` owns `provision/0001_workflows` (`profiles`, `workflows`, `workflow_steps`, with one active workflow and one in-flight step per node), the `workflow` reconciler (sharded by node) and `workflowEngine(ctx)` for `ctx.workflows`. Profiles expand into persisted steps: registered steps, or driver operations sent as one `driver.invoke` command, plus `fallback:`, acceptance, on-failure and cleanup steps. Control-plane steps run between transactions (`stepNotReady` polls), agent steps are exactly one command (`wf:<id>:<step>`, `#<attempt>` on retry), and `await: 'rejoin'` completes on a new boot or role and fails only on timeout. Also `timeout`, `retries`, `onFailure`, `when:`, abort (cleanup still runs), pause/resume and retry-from-step, the built-in `wait.agentJoined` / `wait.generationObserved`, and the `profiles` and `workflows` API groups.

* v0.1 W3b inventory and store: section verification over received bytes, snapshot digests, diffs, devices and MIG atoms, `gpuLinks`, GFD labels, `GpuCountMismatch` / `GpuModelUnknown` / `DuplicateIdentity`, `hardware_key` (with `hardwareKeyFromClaims` for join), `recordInventoryHashes` / `ingestInventorySection` for the gateway, `applyInventory` + `inventoryReconciler`, and `inventory.get`. Migrations `core/0003_v01` (v0.1 columns, `node_inventory`, `node_inventory_sections`) and `core/0004_command_cancel`. The outbox conformance reader orders by `outbox.txid`, not its text alias.

* v0.2 PKI (ready for the server to wire): `localSealedSecretStore` over `sealed_secrets` (`pki/0001_sealed_secrets`, AES-256-GCM with ref+tenant AAD, key rotation via `reseal`), the `SecretRequest`/`SecretResponse` broker crypto with node signatures checked against `node_certs`, a command-signing root with 24 h worker keys and `signCommand`/`verifyCommand`/`signDesiredState`/`verifyDesiredState`, trust-root rotation, and `verifyAttestedUpdate`/`verifyAttestedReport`. Also: PKI bundles sealed under the deployment secret, `agentIdentityFromChain`, the agent-side join/renew helpers, `doctor` PKI findings, the capacity CLI shortcuts and `iam setup`.

* v0.1 W5c CLI: `pki init` (root + node/serving/internal CAs + JoinService keys, sealed under `BETTER_COMPUTE_SECRET`, optional `--offline-root`), `pki show` (bcd pins from the public half), `pki issue-gateway` (30-day serving certs, atomic replace), and `daemon install-script` (POSIX sh: token from `BCD_JOIN_TOKEN`/stdin only, pinned sha256 download, 0600 `config.toml` in bcd's TOML subset, hardened `bcd.service`, in-place upgrade without a token). The CLI framework routes two-word commands and `help <group>`.

* v0.1 W2 PKI: `@better-compute/pki`. Local root, node, serving and internal CAs with per-kind issuance policy (SPIFFE URI SAN + role extension, ≤ 24 h node certs, ≤ 2 h ramdisk), a strict DER/X.509/PKCS#10 layer on `node:crypto`, chain verification, join tokens (`issueJoinToken`, `verifyJoinToken`, `joinTokenAttestor` on `join_tokens` at `compute_now()`), sealed tokens bound to the join CSR (JWE `ECDH-ES+A256KW`/`A256GCM`), signed `JoinResponse` and `SignedTrustBundle`, `verifyRenewPop` with a malleability-proof replay cache, and a `RevocationSet`. `hardware_key` stays in `inventory`, the single source. Tests include a mutual-TLS handshake and the RFC 7518/3394 vectors.

* P0 scaffold: a pnpm workspace mirroring better-iam's, with the packages, contracts, repository rules and CI that v0.1 builds on. Plan `PLAN.md`; guide `docs/development.md`.
  * Packages `@better-compute/core`, `protocol`, `inventory`, `provision`, `metal`, `vps`, `postgres`, `server`, `client`, `cli` and `testing`, and the umbrella `better-compute` with subpaths `./core`, `./protocol`, `./inventory`, `./provision`, `./metal`, `./vps`, `./server`, `./postgres`, `./pglite`, `./client`, `./cli`, `./testing` and bin `better-compute`. `inventory`, `provision`, `metal` and `vps` are stubs: they export the frozen contract types, and their factories throw `NOT_IMPLEMENTED` naming the phase that implements them.
  * better-iam 0.1.0 is vendored as 23 tarballs in `vendor/better-iam` and pinned through `pnpm-workspace.yaml` `overrides` (`scripts/sync-better-iam.mjs`, `pnpm sync:iam`).
  * Repository rules under `pnpm lint:repo` (part of `pnpm check`): `scripts/check-deps.mjs` (server never imports injected modules, `gateway`, `iam`, `postgres`, providers or drivers; core's root entry imports no Node.js built-in; workspace dependencies match tsconfig references; nothing imports a `provider-*`), `scripts/check-sql-now.mjs` (SQL reads time only through `compute_now()`) and `scripts/umbrella.mjs --check` (the umbrella is generated from `packages/better-compute/umbrella.json`; `pnpm umbrella` rewrites it).
  * Packaging, ported from better-iam: `validate:packages` (Apache-2.0, `publishConfig.access`, `repository.directory`, `{ types, import }` exports whose files exist, publint, one version), `pack:all`, `check:packed-types` (`attw --profile esm-only`), `test:packed` (a fresh consumer installs every tarball with the vendored better-iam, imports `better-compute`, `/inventory`, `/provision`, `/metal`, `/vps`, `/pglite` and `/client`, and runs `better-compute --help`) and `scripts/release-version.mjs`.
  * CI: `.github/workflows/ci.yml` (Ubuntu and Windows × Node 22 and 24: `pnpm check`, `pnpm proto:check`, `buf breaking`, pack, attw, packed smoke; a `postgres:17` job for `pnpm test:postgres`; npm publishing that waits until better-iam is on npm) and `.github/workflows/daemon.yml` (`go vet`, `go test -tags nonvml,fakehw ./...`, `goreleaser check`).
  * `bench/REFERENCE.md` defines the reference hardware for the v0.1 gateway, IAM-lane and placement benchmarks.


# CLI (/docs/reference/cli)

> The better-compute command line 0.3.0: 193 commands.



Run `better-compute help <command>` for the same text in a terminal, or `better-compute help --json` for this manifest.

## Setup [#setup]

### init [#init]

Write a starter configuration file

```sh
better-compute init [--config better-compute.config.ts]
```

init writes a starter better-compute.config.ts (or the JavaScript form when --config names a .mjs or .js file): a configuration factory that uses PGlite in a local .better-compute-pglite directory for development and PostgreSQL when BETTER\_COMPUTE\_DATABASE\_URL is set, reads the secret and base URL from the environment, shows where better-iam plugs in, and exports an empty list of project commands. It never overwrites an existing file.

| Flag            | Description                                               | Default                    | Environment |
| --------------- | --------------------------------------------------------- | -------------------------- | ----------- |
| `--config PATH` | Where to write the configuration (.ts, .mts, .js or .mjs) | `better-compute.config.ts` |             |

```sh
better-compute init
better-compute init --config better-compute.config.mjs
```

### migrate [#migrate]

Create or upgrade the database schema and apply module migrations

```sh
better-compute migrate [--config PATH]
```

migrate runs compute.initialize(): it checks every method's IAM action against the merged catalog and the normative method table (INVALID\_CONFIG on a mismatch), then creates or upgrades the compute schema and applies every module's and plugin's migrations. Run it on every deploy before the application starts; it is idempotent.

| Flag            | Description                                                                       | Default | Environment             |
| --------------- | --------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--config PATH` | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts} |         | `BETTER_COMPUTE_CONFIG` |

```sh
better-compute migrate
BETTER_COMPUTE_DATABASE_URL=pglite:./data better-compute migrate
```

### doctor [#doctor]

Check the database, schema, clock, method registry and IAM wiring

```sh
better-compute doctor [--strict] [--pki PATH] [--config PATH]
```

doctor runs compute.selfCheck() (applied migrations, btree\_gist, compute\_now() against the database clock, the method → action registry, and each module's own checks) and adds the CLI's findings, such as a deployment with no IAM configured, the PKI bundle, and the configuration's host export (whether the gateway role can authenticate daemons by node certificate, reach the api's pokes, tunnels and revocations, and whether host routes fit beside the API). It only reads. It prints the report, then exits non-zero (DOCTOR\_FINDINGS) when any finding is an error; --strict also exits non-zero on a warning. Informational findings never fail it.

| Flag              | Description                                                                               | Default                    | Environment             |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | ----------------------- |
| `--strict`        | Also exit non-zero on warnings (errors always do)                                         |                            |                         |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`    |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |                            | `BETTER_COMPUTE_CONFIG` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                         |

```sh
better-compute doctor
better-compute doctor --strict --query "findings[].check"
```

### version [#version]

Print the better-compute version

```sh
better-compute version
```

version prints the version of this command line (also --version or -v).

### completion [#completion]

Print a shell completion script (bash, zsh or fish)

```sh
better-compute completion SHELL
```

completion prints a script that completes commands, their flags and the values of flags with choices, generated from this program's commands (a configuration's project commands included). Load it in your shell's startup file.

```sh
source <(better-compute completion bash)
source <(better-compute completion zsh)
better-compute completion fish | source
```

### serve [#serve]

Run the API and the workers in this process

```sh
better-compute serve [--host HOST] [--port PORT] [--roles ROLES] [--no-migrate] [--config PATH]
```

serve loads the configuration, migrates (skip with --no-migrate once a deploy step runs better-compute migrate), starts the worker roles (controllers: reconcilers and the command sweeper; scheduler; sinks: outbox dispatch; iam: IAM jobs) under their leases, and serves the API with compute.nodeHandler on --host:--port. Run several processes against one database: leases and shard ownership keep each job single-writer. --roles api serves the API only; --roles controllers,sinks runs a worker with no listener; --roles gateway runs the daemon gateway over mTLS from the configuration's host.gateway (all includes it when configured), with the report hooks of the built-in modules and tunnels when the deployment issues exec tickets. Beside the API the listener answers /healthz, /readyz, /.well-known/better-compute (the server's document, with JoinService's public keys when pki.join publishes them: the fallback bcd checks against its pins) and the host export's own routes (for example GET /watch and GET /metrics). SIGINT or SIGTERM drains the gateway first, then stops the listener and the workers, releasing their leases.

| Flag            | Description                                                                       | Default   | Environment             |
| --------------- | --------------------------------------------------------------------------------- | --------- | ----------------------- |
| `--host HOST`   | The address to listen on                                                          | `0.0.0.0` | `HOST`                  |
| `--port PORT`   | The port to listen on (0 picks a free one)                                        | `3000`    | `PORT`                  |
| `--roles ROLES` | all, or a comma list of api, gateway and controllers, scheduler, sinks, iam       | `all`     | `BETTER_COMPUTE_ROLES`  |
| `--no-migrate`  | Do not run migrations first                                                       |           |                         |
| `--config PATH` | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts} |           | `BETTER_COMPUTE_CONFIG` |

```sh
better-compute serve
better-compute serve --roles api --port 8080
better-compute serve --roles controllers,sinks,iam --no-migrate
better-compute serve --roles gateway,controllers
```

### dev [#dev]

Run a local development server with no setup

```sh
better-compute dev [--host HOST] [--port PORT] [--data DIR] [--memory] [--sim N] [--sim-variant VARIANT] [--config PATH]
```

dev runs the API and every worker role on 127.0.0.1. With a configuration file (or BETTER\_COMPUTE\_DATABASE\_URL) it uses that deployment; without one it keeps PGlite in --data (default .better-compute/dev-db; --memory for a throwaway database) and a generated deployment secret in .better-compute/dev-secret (owner-only), so repeated runs keep their data. It always migrates first. Without IAM configured only public methods answer; see better-compute iam setup. --sim N adds a gateway on loopback cleartext HTTP/2 (sessions authenticate by a header; local development only) and N simulated machines that join the way bcd does: a join token sealed to throwaway dev JoinService keys, the Join RPC, a Session stream, diag.nvml acceptance and adoption by the real runtime. Without configured IAM, --sim starts an embedded in-memory better-iam and prints its root token; with configured IAM it acts as BETTER\_COMPUTE\_TOKEN, which must be a root administrator.

| Flag                    | Description                                                                                            | Default                  | Environment             |
| ----------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------ | ----------------------- |
| `--host HOST`           | The address to listen on                                                                               | `127.0.0.1`              |                         |
| `--port PORT`           | The port to listen on (0 picks a free one)                                                             | `3000`                   | `PORT`                  |
| `--data DIR`            | Where PGlite keeps the development database (no configuration only)                                    | `.better-compute/dev-db` |                         |
| `--memory`              | Use a throwaway in-memory database                                                                     |                          |                         |
| `--config PATH`         | The configuration file (default: discovered)                                                           |                          | `BETTER_COMPUTE_CONFIG` |
| `--sim N`               | Join N simulated machines (FakeDaemons) through a loopback dev gateway (needs @better-compute/testing) |                          |                         |
| `--sim-variant VARIANT` | Hardware of the simulated machines: hgx-h100, hgx-h100-gpu-missing, l40s, cpu                          | `hgx-h100`               |                         |

```sh
better-compute dev
better-compute dev --memory --port 0
better-compute dev --memory --sim 8
better-compute dev --sim 4 --sim-variant l40s
```

### sim [#sim]

Run the deterministic fleet simulator and check its invariants

```sh
better-compute sim [--seed SEED] [--nodes N] [--duration DURATION] [--settle DURATION] [--chaos LEVEL] [--log PATH]
```

sim runs --nodes FakeDaemons on fake hardware against the in-memory gateway stub for --duration of simulated time, with a steady workload (diagnostics, inventory refreshes, log collection, reboots, GPU bookings) and --chaos faults (none, light, heavy: crashes, reboots, journal loss, partitions, half-open links, resets, stalls, gateway kills), then quiesces for --settle and checks invariants I1 (no double booking), I2 (at-most-once side effects), I5 (no allocation released only because its node disconnected) and I8 (usage). Nothing reads the wall clock: the same --seed prints the same digest. It exits SIM\_INVARIANT\_VIOLATED when an invariant fails. Needs @better-compute/testing.

| Flag                  | Description                                                                               | Default | Environment |
| --------------------- | ----------------------------------------------------------------------------------------- | ------- | ----------- |
| `--seed SEED`         | The seed                                                                                  | `sim`   |             |
| `--nodes N`           | Simulated nodes                                                                           | `50`    |             |
| `--duration DURATION` | Simulated time under workload                                                             | `10m`   |             |
| `--settle DURATION`   | Quiet time afterwards (sweeper and replays settle)                                        | `16m`   |             |
| `--chaos LEVEL`       | none, light or heavy                                                                      | `light` |             |
| `--log PATH`          | Write the event log to a new file                                                         |         |             |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                          |         |             |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |             |

```sh
better-compute sim
better-compute sim --seed nightly-42 --nodes 500 --duration 1h --chaos heavy
```

### support bundle [#support-bundle]

Write a redacted diagnostics bundle for a support request

```sh
better-compute support bundle [--out PATH] [--pki PATH] [--config PATH]
```

support bundle collects what a support request about the control plane needs: the CLI, Node and platform versions; the configuration's shape (database kind, modules, whether IAM, PKI, tunnels and a notifier are configured; never URLs or values); doctor's findings; applied migrations; gateway and worker heartbeats; work-queue, command and IAM-inbox backlogs; and node counts by state. Environment variables appear by name only. Before anything is written, the bundle is scanned for join tokens (bcjt\*), IAM tokens (biam\*), JWTs, private keys, URL passwords and the deployment's own secrets, and each is replaced; a bundle in which a secret survives is refused (SUPPORT\_BUNDLE\_LEAK). The file is written owner-only. It only reads.

| Flag              | Description                                                                               | Default                    | Environment             |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | ----------------------- |
| `--out PATH`      | Where to write it (default better-compute-support-\<time>.json; - prints it)              |                            |                         |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`    |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |                            | `BETTER_COMPUTE_CONFIG` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                         |

```sh
better-compute support bundle
better-compute support bundle --out - > bundle.json
```

## API [#api]

### api [#api-1]

Call any compute API method as the current token

```sh
better-compute api GROUP.METHOD [key=value ...] [--data JSON|@FILE|-] [--list] [--config PATH] [--url URL]
```

api calls any method of the HTTP API (the same routes, envelope and permission checks as POST \{basePath}/\{group}/\{method}) as BETTER\_COMPUTE\_TOKEN, in process through the configuration or against --url, and prints the result. Public methods (system.version, system.whoami) need no token; system.whoami reports the token when one is set. Build the input with --data (JSON, @file.json, or - for standard input) and key=value items: name=gpu-a sets a string, limit:=10 or labels:='\["a","b"]' sets JSON, spec:=@spec.json reads a JSON file, notes=@notes.md reads a text file, resource.type=pool nests, and ids\[]=a appends. Plugin methods are plugins.ID.GROUP.METHOD. api --list \[GROUP] lists every route with whether it needs a token and the IAM action it is decided on.

\| Flag              | Description                                                                                                   | Default | Environment                                          |
\| ----------------- | ------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------- | --- | --- |
\| `--data JSON      | @FILE                                                                                                         | -`      | Request body as JSON, a JSON file, or standard input |     |     |
\| `--list`          | List routes (optionally of one group) instead of calling one                                                  |         |                                                      |
\| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                            |         | `BETTER_COMPUTE_CONFIG`                              |
\| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`                                 |
\| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                              |         |                                                      |
\| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                      |         |                                                      |

```sh
better-compute api --list
better-compute api system.version
better-compute api system.selfCheck --query "findings[].check"
better-compute api system.whoami --url https://compute.example.com
better-compute api inventory.get tenantId=tenant-acme nodeId=node_01jz0000000000000000000000
echo '{"tenantId":"tenant-acme","widgetId":"w1"}' | better-compute api plugins.acme.widgets.get --data -
```

## Access [#access]

### auth login [#auth-login]

Save a token for later commands (checked with system.whoami)

```sh
better-compute auth login [--profile NAME] [--no-verify] [--config PATH] [--url URL]
```

auth login reads an IAM session token, API key or session JWT from standard input (or BETTER\_COMPUTE\_TOKEN), checks it with system.whoami against the deployment (--url, or the configuration in process; --no-verify skips the check), and saves it under --profile in better-compute's credentials file (BETTER\_COMPUTE\_CREDENTIALS, else %APPDATA%\better-compute or \~/.config/better-compute), written owner-only. Token commands then use it when BETTER\_COMPUTE\_TOKEN is unset and no better-iam session is saved. A session that better-iam's own CLI saved (better-iam login) is used without this step.

| Flag              | Description                                                                                                                                               | Default | Environment              |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------ |
| `--profile NAME`  | The saved profile (default: the current one, else default)                                                                                                |         | `BETTER_COMPUTE_PROFILE` |
| `--no-verify`     | Save without calling system.whoami                                                                                                                        |         |                          |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG`  |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`     |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                          |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                          |

```sh
better-compute auth login --url https://compute.example.com < token.txt
BETTER_COMPUTE_TOKEN=biam_key_… better-compute auth login --profile ci --no-verify
```

### auth status [#auth-status]

Show which credential token commands use (never the token)

```sh
better-compute auth status [--profile NAME]
```

auth status says where token commands get their credential, in PLAN §9.3's order: BETTER\_COMPUTE\_TOKEN, else better-iam's saved session (better-iam login, when @better-iam/cli is installed), else better-compute's saved profile. It lists the saved profiles without their tokens.

| Flag              | Description                                                                               | Default | Environment              |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ------------------------ |
| `--profile NAME`  | The saved profile (default: the current one, else default)                                |         | `BETTER_COMPUTE_PROFILE` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |                          |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |                          |

```sh
better-compute auth status
```

### auth logout [#auth-logout]

Forget a saved profile's token

```sh
better-compute auth logout [--profile NAME]
```

auth logout removes a profile from better-compute's credentials file (the current one by default). It does not revoke the session in better-iam: run better-iam logout for that.

| Flag              | Description                                                                               | Default | Environment              |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ------------------------ |
| `--profile NAME`  | The saved profile (default: the current one, else default)                                |         | `BETTER_COMPUTE_PROFILE` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |                          |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |                          |

```sh
better-compute auth logout
better-compute auth logout --profile ci
```

### ssh-cert [#ssh-cert]

Get a break-glass SSH certificate for one node, next to your key

```sh
better-compute ssh-cert [COMMAND...] --tenant ID --node ID [--key PATH] [--account NAME] [--minutes N] [--connect HOST] [--ssh PROGRAM] [--config PATH] [--url URL]
```

ssh-cert asks nodes.issueSshCert for a short-lived OpenSSH user certificate for your own public key (--key, default \~/.ssh/id\_ed25519.pub, else id\_ecdsa.pub) and writes it beside the key as \<key>-cert.pub, where ssh -i \<key> finds it. The certificate is bound to the node (principal \<account>@\<nodeId>): it opens that node only, as --account (default root), for --minutes (default 15, at most 60, and never past your session). It needs compute:nodes:ssh-root and a session verified with MFA. The private key never leaves your machine. Revoke it early with api nodes.revokeSshCert.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node                                                                                                                                                  |         |                         |
| `--key PATH`      | Your OpenSSH public key (.pub)                                                                                                                            |         |                         |
| `--account NAME`  | The Unix account on the node                                                                                                                              | `root`  |                         |
| `--minutes N`     | Validity in minutes                                                                                                                                       | `15`    |                         |
| `--connect HOST`  | Then run ssh to the node's address with the new certificate (the command after --, or a shell)                                                            |         |                         |
| `--ssh PROGRAM`   | The OpenSSH client for --connect                                                                                                                          | `ssh`   |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute ssh-cert --tenant org_acme --node node_01k5w… && ssh -i ~/.ssh/id_ed25519 root@10.0.0.5
better-compute ssh-cert --tenant org_acme --node node_01k5w… --key ./ops.pub --minutes 5
better-compute ssh-cert --node node_01k5w… --connect 10.0.0.5 -- journalctl -u bcd -n 50
```

### console [#console]

Open an interactive console on a node, or an instance’s serial console

```sh
better-compute console --tenant ID [--node ID] [--instance ID] [--config PATH] [--url URL]
```

console opens a root console on one node (--node, compute:nodes:console on the node), or the serial console of one running instance (--instance, compute:instances:exec on the instance), through tunnels.open. The session runs over the server’s tunnel WebSocket, so it needs --url (or BETTER\_COMPUTE\_URL). The terminal is in raw mode until the remote program exits: every key, Ctrl-C included, goes to the remote, and Ctrl-] detaches. The command exits with the remote program’s status (255 when the tunnel breaks), like ssh. Tickets are single-use and the gateway and the node verify them again; revoking your session or the delegation closes the tunnel.

| Flag            | Description                                                                                                                                               | Default | Environment             |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`   | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`     | The node (a root shell on it)                                                                                                                             |         |                         |
| `--instance ID` | The instance (its serial console)                                                                                                                         |         |                         |
| `--config PATH` | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`     | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |

```sh
better-compute console --url https://compute.example.com --tenant org_acme --node node_01k5w…
better-compute console --url https://compute.example.com --tenant org_acme --instance inst_01k5w…
```

### exec [#exec]

Open an interactive shell in a running instance

```sh
better-compute exec --tenant ID --instance ID [--config PATH] [--url URL]
```

exec opens a shell in one running instance through tunnels.open (compute:instances:exec on the instance); its node's agent runs it. The session runs over the server’s tunnel WebSocket, so it needs --url (or BETTER\_COMPUTE\_URL). The terminal is in raw mode until the remote program exits: every key, Ctrl-C included, goes to the remote, and Ctrl-] detaches. The command exits with the remote program’s status (255 when the tunnel breaks), like ssh. Tickets are single-use and the gateway and the node verify them again; revoking your session or the delegation closes the tunnel.

| Flag            | Description                                                                                                                                               | Default | Environment             |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`   | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--instance ID` | The instance                                                                                                                                              |         |                         |
| `--config PATH` | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`     | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |

```sh
better-compute exec --url https://compute.example.com --tenant org_acme --instance inst_01k5w…
```

### forward [#forward]

Forward a local port to a port of a running instance

```sh
better-compute forward --tenant ID --instance ID --port PORT [--local-port PORT] [--bind HOST] [--config PATH] [--url URL]
```

forward listens on --bind (default 127.0.0.1) at --local-port (default: the same as --port; 0 picks a free one) and forwards each connection to --port on the instance through its own tcp tunnel (tunnels.open, compute:instances:exec on the instance), like ssh -L. It runs until Ctrl-C. The tunnels run over the server’s WebSocket, so it needs --url (or BETTER\_COMPUTE\_URL). A connection is re-authorized like any tunnel: revoking your session or the delegation closes it. Binding anything but loopback lets whoever reaches that address use your access; even on loopback, other users of this machine and web pages open in your browser can reach the port, as with ssh -L. At most 16 connections are forwarded at once, and new ones are refused while the server rate-limits your tunnels.

| Flag                | Description                                                                                                                                               | Default     | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |             | `BETTER_COMPUTE_TENANT` |
| `--instance ID`     | The instance                                                                                                                                              |             |                         |
| `--port PORT`       | The port on the instance                                                                                                                                  |             |                         |
| `--local-port PORT` | The local port (default: --port; 0 picks a free one)                                                                                                      |             |                         |
| `--bind HOST`       | The local address to listen on                                                                                                                            | `127.0.0.1` |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |             | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |             | `BETTER_COMPUTE_URL`    |

```sh
better-compute forward --url https://compute.example.com --tenant org_acme --instance inst_01k5w… --port 5432
better-compute forward --instance inst_01k5w… --port 8888 --local-port 0
```

## Capacity [#capacity]

### reserve [#reserve]

Reserve GPUs from a pool (reservations.create)

```sh
better-compute reserve --tenant ID --pool ID --gpus N --for DURATION [--selector EXPR] [--model MODEL] [--product PRODUCT] [--nvlink] [--project ID] [--start TIME] [--lease SECONDS] [--key KEY] [--wait SECONDS] [--cpus N] [--memory GIB] [--kubernetes] [--config PATH] [--url URL]
```

reserve asks for --gpus whole GPUs from --pool for --for, decided by better-iam as compute:reservations:create on the reservation draft and placed by the pool's placer (PLAN.md §3.4, §10.3). --selector takes the shorthand `gpuProduct in (H100-SXM5-80GB), vramGiB >= 80`; --model and --product add a gpuModel or gpuProduct match. It waits up to --wait seconds for the placement and prints the reservation, whose binding lists the booked GPU UUIDs once active. Every run sends a new idempotency key unless --key is given: repeat the command with the same --key to retry safely.

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`         | The pool to reserve from (pool\_…)                                                                                                                        |         |                         |
| `--gpus N`          | Whole GPUs to reserve                                                                                                                                     |         |                         |
| `--for DURATION`    | How long: 30m, 8h, 7d                                                                                                                                     |         |                         |
| `--selector EXPR`   | Device selector shorthand, e.g. "vramGiB >= 80"                                                                                                           |         |                         |
| `--model MODEL`     | GPU model family, e.g. H100                                                                                                                               |         |                         |
| `--product PRODUCT` | GPU product, e.g. H100-SXM5-80GB                                                                                                                          |         |                         |
| `--nvlink`          | All GPUs must share one NVLink domain                                                                                                                     |         |                         |
| `--project ID`      | Project to account the reservation to (prj\_…)                                                                                                            |         |                         |
| `--start TIME`      | Start later: ISO 8601 time or +DURATION (default now)                                                                                                     |         |                         |
| `--lease SECONDS`   | Renewable lease: released unless renewed within this many seconds                                                                                         |         |                         |
| `--key KEY`         | Idempotency key, to retry an earlier request                                                                                                              |         |                         |
| `--wait SECONDS`    | How long to wait for the placement                                                                                                                        | `20`    |                         |
| `--cpus N`          | Host CPUs to book with the GPUs (default: the pool's per-GPU share)                                                                                       |         |                         |
| `--memory GIB`      | Host memory in GiB to book with the GPUs (default: the pool's per-GPU share)                                                                              |         |                         |
| `--kubernetes`      | Give the GPUs to the node's kubelet (pools that join a Kubernetes cluster)                                                                                |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute reserve --pool pool_01k… --gpus 4 --for 8h
better-compute reserve --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes
better-compute reserve --pool pool_01k… --gpus 8 --for 3d --model H100 --nvlink --selector "vramGiB >= 80"
better-compute reserve --pool pool_01k… --gpus 2 --for 1h --start +2h --key nightly-2026-09-24
```

### release [#release]

Release a reservation early (reservations.release)

```sh
better-compute release RESERVATION --tenant ID [--cancel] [--config PATH] [--url URL]
```

release ends an active or placed reservation now: its bookings are truncated, the GPUs return to the pool and billing stops at the release time. A reservation that has not started yet can be withdrawn with --cancel instead (reservations.cancel).

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--cancel`        | Withdraw a reservation that has not started                                                                                                               |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute release rsv_01k…
better-compute release rsv_01k… --cancel
```

### extend [#extend]

Extend a reservation (reservations.extend)

```sh
better-compute extend RESERVATION --tenant ID [--by DURATION] [--until TIME] [--renew] [--key KEY] [--config PATH] [--url URL]
```

extend moves the end of a reservation later, by --by or to --until. It is admitted like a new reservation: better-iam decides compute:reservations:extend on the total duration after the change, so policy caps also bound extensions, and the booking must still be free. With --renew it instead renews a leased reservation (reservations.renew).

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--by DURATION`   | Extend by this much: 30m, 4h, 1d                                                                                                                          |         |                         |
| `--until TIME`    | Extend to this time: ISO 8601 or +DURATION                                                                                                                |         |                         |
| `--renew`         | Renew the lease of a renewable reservation instead                                                                                                        |         |                         |
| `--key KEY`       | Idempotency key, to retry an earlier request                                                                                                              |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute extend rsv_01k… --by 4h
better-compute extend rsv_01k… --renew
```

### reservations list [#reservations-list]

List the tenant’s reservations (reservations.list)

```sh
better-compute reservations list --tenant ID [--state STATE] [--pool ID] [--limit N] [--cursor ID] [--config PATH] [--url URL]
```

List the tenant’s reservations (reservations.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--state STATE`   | Only this state: active, placed, pending, …                                                                                                               |         |                         |
| `--pool ID`       | Only this pool                                                                                                                                            |         |                         |
| `--limit N`       | Page size                                                                                                                                                 |         |                         |
| `--cursor ID`     | Continue after this reservation                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute reservations list --state active
better-compute reservations list --format json
```

### reservations get [#reservations-get]

Show one reservation, with its binding (reservations.get)

```sh
better-compute reservations get RESERVATION --tenant ID [--config PATH] [--url URL]
```

Show one reservation, with its binding (reservations.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute reservations get rsv_01k… --query binding
```

### pools list [#pools-list]

List the pools the tenant owns or is offered (pools.list)

```sh
better-compute pools list --tenant ID [--config PATH] [--url URL]
```

List the pools the tenant owns or is offered (pools.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools list
```

### capacity [#capacity-1]

Show free and allocated capacity: a summary, or one pool (capacity.summary, capacity.byPool)

```sh
better-compute capacity --tenant ID [--pool ID] [--preview] [--gpus N] [--for DURATION] [--selector EXPR] [--cpus N] [--memory GIB] [--kubernetes] [--config PATH] [--url URL]
```

capacity prints what the tenant can see: per pool and per GPU model, total, allocatable, allocated, reserved and available. With --pool it prints that pool's rollup. With --preview and a --gpus request it asks where such a reservation would fit right now (capacity.preview) without booking anything.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | One pool                                                                                                                                                  |         |                         |
| `--preview`       | Preview a placement (needs --pool, --gpus and --for)                                                                                                      |         |                         |
| `--gpus N`        | GPUs for --preview                                                                                                                                        |         |                         |
| `--for DURATION`  | Duration for --preview                                                                                                                                    |         |                         |
| `--selector EXPR` | Selector shorthand for --preview                                                                                                                          |         |                         |
| `--cpus N`        | Host CPUs to book with the GPUs (default: the pool's per-GPU share)                                                                                       |         |                         |
| `--memory GIB`    | Host memory in GiB to book with the GPUs (default: the pool's per-GPU share)                                                                              |         |                         |
| `--kubernetes`    | Give the GPUs to the node's kubelet (pools that join a Kubernetes cluster)                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute capacity
better-compute capacity --pool pool_01k…
better-compute capacity --preview --pool pool_01k… --gpus 8 --for 1d --selector "vramGiB >= 80"
```

## Sandboxes [#sandboxes]

### sandbox create [#sandbox-create]

Create a sandbox from a template (sandboxes.create)

```sh
better-compute sandbox create --tenant ID --template ID [--ttl DURATION] [--from-snapshot ID] [--wait SECONDS] [--key KEY] [--config PATH] [--url URL]
```

sandbox create starts a sandbox from --template (sbxt\_…), claimed from the template's warm pool when it keeps one, decided by better-iam as compute:sandboxes:create. It lives --ttl (default: the template's default, at most its maximum) and is deleted then unless extended. --from-snapshot restores a snapshot of yours instead of booting the image. It waits up to --wait seconds (default 30) for the sandbox to leave `starting` and prints it. Every run sends a new idempotency key unless --key is given: repeat with the same --key to retry safely.

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--template ID`      | The template (sbxt\_…)                                                                                                                                    |         |                         |
| `--ttl DURATION`     | How long it lives: 15m, 2h                                                                                                                                |         |                         |
| `--from-snapshot ID` | Restore this snapshot (snap\_…)                                                                                                                           |         |                         |
| `--wait SECONDS`     | Wait this long for it to start                                                                                                                            | `30`    |                         |
| `--key KEY`          | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox create --template sbxt_01k5w… --ttl 30m
better-compute sandbox create --template sbxt_01k5w… --from-snapshot snap_01k5w…
```

### sandbox list [#sandbox-list]

List sandboxes: yours, or the tenant’s with --all (sandboxes.list)

```sh
better-compute sandbox list --tenant ID [--all] [--deleted] [--template ID] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List sandboxes: yours, or the tenant’s with --all (sandboxes.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--all`           | Every sandbox of the tenant, not only yours                                                                                                               |         |                         |
| `--deleted`       | Include deleted sandboxes                                                                                                                                 |         |                         |
| `--template ID`   | Only this template’s (sbxt\_…)                                                                                                                            |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox list
better-compute sandbox list --all --query "items[*].id"
```

### sandbox get [#sandbox-get]

Show a sandbox, its expiry and its snapshots (sandboxes.get)

```sh
better-compute sandbox get SANDBOX --tenant ID [--config PATH] [--url URL]
```

Show a sandbox, its expiry and its snapshots (sandboxes.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox get sbx_01k5w…
```

### sandbox extend [#sandbox-extend]

Move a sandbox’s expiry (sandboxes.extend)

```sh
better-compute sandbox extend SANDBOX --tenant ID [--by DURATION] [--until TIME] [--config PATH] [--url URL]
```

sandbox extend adds --by to the current expiry, or sets it to --until (ISO 8601, epoch ms or +DURATION from now), within the template's maximum lifetime. Decided as compute:sandboxes:update on the sandbox.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--by DURATION`   | Add this to the expiry: 30m, 2h                                                                                                                           |         |                         |
| `--until TIME`    | The new expiry                                                                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox extend sbx_01k5w… --by 30m
better-compute sandbox extend sbx_01k5w… --until +2h
```

### sandbox snapshot [#sandbox-snapshot]

Snapshot a sandbox, memory included unless --no-memory (sandboxes.snapshot)

```sh
better-compute sandbox snapshot SANDBOX --tenant ID [--no-memory] [--keep] [--key KEY] [--config PATH] [--url URL]
```

sandbox snapshot asks the sandbox's host for its instance's state; the snapshot is pending until the host reports it. Snapshots go with the sandbox unless --keep. Restore one with sandbox create --from-snapshot.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--no-memory`     | Disk only (not a running state)                                                                                                                           |         |                         |
| `--keep`          | Keep it after the sandbox is deleted                                                                                                                      |         |                         |
| `--key KEY`       | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox snapshot sbx_01k5w… --keep
```

### sandbox delete [#sandbox-delete]

Delete a sandbox now (sandboxes.delete)

```sh
better-compute sandbox delete SANDBOX --tenant ID [--config PATH] [--url URL]
```

Delete a sandbox now (sandboxes.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox delete sbx_01k5w…
```

### sandbox exec [#sandbox-exec]

Run a command (or a shell) in a sandbox over SSH through a tunnel (sandboxes.exec)

```sh
better-compute sandbox exec SANDBOX [COMMAND...] --tenant ID [--key PATH] [--minutes N] [--ssh PROGRAM] [--accept-host-key] [--config PATH] [--url URL]
```

sandbox exec asks sandboxes.exec for a short-lived SSH certificate for your own key (--key, default \~/.ssh/id\_ed25519; its .pub is sent, the private key never leaves this machine) and a single-use tunnel ticket to the sandbox's instance, then runs your OpenSSH client (--ssh, default ssh) through that tunnel: the command after --, or a login shell. There is no agent in the guest; its sshd trusts the sandbox CA for \<account>@\<instance>. The certificate and a known\_hosts file for this one connection live in a temporary directory that is removed afterwards. It exits with ssh's status. Needs --url (tunnels run over the server's WebSocket) and a signed-in session.

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--key PATH`        | Your private key (its .pub beside it is sent)                                                                                                             |         |                         |
| `--minutes N`       | Certificate lifetime in minutes                                                                                                                           | `10`    |                         |
| `--ssh PROGRAM`     | The OpenSSH client to run                                                                                                                                 | `ssh`   |                         |
| `--accept-host-key` | Connect although the guest's host key is not certified (older guests, or an expired host certificate): accepted on first use                              |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |

```sh
better-compute sandbox exec sbx_01k5w… -- uname -a
better-compute sandbox exec sbx_01k5w…
```

### sandbox-template create [#sandbox-template-create]

Create a sandbox template for the tenant (sandboxTemplates.create)

```sh
better-compute sandbox-template create --tenant ID --name NAME --pool ID [--runtime ID] [--flavor NAME] [--image ID] [--user-data FILE] [--default-ttl DURATION] [--max-ttl DURATION] [--warm-min N] [--warm-max N] [--env NAME=REF,…] [--network MODE] [--volumes NAME:/PATH[:ro],…] [--config PATH] [--url URL]
```

sandbox-template create defines what `sandbox create --template` starts: --pool, --flavor and --image (and the pool's --runtime), the default and longest lifetimes, a warm pool kept ready (--warm-min/--warm-max), user data from a file, environment variables from secret references and the network mode. Decided as compute:sandboxes:manage on the tenant.

| Flag                          | Description                                                                                                                                               | Default | Environment             |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`                 | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--name NAME`                 | The template name                                                                                                                                         |         |                         |
| `--pool ID`                   | The pool its instances run in (pool\_…)                                                                                                                   |         |                         |
| `--runtime ID`                | The pool's runtime driver (checked against it)                                                                                                            |         |                         |
| `--flavor NAME`               | The instance flavor (the pool offers it)                                                                                                                  |         |                         |
| `--image ID`                  | The image                                                                                                                                                 |         |                         |
| `--user-data FILE`            | cloud-init user data for every sandbox, from a file                                                                                                       |         |                         |
| `--default-ttl DURATION`      | Default lifetime: 30m, 2h                                                                                                                                 |         |                         |
| `--max-ttl DURATION`          | Longest lifetime a sandbox may be given or extended to                                                                                                    |         |                         |
| `--warm-min N`                | Warm instances kept ready                                                                                                                                 |         |                         |
| `--warm-max N`                | At most this many warm instances                                                                                                                          |         |                         |
| `--env NAME=REF,…`            | Environment from secret references: NAME=secret://\<tenant>/\<id>, comma-separated                                                                        |         |                         |
| `--network MODE`              | What sandboxes may reach (default egress)                                                                                                                 |         |                         |
| `--volumes NAME:/PATH[:ro],…` | Persistent volumes (agent memory) every sandbox mounts, in order; "" removes them                                                                         |         |                         |
| `--config PATH`               | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`                   | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`             | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`                | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-template create --name python --pool pool_01k5w… --flavor small --image img_01k5w… --warm-min 2 --max-ttl 4h
```

### sandbox-template update [#sandbox-template-update]

Change a sandbox template (sandboxTemplates.update)

```sh
better-compute sandbox-template update TEMPLATE --tenant ID [--flavor NAME] [--image ID] [--user-data FILE] [--default-ttl DURATION] [--max-ttl DURATION] [--warm-min N] [--warm-max N] [--env NAME=REF,…] [--network MODE] [--volumes NAME:/PATH[:ro],…] [--no-user-data] [--config PATH] [--url URL]
```

sandbox-template update changes only the fields given; --no-user-data removes the user data. Sandboxes already running keep what they started with; the warm pool is refilled from the new definition.

| Flag                          | Description                                                                                                                                               | Default | Environment             |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`                 | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--flavor NAME`               | The instance flavor (the pool offers it)                                                                                                                  |         |                         |
| `--image ID`                  | The image                                                                                                                                                 |         |                         |
| `--user-data FILE`            | cloud-init user data for every sandbox, from a file                                                                                                       |         |                         |
| `--default-ttl DURATION`      | Default lifetime: 30m, 2h                                                                                                                                 |         |                         |
| `--max-ttl DURATION`          | Longest lifetime a sandbox may be given or extended to                                                                                                    |         |                         |
| `--warm-min N`                | Warm instances kept ready                                                                                                                                 |         |                         |
| `--warm-max N`                | At most this many warm instances                                                                                                                          |         |                         |
| `--env NAME=REF,…`            | Environment from secret references: NAME=secret://\<tenant>/\<id>, comma-separated                                                                        |         |                         |
| `--network MODE`              | What sandboxes may reach (default egress)                                                                                                                 |         |                         |
| `--volumes NAME:/PATH[:ro],…` | Persistent volumes (agent memory) every sandbox mounts, in order; "" removes them                                                                         |         |                         |
| `--no-user-data`              | Remove the user data                                                                                                                                      |         |                         |
| `--config PATH`               | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`                   | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`             | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`                | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-template update sbxt_01k5w… --warm-min 4 --image img_01k5w…
```

### sandbox-template delete [#sandbox-template-delete]

Delete a sandbox template (sandboxTemplates.delete)

```sh
better-compute sandbox-template delete TEMPLATE --tenant ID [--config PATH] [--url URL]
```

Delete a sandbox template (sandboxTemplates.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-template delete sbxt_01k5w…
```

### sandbox templates [#sandbox-templates]

List the tenant’s sandbox templates and their warm pools (sandboxTemplates.list)

```sh
better-compute sandbox templates --tenant ID [--config PATH] [--url URL]
```

List the tenant’s sandbox templates and their warm pools (sandboxTemplates.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox templates --format table
```

### sandbox-volume create [#sandbox-volume-create]

Create a persistent volume (agent memory) on a pool (sandboxVolumes.create)

```sh
better-compute sandbox-volume create --tenant ID --pool ID --name NAME --size GIB [--key KEY] [--config PATH] [--url URL]
```

sandbox-volume create books --size GiB of disk on one host of --pool; sandboxes of templates that mount it (sandbox-template --volumes) run on that host and keep its data across runs. It is billed until deleted. Repeat with the same --key to retry safely.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--name NAME`     | A name unique in the tenant                                                                                                                               |         |                         |
| `--size GIB`      | Size in GiB                                                                                                                                               |         |                         |
| `--key KEY`       | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-volume create --pool pool_01k5w… --name agent-notes --size 20
```

### sandbox-volume list [#sandbox-volume-list]

List the tenant’s persistent volumes (sandboxVolumes.list)

```sh
better-compute sandbox-volume list --tenant ID [--pool ID] [--deleted] [--all] [--config PATH] [--url URL]
```

List the tenant’s persistent volumes (sandboxVolumes.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | Only this pool’s (pool\_…)                                                                                                                                |         |                         |
| `--deleted`       | Include deleted volumes                                                                                                                                   |         |                         |
| `--all`           | Every volume of the tenant you may see, not only yours                                                                                                    |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-volume list --format table
```

### sandbox-volume get [#sandbox-volume-get]

Show a persistent volume (sandboxVolumes.get)

```sh
better-compute sandbox-volume get VOLUME --tenant ID [--config PATH] [--url URL]
```

Show a persistent volume (sandboxVolumes.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-volume get vol_01k5w…
```

### sandbox-volume resize [#sandbox-volume-resize]

Grow a persistent volume (sandboxVolumes.resize)

```sh
better-compute sandbox-volume resize VOLUME --tenant ID --size GIB [--config PATH] [--url URL]
```

sandbox-volume resize grows the volume to --size GiB; volumes never shrink.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--size GIB`      | The new size in GiB                                                                                                                                       |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-volume resize vol_01k5w… --size 40
```

### sandbox-volume delete [#sandbox-volume-delete]

Delete a persistent volume and wipe its data (sandboxVolumes.delete)

```sh
better-compute sandbox-volume delete VOLUME --tenant ID [--config PATH] [--url URL]
```

Delete a persistent volume and wipe its data (sandboxVolumes.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-volume delete vol_01k5w…
```

### sandbox-snapshot list [#sandbox-snapshot-list]

List sandbox snapshots, kept ones of deleted sandboxes too (sandboxSnapshots.list)

```sh
better-compute sandbox-snapshot list --tenant ID [--all] [--sandbox ID] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List sandbox snapshots, kept ones of deleted sandboxes too (sandboxSnapshots.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--all`           | Every snapshot of the tenant, not only yours                                                                                                              |         |                         |
| `--sandbox ID`    | Only this sandbox’s (sbx\_…)                                                                                                                              |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-snapshot list --sandbox sbx_01k5w…
```

### sandbox-snapshot get [#sandbox-snapshot-get]

Show a sandbox snapshot and its state (sandboxSnapshots.get)

```sh
better-compute sandbox-snapshot get SNAPSHOT --tenant ID [--config PATH] [--url URL]
```

Show a sandbox snapshot and its state (sandboxSnapshots.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-snapshot get snap_01k5w…
```

### sandbox-snapshot delete [#sandbox-snapshot-delete]

Delete a sandbox snapshot (sandboxSnapshots.delete)

```sh
better-compute sandbox-snapshot delete SNAPSHOT --tenant ID [--config PATH] [--url URL]
```

Delete a sandbox snapshot (sandboxSnapshots.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sandbox-snapshot delete snap_01k5w…
```

## Nodes [#nodes]

### nodes list [#nodes-list]

List the tenant’s nodes (nodes.list)

```sh
better-compute nodes list --tenant ID [--pool ID] [--site ID] [--cordoned] [--state STATE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List the tenant’s nodes (nodes.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | Only this pool’s                                                                                                                                          |         |                         |
| `--site ID`       | Only this site’s (site\_…)                                                                                                                                |         |                         |
| `--cordoned`      | Only cordoned nodes                                                                                                                                       |         |                         |
| `--state STATE`   | Only nodes in this provisioning state                                                                                                                     |         |                         |
| `--limit N`       | Page size (at most 50)                                                                                                                                    |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes list --pool h100-eu
better-compute nodes list --cordoned --format json
```

### nodes get [#nodes-get]

Show one node: its state, labels, conditions and session (nodes.get)

```sh
better-compute nodes get NODE --tenant ID [--config PATH] [--url URL]
```

Show one node: its state, labels, conditions and session (nodes.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes get node_01k5w…
better-compute nodes get node_01k5w… --query conditions
```

### nodes events [#nodes-events]

List a node’s events, newest first (nodes.events)

```sh
better-compute nodes events NODE --tenant ID [--type TYPE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List a node’s events, newest first (nodes.events)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--type TYPE`     | Only events of this type (node.cordoned, gpu.xid, …)                                                                                                      |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes events node_01k5w… --type gpu.xid
```

### nodes cordon [#nodes-cordon]

Stop placing new work on a node (nodes.cordon)

```sh
better-compute nodes cordon NODE --tenant ID [--reason TEXT] [--if-version N] [--config PATH] [--url URL]
```

nodes cordon keeps the scheduler from placing anything new on the node; what runs there keeps running. Decided as compute:nodes:cordon. --if-version makes it conditional on the version nodes get printed.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Why (recorded in the node’s events)                                                                                                                       |         |                         |
| `--if-version N`  | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes cordon node_01k5w… --reason "disk replacement"
```

### nodes uncordon [#nodes-uncordon]

Let the scheduler place work on a node again (nodes.uncordon)

```sh
better-compute nodes uncordon NODE --tenant ID [--reason TEXT] [--if-version N] [--config PATH] [--url URL]
```

nodes uncordon undoes nodes cordon. Decided as compute:nodes:cordon.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Why (recorded in the node’s events)                                                                                                                       |         |                         |
| `--if-version N`  | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes uncordon node_01k5w… --reason "disk replacement"
```

### nodes drain [#nodes-drain]

Cordon a node and move its work off (nodes.drain)

```sh
better-compute nodes drain NODE --tenant ID [--reason TEXT] [--if-version N] [--config PATH] [--url URL]
```

nodes drain cordons the node and hands its allocations to the Drain controller, which releases them within the pool’s disruption budget. Decided as compute:nodes:drain; follow it with nodes events (node.draining, then the releases).

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Why (recorded in the node’s events)                                                                                                                       |         |                         |
| `--if-version N`  | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes drain node_01k5w… --reason "disk replacement"
```

### nodes label [#nodes-label]

Set or remove a node’s labels (nodes.label)

```sh
better-compute nodes label NODE --tenant ID [--set KEY=VALUE,…] [--remove KEY,…] [--if-version N] [--config PATH] [--url URL]
```

nodes label sets labels from --set KEY=VALUE,… and removes the keys in --remove KEY,…; selectors (reservations, pools) match on them. Decided as compute:nodes:update.

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--set KEY=VALUE,…` | Labels to set                                                                                                                                             |         |                         |
| `--remove KEY,…`    | Label keys to remove                                                                                                                                      |         |                         |
| `--if-version N`    | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes label node_01k5w… --set rack=r12,fabric=ib-a --remove old
```

### nodes taint [#nodes-taint]

Add or remove a node’s taints (nodes.taint)

```sh
better-compute nodes taint NODE --tenant ID [--add TAINT,…] [--remove TAINT,…] [--if-version N] [--config PATH] [--url URL]
```

nodes taint adds the taints in --add and removes those in --remove (comma-separated). A taint joins the node’s NoSchedule reasons as taint:\<taint>, so the scheduler places nothing new there until it is removed. Decided as compute:nodes:update.

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--add TAINT,…`    | Taints to add                                                                                                                                             |         |                         |
| `--remove TAINT,…` | Taints to remove                                                                                                                                          |         |                         |
| `--if-version N`   | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes taint node_01k5w… --add gpu=degraded:NoSchedule
```

### nodes delete [#nodes-delete]

Remove a retired or cordoned adopted node for good (nodes.delete)

```sh
better-compute nodes delete NODE --tenant ID [--reason TEXT] [--if-version N] [--yes] [--config PATH] [--url URL]
```

nodes delete tombstones a node that is retired, a cordoned adopted node (cordon it first), or one that never got past adoption or enrollment, once it holds no allocation. Its certificates are revoked and its hardware key and attestor bindings (e.g. a TPM) are freed, so the machine can join again as a new node: after a TPM swap, cordon, delete, then run `bcd join --reset --yes` and `bcd join` on the machine. It cannot be undone, so it needs --yes. Decided as compute:nodes:delete.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Why (recorded in the node’s events)                                                                                                                       |         |                         |
| `--if-version N`  | Only if the node is still at this version (nodes.get’s version)                                                                                           |         |                         |
| `--yes`           | Confirm: the node is removed for good                                                                                                                     |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute nodes delete node_01k5w… --reason "TPM replaced" --yes
```

### join-tokens create [#join-tokens-create]

Create a join token for a pool: what bcd join presents (joinTokens.create)

```sh
better-compute join-tokens create --tenant ID --pool ID --ttl DURATION [--max-uses N] [--allow-reclaim] [--hosted WHERE] [--expect-product-uuid UUID] [--expect-serial SERIAL] [--expect-mac MAC] [--expect-gpu-uuid GPU-…] [--out PATH] [--key KEY] [--config PATH] [--url URL]
```

join-tokens create mints a token that lets up to --max-uses machines (default 1) join --pool for --ttl (60s to 24h). --expect-\* bind it to one machine's identity (product UUID, serial, MAC or GPU UUID). --hosted kubernetes marks it for bcd as a DaemonSet (deploy/kubernetes): the cluster's admins have root on its machines, so they join tenant-controlled unless the pool's policy.kubernetes.platformOperated is true. The token is printed once and never again: with --out it goes to a new file (0600, never overwritten) instead of the output, to hand to the server out of band (the install script reads BCD\_JOIN\_TOKEN or stdin). Decided as compute:join-tokens:create.

| Flag                         | Description                                                                                                                                               | Default | Environment             |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`                | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`                  | The pool machines join                                                                                                                                    |         |                         |
| `--ttl DURATION`             | How long it is valid: 30m, 2h (at most 24h)                                                                                                               |         |                         |
| `--max-uses N`               | Machines it admits (default 1)                                                                                                                            |         |                         |
| `--allow-reclaim`            | May re-admit a machine the pool already knew                                                                                                              |         |                         |
| `--hosted WHERE`             | For bcd as a DaemonSet on a Kubernetes cluster: its nodes join tenant-controlled unless the pool says the platform operates the cluster                   |         |                         |
| `--expect-product-uuid UUID` | Only the machine with this SMBIOS product UUID                                                                                                            |         |                         |
| `--expect-serial SERIAL`     | Only the machine with this serial number                                                                                                                  |         |                         |
| `--expect-mac MAC`           | Only the machine with this MAC address                                                                                                                    |         |                         |
| `--expect-gpu-uuid GPU-…`    | Only the machine with this GPU                                                                                                                            |         |                         |
| `--out PATH`                 | Write the token to this new file instead of printing it                                                                                                   |         |                         |
| `--key KEY`                  | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`              | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`                  | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`            | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`               | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-tokens create --pool h100-eu --ttl 2h --max-uses 8 --out ./join-token
better-compute join-tokens create --pool h100-eu --ttl 30m --expect-serial SN12345
better-compute join-tokens create --pool k8s-gpu --ttl 2h --max-uses 16 --hosted kubernetes --out ./join-token
```

### join-tokens list [#join-tokens-list]

List a pool’s join tokens, never their secrets (joinTokens.list)

```sh
better-compute join-tokens list --tenant ID --pool ID [--all] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List a pool’s join tokens, never their secrets (joinTokens.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool                                                                                                                                                  |         |                         |
| `--all`           | Also revoked, expired and used-up tokens                                                                                                                  |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-tokens list --pool h100-eu
```

### join-tokens revoke [#join-tokens-revoke]

Revoke a join token before it expires (joinTokens.revoke)

```sh
better-compute join-tokens revoke TOKEN --tenant ID --pool ID [--config PATH] [--url URL]
```

Revoke a join token before it expires (joinTokens.revoke)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool                                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-tokens revoke jt_01k5w… --pool h100-eu
```

### join-requests list [#join-requests-list]

List a pool’s join requests, pending by default (joinRequests.list)

```sh
better-compute join-requests list --tenant ID --pool ID [--state STATE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List a pool’s join requests, pending by default (joinRequests.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool                                                                                                                                                  |         |                         |
| `--state STATE`   | Which requests (default pending)                                                                                                                          |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-requests list --pool h100-eu
```

### join-requests approve [#join-requests-approve]

Let a machine that asked join the pool (joinRequests.approve)

```sh
better-compute join-requests approve REQUEST --tenant ID --pool ID [--reason TEXT] [--ttl SECONDS] [--config PATH] [--url URL]
```

join-requests approve admits the machine behind the request into --pool; the approval waits --ttl seconds (default 3600) for the machine to retry. Check its claims (join-requests list) first: approving is what makes an unknown machine a node. Decided as compute:join-tokens:create.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool                                                                                                                                                  |         |                         |
| `--reason TEXT`   | Recorded on the request                                                                                                                                   |         |                         |
| `--ttl SECONDS`   | How long it may join                                                                                                                                      |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-requests approve jreq_01k5w… --pool h100-eu
```

### join-requests reject [#join-requests-reject]

Turn a machine’s join request down (joinRequests.reject)

```sh
better-compute join-requests reject REQUEST --tenant ID --pool ID [--reason TEXT] [--config PATH] [--url URL]
```

Turn a machine’s join request down (joinRequests.reject)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool                                                                                                                                                  |         |                         |
| `--reason TEXT`   | Recorded on the request                                                                                                                                   |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute join-requests reject jreq_01k5w… --pool h100-eu
```

### inventory get [#inventory-get]

Show a node’s current hardware inventory (inventory.get)

```sh
better-compute inventory get NODE --tenant ID [--config PATH] [--url URL]
```

Show a node’s current hardware inventory (inventory.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute inventory get node_01k5w… --query gpus
```

### inventory history [#inventory-history]

List a node’s inventory snapshots, newest first (inventory.history)

```sh
better-compute inventory history NODE --tenant ID [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List a node’s inventory snapshots, newest first (inventory.history)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--limit N`       | Page size (at most 100)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page (inv\_…)                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute inventory history node_01k5w…
```

### inventory diff [#inventory-diff]

Show what changed in a node’s hardware between two snapshots (inventory.diff)

```sh
better-compute inventory diff NODE --tenant ID [--from SNAPSHOT] [--to SNAPSHOT] [--config PATH] [--url URL]
```

inventory diff compares --from and --to (snapshot ids from inventory history, or inventory digests). Without --to it is the latest snapshot, without --from the one before --to.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--from SNAPSHOT` | The older snapshot (inv\_…) or digest                                                                                                                     |         |                         |
| `--to SNAPSHOT`   | The newer snapshot (inv\_…) or digest                                                                                                                     |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute inventory diff node_01k5w…
```

### site-tokens create [#site-tokens-create]

Create a token a site agent joins a site with (siteTokens.create)

```sh
better-compute site-tokens create --tenant ID --site ID --ttl SECONDS [--max-uses N] [--out PATH] [--key KEY] [--config PATH] [--url URL]
```

site-tokens create mints a token for up to --max-uses site agents (default 1) to join --site within --ttl seconds. It is printed once: with --out it goes to a new 0600 file instead. Decided as compute:sites:manage.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--site ID`       | The site (site\_…)                                                                                                                                        |         |                         |
| `--ttl SECONDS`   | How long it is valid                                                                                                                                      |         |                         |
| `--max-uses N`    | Site agents it admits (default 1)                                                                                                                         |         |                         |
| `--out PATH`      | Write the token to this new file instead of printing it                                                                                                   |         |                         |
| `--key KEY`       | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute site-tokens create --site site_01k5w… --ttl 3600 --out ./site-token
```

### site-tokens revoke [#site-tokens-revoke]

Revoke a site token (siteTokens.revoke)

```sh
better-compute site-tokens revoke TOKEN --tenant ID --site ID [--config PATH] [--url URL]
```

Revoke a site token (siteTokens.revoke)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--site ID`       | The site                                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute site-tokens revoke jt_01k5w… --site site_01k5w…
```

## Instances [#instances]

### instances list [#instances-list]

List instances: yours, or the tenant’s with --all (instances.list)

```sh
better-compute instances list --tenant ID [--all] [--state STATE] [--pool ID] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List instances: yours, or the tenant’s with --all (instances.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--all`           | Every instance of the tenant, not only yours                                                                                                              |         |                         |
| `--state STATE`   | Only instances in this state                                                                                                                              |         |                         |
| `--pool ID`       | Only this pool’s                                                                                                                                          |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances list
better-compute instances list --all --state running
```

### instances get [#instances-get]

Show one instance (instances.get)

```sh
better-compute instances get INSTANCE --tenant ID [--config PATH] [--url URL]
```

Show one instance (instances.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances get inst_01k5w…
```

### instances create [#instances-create]

Create an instance in a pool (instances.create)

```sh
better-compute instances create --tenant ID --pool ID --flavor NAME --image ID [--project ID] [--ssh-key PATH,…] [--user-data FILE|secret://…] [--stopped] [--wait SECONDS] [--key KEY] [--config PATH] [--url URL]
```

instances create boots --image with --flavor in --pool, decided as compute:instances:create. --ssh-key reads OpenSSH public keys from files (comma-separated paths; every non-empty line is a key), --user-data cloud-init from a file, or a secret://\<tenant>/\<id> reference. --stopped creates it without starting it. It waits up to --wait seconds (default 30) for the instance to settle. Every run sends a new idempotency key unless --key is given: repeat with the same --key to retry safely.

\| Flag               | Description                                                                                                   | Default                               | Environment             |
\| ------------------ | ------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ----------------------- | --- |
\| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                 |                                       | `BETTER_COMPUTE_TENANT` |
\| `--pool ID`        | The pool                                                                                                      |                                       |                         |
\| `--flavor NAME`    | The flavor (the pool offers it)                                                                               |                                       |                         |
\| `--image ID`       | The image to boot                                                                                             |                                       |                         |
\| `--project ID`     | The project it belongs to                                                                                     |                                       |                         |
\| `--ssh-key PATH,…` | Files of OpenSSH public keys to authorize                                                                     |                                       |                         |
\| `--user-data FILE  | secret://…`                                                                                                   | cloud-init user data (at most 32 KiB) |                         |     |
\| `--stopped`        | Create it stopped                                                                                             |                                       |                         |
\| `--wait SECONDS`   | Wait this long for it to settle                                                                               | `30`                                  |                         |
\| `--key KEY`        | Idempotency key, to retry the same request                                                                    |                                       |                         |
\| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                            |                                       | `BETTER_COMPUTE_CONFIG` |
\| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |                                       | `BETTER_COMPUTE_URL`    |
\| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                              |                                       |                         |
\| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                      |                                       |                         |

```sh
better-compute instances create --pool vps-eu --flavor g1.small --image img_01k5w… --ssh-key ~/.ssh/id_ed25519.pub
```

### instances start [#instances-start]

Start a stopped instance (instances.start)

```sh
better-compute instances start INSTANCE --tenant ID [--config PATH] [--url URL]
```

Start a stopped instance (instances.start)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances start inst_01k5w…
```

### instances stop [#instances-stop]

Stop a running instance (instances.stop)

```sh
better-compute instances stop INSTANCE --tenant ID [--config PATH] [--url URL]
```

Stop a running instance (instances.stop)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances stop inst_01k5w…
```

### instances delete [#instances-delete]

Delete an instance and its disks (instances.delete)

```sh
better-compute instances delete INSTANCE --tenant ID [--config PATH] [--url URL]
```

Delete an instance and its disks (instances.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances delete inst_01k5w…
```

### instances resize [#instances-resize]

Move an instance to another flavor (instances.resize)

```sh
better-compute instances resize INSTANCE --tenant ID --flavor NAME [--config PATH] [--url URL]
```

Move an instance to another flavor (instances.resize)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--flavor NAME`   | The new flavor                                                                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute instances resize inst_01k5w… --flavor g1.large
```

## Events [#events]

### events list [#events-list]

List the tenant’s events, newest first (events.list)

```sh
better-compute events list --tenant ID [--type TYPE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List the tenant’s events, newest first (events.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--type TYPE`     | Only events of this type (node.cordoned, reservation.placed, …)                                                                                           |         |                         |
| `--limit N`       | Page size (at most 500)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute events list --type node.draining
better-compute events list --query "items[*].type"
```

### events watch [#events-watch]

Follow the tenant’s events as they happen, one JSON line each (GET /watch)

```sh
better-compute events watch --tenant ID [--types PATTERN,…] [--since CURSOR] [--count N] [--path PATH] [--config PATH] [--url URL]
```

events watch follows the host's server-sent events route (--path, default /watch; the console app serves /api/compute/watch) as BETTER\_COMPUTE\_TOKEN, so it needs --url. Each event is printed as one line of JSON with its `sequence`, the cursor: pass the last one to --since to resume where a previous run stopped. It reconnects after a broken connection, resuming from the last event, until Ctrl-C, --count events, or a refusal: a revoked session or a withdrawn compute:events:read ends it.

| Flag                | Description                                                                                                                                               | Default  | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |          | `BETTER_COMPUTE_TENANT` |
| `--types PATTERN,…` | Event type patterns, comma-separated (node.\*, reservation.placed); default all                                                                           |          |                         |
| `--since CURSOR`    | Resume after this event (a printed sequence)                                                                                                              |          |                         |
| `--count N`         | Stop after N events                                                                                                                                       |          |                         |
| `--path PATH`       | The watch route on the server                                                                                                                             | `/watch` |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |          | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |          | `BETTER_COMPUTE_URL`    |

```sh
better-compute events watch --url https://compute.example.com --types "node.*,reservation.*"
better-compute events watch --url https://compute.example.com --since 184467.12 --count 100
```

## Operations [#operations]

### releases register [#releases-register]

Register a signed bcd release manifest for rollouts (releases.register)

```sh
better-compute releases register MANIFEST --tenant ID [--config PATH] [--url URL]
```

releases register sends a published manifest.json (\{ manifest, signatures }) as it is; the server checks the signatures against its release keys. Decided as compute:fleet:admin.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute releases register ./dist/manifest.json
```

### releases list [#releases-list]

List the registered bcd releases (releases.list)

```sh
better-compute releases list --tenant ID [--config PATH] [--url URL]
```

List the registered bcd releases (releases.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute releases list
```

### rollouts create [#rollouts-create]

Roll a registered bcd release out in waves (rollouts.create)

```sh
better-compute rollouts create --tenant ID --version SEMVER [--pools ID,…] [--waves PCT,…] [--max-failure-ratio RATIO] [--soak DURATION] [--upgrade-timeout DURATION] [--key KEY] [--config PATH] [--url URL]
```

rollouts create upgrades the nodes of --pools (default: every pool the tenant owns) to --version in cumulative --waves of percentages (default 1,10,50,100). Each wave soaks --soak (default 10m) before the next, a node has --upgrade-timeout (default 15m) to come back on the new version, and more than --max-failure-ratio failed nodes (default 0.05) pause the rollout. Decided as compute:fleet:admin.

| Flag                         | Description                                                                                                                                               | Default | Environment             |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`                | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--version SEMVER`           | The registered release                                                                                                                                    |         |                         |
| `--pools ID,…`               | The pools to upgrade (pool\_…)                                                                                                                            |         |                         |
| `--waves PCT,…`              | Cumulative percentages ending at 100                                                                                                                      |         |                         |
| `--max-failure-ratio RATIO`  | The failed share that pauses it (0–1)                                                                                                                     |         |                         |
| `--soak DURATION`            | How long each wave soaks                                                                                                                                  |         |                         |
| `--upgrade-timeout DURATION` | How long a node has to come back                                                                                                                          |         |                         |
| `--key KEY`                  | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`              | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`                  | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`            | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`               | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts create --version 0.9.1 --waves 5,25,100 --soak 30m
```

### rollouts list [#rollouts-list]

List rollouts (rollouts.list)

```sh
better-compute rollouts list --tenant ID [--config PATH] [--url URL]
```

List rollouts (rollouts.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts list
```

### rollouts get [#rollouts-get]

Show a rollout: its waves and per-node progress (rollouts.get)

```sh
better-compute rollouts get ROLLOUT --tenant ID [--config PATH] [--url URL]
```

Show a rollout: its waves and per-node progress (rollouts.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts get rollout_01k5w…
```

### rollouts pause [#rollouts-pause]

Pause a rollout before its next wave (rollouts.pause)

```sh
better-compute rollouts pause ROLLOUT --tenant ID [--reason TEXT] [--config PATH] [--url URL]
```

Pause a rollout before its next wave (rollouts.pause)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Recorded on the rollout                                                                                                                                   |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts pause rollout_01k5w… --reason "XIDs after wave 2"
```

### rollouts resume [#rollouts-resume]

Resume a paused rollout (rollouts.resume)

```sh
better-compute rollouts resume ROLLOUT --tenant ID [--config PATH] [--url URL]
```

Resume a paused rollout (rollouts.resume)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts resume rollout_01k5w…
```

### rollouts abort [#rollouts-abort]

Stop a rollout for good, optionally rolling nodes back (rollouts.abort)

```sh
better-compute rollouts abort ROLLOUT --tenant ID [--reason TEXT] [--rollback] [--config PATH] [--url URL]
```

rollouts abort ends the rollout: no further node is upgraded. --rollback also swaps back the nodes whose upgrade is not confirmed yet. A node that confirmed the new release keeps it (it never goes below its release floor), so a bad release is undone by rolling a newer one forward.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Recorded on the rollout                                                                                                                                   |         |                         |
| `--rollback`      | Also swap back upgrades not confirmed yet                                                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute rollouts abort rollout_01k5w… --rollback --reason "boot loop on H100 nodes"
```

### workflows list [#workflows-list]

List a node’s workflows, newest first (workflows.list)

```sh
better-compute workflows list --tenant ID --node ID [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List a node’s workflows, newest first (workflows.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--limit N`       | Page size (at most 100)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows list --node node_01k5w…
```

### workflows get [#workflows-get]

Show a workflow and its steps (workflows.get)

```sh
better-compute workflows get WORKFLOW --tenant ID --node ID [--config PATH] [--url URL]
```

Show a workflow and its steps (workflows.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows get wf_01k5w… --node node_01k5w… --query steps
```

### workflows abort [#workflows-abort]

Abort a workflow: pending steps are cancelled, its cleanup still runs (workflows.abort)

```sh
better-compute workflows abort WORKFLOW --tenant ID --node ID [--reason TEXT] [--config PATH] [--url URL]
```

Abort a workflow: pending steps are cancelled, its cleanup still runs (workflows.abort)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--reason TEXT`   | Recorded on the workflow                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows abort wf_01k5w… --node node_01k5w…
```

### workflows pause [#workflows-pause]

Pause a workflow: the step in flight finishes, no further one starts (workflows.pause)

```sh
better-compute workflows pause WORKFLOW --tenant ID --node ID [--config PATH] [--url URL]
```

Pause a workflow: the step in flight finishes, no further one starts (workflows.pause)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows pause wf_01k5w… --node node_01k5w…
```

### workflows resume [#workflows-resume]

Resume a paused workflow (workflows.resume)

```sh
better-compute workflows resume WORKFLOW --tenant ID --node ID [--config PATH] [--url URL]
```

Resume a paused workflow (workflows.resume)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows resume wf_01k5w… --node node_01k5w…
```

### workflows retry [#workflows-retry]

Retry a failed workflow, from a step (workflows.retry)

```sh
better-compute workflows retry WORKFLOW --tenant ID --node ID [--from-step STEP] [--config PATH] [--url URL]
```

Retry a failed workflow, from a step (workflows.retry)

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`        | The node (node\_…)                                                                                                                                        |         |                         |
| `--from-step STEP` | Start here (default: the first main step that did not succeed)                                                                                            |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute workflows retry wf_01k5w… --node node_01k5w…
```

### commands list [#commands-list]

List the commands queued for a node, newest first (commands.list)

```sh
better-compute commands list --tenant ID --node ID [--state STATE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List the commands queued for a node, newest first (commands.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--state STATE`   | Only commands in this state                                                                                                                               |         |                         |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute commands list --node node_01k5w… --state failed
```

### commands get [#commands-get]

Show one command and its result (commands.get)

```sh
better-compute commands get COMMAND --tenant ID --node ID [--config PATH] [--url URL]
```

Show one command and its result (commands.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute commands get cmd_01k5w… --node node_01k5w…
```

### commands cancel [#commands-cancel]

Cancel a command: a pending one never runs, a running one is asked to stop (commands.cancel)

```sh
better-compute commands cancel COMMAND --tenant ID --node ID [--config PATH] [--url URL]
```

Cancel a command: a pending one never runs, a running one is asked to stop (commands.cancel)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--node ID`       | The node (node\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute commands cancel cmd_01k5w… --node node_01k5w…
```

## Catalog [#catalog]

### sites list [#sites-list]

List sites (sites.list)

```sh
better-compute sites list --tenant ID [--kind KIND] [--region REGION] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List sites (sites.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--kind KIND`     | Only this kind                                                                                                                                            |         |                         |
| `--region REGION` | Only this region                                                                                                                                          |         |                         |
| `--limit N`       | Page size                                                                                                                                                 |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sites list --region eu-west
```

### sites get [#sites-get]

Show a site (sites.get)

```sh
better-compute sites get SITE --tenant ID [--config PATH] [--url URL]
```

Show a site (sites.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sites get site_01k5w…
```

### sites create [#sites-create]

Create a site (sites.create)

```sh
better-compute sites create --tenant ID --kind KIND --region REGION [--iam-region REGION] [--key KEY] [--config PATH] [--url URL]
```

Create a site (sites.create)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--kind KIND`         | The site kind                                                                                                                                             |         |                         |
| `--region REGION`     | Its region, e.g. eu-west                                                                                                                                  |         |                         |
| `--iam-region REGION` | The IAM region it answers to                                                                                                                              |         |                         |
| `--key KEY`           | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sites create --kind dedicated --region eu-west
```

### sites update [#sites-update]

Change a site’s region or IAM region (sites.update)

```sh
better-compute sites update SITE --tenant ID [--region REGION] [--iam-region REGION] [--clear-iam-region] [--if-version N] [--config PATH] [--url URL]
```

Change a site’s region or IAM region (sites.update)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--region REGION`     | The new region                                                                                                                                            |         |                         |
| `--iam-region REGION` | The new IAM region                                                                                                                                        |         |                         |
| `--clear-iam-region`  | Remove the IAM region                                                                                                                                     |         |                         |
| `--if-version N`      | Only if the site is still at this version                                                                                                                 |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sites update site_01k5w… --iam-region eu
```

### sites delete [#sites-delete]

Delete a site (sites.delete)

```sh
better-compute sites delete SITE --tenant ID [--if-version N] [--config PATH] [--url URL]
```

Delete a site (sites.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--if-version N`  | Only if the site is still at this version                                                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute sites delete site_01k5w…
```

### projects list [#projects-list]

List the tenant’s projects (projects.list)

```sh
better-compute projects list --tenant ID [--state STATE] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List the tenant’s projects (projects.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--state STATE`   | Only this state                                                                                                                                           |         |                         |
| `--limit N`       | Page size                                                                                                                                                 |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute projects list
```

### projects get [#projects-get]

Show a project (projects.get)

```sh
better-compute projects get PROJECT --tenant ID [--config PATH] [--url URL]
```

Show a project (projects.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute projects get prj_01k5w…
```

### projects create [#projects-create]

Create a project (projects.create)

```sh
better-compute projects create --tenant ID --name NAME [--key KEY] [--config PATH] [--url URL]
```

Create a project (projects.create)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--name NAME`     | Lowercase letters, digits and dashes                                                                                                                      |         |                         |
| `--key KEY`       | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute projects create --name training
```

### projects archive [#projects-archive]

Archive a project (projects.archive)

```sh
better-compute projects archive PROJECT --tenant ID [--config PATH] [--url URL]
```

Archive a project (projects.archive)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute projects archive prj_01k5w…
```

### images list [#images-list]

List catalog images (images.list)

```sh
better-compute images list --tenant ID [--kind KIND] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List catalog images (images.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--kind KIND`     | Only this kind                                                                                                                                            |         |                         |
| `--limit N`       | Page size                                                                                                                                                 |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute images list --kind os
```

### images get [#images-get]

Show a catalog image (images.get)

```sh
better-compute images get IMAGE --tenant ID [--config PATH] [--url URL]
```

Show a catalog image (images.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute images get img_01k5w…
```

### images register [#images-register]

Register an image in the catalog by digest (images.register)

```sh
better-compute images register --tenant ID --ref NAME@VERSION --source URL --disk-format FORMAT --sha256 HEX [--kind KIND] [--compression ALG] [--size BYTES] [--signature BASE64] [--signer ID] [--config PATH] [--url URL]
```

images register records name\@version (--ref), where it is fetched from (--source, https\:// or oci://), its format and --sha256, and optionally a signature over its identity checked with a registered signer. Nodes verify the digest of what they fetch. Decided as compute:images:publish.

| Flag                   | Description                                                                                                                                               | Default | Environment             |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`          | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--ref NAME@VERSION`   | e.g. ubuntu-24.04\@2026.09                                                                                                                                |         |                         |
| `--source URL`         | Where nodes fetch it: https\:// or oci://                                                                                                                 |         |                         |
| `--disk-format FORMAT` | The disk format                                                                                                                                           |         |                         |
| `--sha256 HEX`         | The image digest, 64 hex digits                                                                                                                           |         |                         |
| `--kind KIND`          | What it boots as (default os)                                                                                                                             |         |                         |
| `--compression ALG`    | Default none                                                                                                                                              |         |                         |
| `--size BYTES`         | Its size in bytes                                                                                                                                         |         |                         |
| `--signature BASE64`   | A signature over its identity                                                                                                                             |         |                         |
| `--signer ID`          | The registered signer that checks --signature                                                                                                             |         |                         |
| `--config PATH`        | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`            | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`      | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`         | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute images register --ref ubuntu-24.04@2026.09 --source https://images.example.com/u2404.qcow2 --disk-format qcow2 --sha256 3f…
```

### images promote [#images-promote]

Promote a catalog image (images.promote)

```sh
better-compute images promote IMAGE --tenant ID [--config PATH] [--url URL]
```

Promote a catalog image (images.promote)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute images promote img_01k5w…
```

### flavors list [#flavors-list]

List a pool’s instance flavors (flavors.list)

```sh
better-compute flavors list --tenant ID --pool ID [--config PATH] [--url URL]
```

List a pool’s instance flavors (flavors.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute flavors list --pool pool_01k5w…
```

### flavors create [#flavors-create]

Define an instance flavor in a pool (flavors.create)

```sh
better-compute flavors create --tenant ID --pool ID --name NAME --vcpu N --memory-gib GIB --disk-gib GIB [--gpus N] [--offer ID] [--config PATH] [--url URL]
```

Define an instance flavor in a pool (flavors.create)

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`        | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--name NAME`      | The flavor name, e.g. g1.small                                                                                                                            |         |                         |
| `--vcpu N`         | Virtual CPUs                                                                                                                                              |         |                         |
| `--memory-gib GIB` | Memory in GiB                                                                                                                                             |         |                         |
| `--disk-gib GIB`   | Root disk in GiB                                                                                                                                          |         |                         |
| `--gpus N`         | GPUs passed through (default 0)                                                                                                                           |         |                         |
| `--offer ID`       | Provider pools: the provider offer it creates                                                                                                             |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute flavors create --pool pool_01k5w… --name g1.small --vcpu 4 --memory-gib 16 --disk-gib 80
```

### flavors update [#flavors-update]

Redefine an instance flavor; running instances keep theirs (flavors.update)

```sh
better-compute flavors update --tenant ID --pool ID --name NAME --vcpu N --memory-gib GIB --disk-gib GIB [--gpus N] [--offer ID] [--config PATH] [--url URL]
```

Redefine an instance flavor; running instances keep theirs (flavors.update)

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`        | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--name NAME`      | The flavor name, e.g. g1.small                                                                                                                            |         |                         |
| `--vcpu N`         | Virtual CPUs                                                                                                                                              |         |                         |
| `--memory-gib GIB` | Memory in GiB                                                                                                                                             |         |                         |
| `--disk-gib GIB`   | Root disk in GiB                                                                                                                                          |         |                         |
| `--gpus N`         | GPUs passed through (default 0)                                                                                                                           |         |                         |
| `--offer ID`       | Provider pools: the provider offer it creates                                                                                                             |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute flavors update --pool pool_01k5w… --name g1.small --vcpu 4 --memory-gib 32 --disk-gib 80
```

### flavors delete [#flavors-delete]

Remove an instance flavor from a pool (flavors.delete)

```sh
better-compute flavors delete --tenant ID --pool ID --name NAME [--config PATH] [--url URL]
```

Remove an instance flavor from a pool (flavors.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--name NAME`     | The flavor name, e.g. g1.small                                                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute flavors delete --pool pool_01k5w… --name g1.small
```

### quotas get [#quotas-get]

Show the tenant’s quotas and what is used, per pool (quotas.get)

```sh
better-compute quotas get --tenant ID [--pool ID] [--config PATH] [--url URL]
```

Show the tenant’s quotas and what is used, per pool (quotas.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | Only this pool (pool\_…)                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute quotas get --pool pool_01k5w…
```

### quotas set [#quotas-set]

Set one quota of the tenant in a pool (quotas.set)

```sh
better-compute quotas set --tenant ID --pool ID --resource NAME --max N [--config PATH] [--url URL]
```

Set one quota of the tenant in a pool (quotas.set)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The pool (pool\_…)                                                                                                                                        |         |                         |
| `--resource NAME` | The quota resource, e.g. gpus                                                                                                                             |         |                         |
| `--max N`         | The new limit                                                                                                                                             |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute quotas set --pool pool_01k5w… --resource gpus --max 64
```

## Usage [#usage]

### usage list [#usage-list]

List hourly usage windows (usage.list)

```sh
better-compute usage list --tenant ID [--project ID] [--meter METER] [--from TIME] [--to TIME] [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List hourly usage windows (usage.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--project ID`    | Only this project’s (prj\_…)                                                                                                                              |         |                         |
| `--meter METER`   | Only this meter (gpu.hours, …)                                                                                                                            |         |                         |
| `--from TIME`     | Windows ending after: ISO 8601, epoch ms, or -7d                                                                                                          |         |                         |
| `--to TIME`       | Windows starting before                                                                                                                                   |         |                         |
| `--limit N`       | Page size                                                                                                                                                 |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute usage list --from -7d --meter gpu.hours
```

### usage export [#usage-export]

Export usage for billing as CSV or JSON lines (usage.export)

```sh
better-compute usage export --tenant ID --from TIME --to TIME [--project ID] [--meter METER] [--export-format FORMAT] [--out PATH] [--config PATH] [--url URL]
```

usage export returns every hourly window between --from and --to (at most 93 days apart) and the total per meter. With --out the CSV (or JSON lines) goes to a new file, never over one, and the output keeps the count, the totals and whether the export was truncated (too many windows: narrow the range or the meter).

| Flag                     | Description                                                                                                                                               | Default | Environment             |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`            | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--from TIME`            | ISO 8601, epoch ms, or -30d                                                                                                                               |         |                         |
| `--to TIME`              | ISO 8601, epoch ms, -1d or +0s (now)                                                                                                                      |         |                         |
| `--project ID`           | Only this project’s (prj\_…)                                                                                                                              |         |                         |
| `--meter METER`          | Only this meter                                                                                                                                           |         |                         |
| `--export-format FORMAT` | Default csv                                                                                                                                               |         |                         |
| `--out PATH`             | Write the export to this new file                                                                                                                         |         |                         |
| `--config PATH`          | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`              | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`        | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`           | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute usage export --from 2026-09-01 --to 2026-10-01 --out september.csv
```

## Webhooks [#webhooks]

### webhooks list [#webhooks-list]

List the tenant’s webhooks (webhooks.list)

```sh
better-compute webhooks list --tenant ID [--config PATH] [--url URL]
```

List the tenant’s webhooks (webhooks.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks list
```

### webhooks get [#webhooks-get]

Show a webhook (webhooks.get)

```sh
better-compute webhooks get WEBHOOK --tenant ID [--config PATH] [--url URL]
```

Show a webhook (webhooks.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks get whk_01k5w…
```

### webhooks create [#webhooks-create]

Send the tenant’s events to a URL, signed (webhooks.create)

```sh
better-compute webhooks create --tenant ID --endpoint URL --events PATTERN,… [--description TEXT] [--secret-out PATH] [--config PATH] [--url URL]
```

webhooks create posts every event matching --events (type patterns: node.\_, reservation.placed, \_) to --endpoint, signed with X-Better-Compute-Signature. The signing secret is shown once: with --secret-out it goes to a new 0600 file instead of the output.

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--endpoint URL`     | Where events are posted (https)                                                                                                                           |         |                         |
| `--events PATTERN,…` | Event type patterns                                                                                                                                       |         |                         |
| `--description TEXT` | What it is for                                                                                                                                            |         |                         |
| `--secret-out PATH`  | Write the signing secret to this new file instead of printing it                                                                                          |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks create --endpoint https://hooks.example.com/compute --events "node.*,reservation.*" --secret-out ./whk.secret
```

### webhooks update [#webhooks-update]

Change a webhook’s URL, events, description, or pause it (webhooks.update)

```sh
better-compute webhooks update WEBHOOK --tenant ID [--endpoint URL] [--events PATTERN,…] [--description TEXT] [--pause] [--resume] [--config PATH] [--url URL]
```

Change a webhook’s URL, events, description, or pause it (webhooks.update)

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--endpoint URL`     | The new URL events are posted to                                                                                                                          |         |                         |
| `--events PATTERN,…` | The new event type patterns                                                                                                                               |         |                         |
| `--description TEXT` | The new description                                                                                                                                       |         |                         |
| `--pause`            | Stop delivering (active = false)                                                                                                                          |         |                         |
| `--resume`           | Deliver again (active = true)                                                                                                                             |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks update whk_01k5w… --pause
```

### webhooks delete [#webhooks-delete]

Delete a webhook (webhooks.delete)

```sh
better-compute webhooks delete WEBHOOK --tenant ID [--config PATH] [--url URL]
```

Delete a webhook (webhooks.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks delete whk_01k5w…
```

### webhooks ping [#webhooks-ping]

Send a webhook a test delivery (webhooks.ping)

```sh
better-compute webhooks ping WEBHOOK --tenant ID [--config PATH] [--url URL]
```

Send a webhook a test delivery (webhooks.ping)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks ping whk_01k5w…
```

### webhooks rotate-secret [#webhooks-rotate-secret]

Replace a webhook’s signing secret (webhooks.rotateSecret)

```sh
better-compute webhooks rotate-secret WEBHOOK --tenant ID [--keep-previous DURATION] [--secret-out PATH] [--config PATH] [--url URL]
```

webhooks rotate-secret issues a new signing secret, shown once (or written to --secret-out). --keep-previous signs with the old one too for that long (at most 24h), so the receiver can switch without dropping deliveries.

| Flag                       | Description                                                                                                                                               | Default | Environment             |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`              | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--keep-previous DURATION` | Keep signing with the old secret this long                                                                                                                |         |                         |
| `--secret-out PATH`        | Write the signing secret to this new file instead of printing it                                                                                          |         |                         |
| `--config PATH`            | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`                | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`          | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`             | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks rotate-secret whk_01k5w… --keep-previous 1h --secret-out ./whk.secret.new
```

### webhooks deliveries [#webhooks-deliveries]

List a webhook’s deliveries (webhooks.deliveries)

```sh
better-compute webhooks deliveries WEBHOOK --tenant ID [--state STATE] [--limit N] [--config PATH] [--url URL]
```

List a webhook’s deliveries (webhooks.deliveries)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--state STATE`   | Only deliveries in this state                                                                                                                             |         |                         |
| `--limit N`       | At most this many                                                                                                                                         |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks deliveries whk_01k5w… --state dead
```

### webhooks redeliver [#webhooks-redeliver]

Deliver one event again (webhooks.redeliver)

```sh
better-compute webhooks redeliver DELIVERY --tenant ID [--config PATH] [--url URL]
```

Deliver one event again (webhooks.redeliver)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks redeliver dlv_01k5w…
```

### webhooks event-types [#webhooks-event-types]

List the event types a webhook can subscribe to (webhooks.eventTypes)

```sh
better-compute webhooks event-types --tenant ID [--config PATH] [--url URL]
```

List the event types a webhook can subscribe to (webhooks.eventTypes)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute webhooks event-types
```

## Pools [#pools]

### pools get [#pools-get]

Show a pool: its class, policy and offers (pools.get)

```sh
better-compute pools get POOL --tenant ID [--config PATH] [--url URL]
```

Show a pool: its class, policy and offers (pools.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools get pool_01k5w…
```

### pools create [#pools-create]

Create a pool in a site (pools.create)

```sh
better-compute pools create --tenant ID --site ID --class CLASS [--gpu-model MODEL] [--sharing MODE] [--policy-file PATH] [--owner-quota RES=N,…] [--provider-account ID] [--key KEY] [--config PATH] [--url URL]
```

pools create makes a --class pool in --site, owned by the tenant. --policy-file is a PoolPolicy as JSON; --owner-quota sets the owner’s own limits (default: unlimited GPUs). Provider pools name their --provider-account and run no agent. Decided as compute:pools:create.

| Flag                    | Description                                                                                                                                               | Default | Environment             |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`           | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--site ID`             | The site (site\_…)                                                                                                                                        |         |                         |
| `--class CLASS`         | The pool class                                                                                                                                            |         |                         |
| `--gpu-model MODEL`     | The GPU model it holds (H100, …)                                                                                                                          |         |                         |
| `--sharing MODE`        | How GPUs are shared (default exclusive)                                                                                                                   |         |                         |
| `--policy-file PATH`    | The PoolPolicy, as JSON                                                                                                                                   |         |                         |
| `--owner-quota RES=N,…` | The owner’s limits, e.g. gpus=64                                                                                                                          |         |                         |
| `--provider-account ID` | Provider pools: the account (pacct\_…)                                                                                                                    |         |                         |
| `--key KEY`             | Idempotency key: a retry returns the same pool                                                                                                            |         |                         |
| `--config PATH`         | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`             | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools create --site site_01k5w… --class metal-host --gpu-model H100 --owner-quota gpus=64
```

### pools update [#pools-update]

Replace a pool’s policy or change its GPU model (pools.update)

```sh
better-compute pools update POOL --tenant ID [--policy-file PATH] [--gpu-model MODEL] [--config PATH] [--url URL]
```

Replace a pool’s policy or change its GPU model (pools.update)

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--policy-file PATH` | The new PoolPolicy, as JSON (replaces the old one)                                                                                                        |         |                         |
| `--gpu-model MODEL`  | The new GPU model                                                                                                                                         |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools update pool_01k5w… --policy-file policy.json
```

### pools archive [#pools-archive]

Archive a pool (pools.archive)

```sh
better-compute pools archive POOL --tenant ID [--config PATH] [--url URL]
```

Archive a pool (pools.archive)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools archive pool_01k5w…
```

### pools offer [#pools-offer]

Offer a pool to another tenant with a quota (pools.offer)

```sh
better-compute pools offer POOL --tenant ID --to TENANT --quota RES=N,… [--include-descendants] [--priority N] [--config PATH] [--url URL]
```

Offer a pool to another tenant with a quota (pools.offer)

| Flag                    | Description                                                                                                                                               | Default | Environment             |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`           | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--to TENANT`           | The tenant it is offered to                                                                                                                               |         |                         |
| `--quota RES=N,…`       | That tenant’s limits, e.g. gpus=8                                                                                                                         |         |                         |
| `--include-descendants` | Also that tenant’s child tenants                                                                                                                          |         |                         |
| `--priority N`          | Scheduling priority                                                                                                                                       |         |                         |
| `--config PATH`         | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`             | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools offer pool_01k5w… --to org_research --quota gpus=16
```

### pools unoffer [#pools-unoffer]

Withdraw a pool’s offer to a tenant (pools.unoffer)

```sh
better-compute pools unoffer POOL --tenant ID --to TENANT [--config PATH] [--url URL]
```

Withdraw a pool’s offer to a tenant (pools.unoffer)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--to TENANT`     | The tenant to withdraw it from                                                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute pools unoffer pool_01k5w… --to org_research
```

### drivers list [#drivers-list]

List the drivers this deployment runs (drivers.list)

```sh
better-compute drivers list --tenant ID [--kind KIND] [--config PATH] [--url URL]
```

List the drivers this deployment runs (drivers.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--kind KIND`     | Only this kind (runtime, bmc, …)                                                                                                                          |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute drivers list --kind runtime
```

## Secrets [#secrets]

### secrets put [#secrets-put]

Store a secret for secret:// references (secrets.put)

```sh
better-compute secrets put --tenant ID --name NAME [--value-file PATH] [--stdin] [--scope SCOPE] [--key KEY] [--config PATH] [--url URL]
```

secrets put stores the value of --value-file, or of standard input with --stdin, under --name; user data, sandbox environments and plugins name it as secret://\<tenant>/\<name>. The value never goes on the command line (shell history, the process list). --scope fleet stores a platform secret. Decided as compute:secrets:manage.

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--name NAME`       | The secret name                                                                                                                                           |         |                         |
| `--value-file PATH` | Read the value from this file                                                                                                                             |         |                         |
| `--stdin`           | Read the value from standard input                                                                                                                        |         |                         |
| `--scope SCOPE`     | Default tenant                                                                                                                                            |         |                         |
| `--key KEY`         | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute secrets put --name hf-token --value-file ./hf-token
vault read -field=token secret/hf | better-compute secrets put --name hf-token --stdin
```

### secrets delete [#secrets-delete]

Delete a secret (secrets.delete)

```sh
better-compute secrets delete --tenant ID --name NAME [--scope SCOPE] [--config PATH] [--url URL]
```

Delete a secret (secrets.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--name NAME`     | The secret name                                                                                                                                           |         |                         |
| `--scope SCOPE`   | Default tenant                                                                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute secrets delete --name hf-token
```

## Metal [#metal]

### metal enroll [#metal-enroll]

Enroll a machine by its BMC into a pool (metal.enroll)

```sh
better-compute metal enroll --tenant ID --site ID --pool ID --bmc-address HOST|URL --bmc-driver DRIVER [--bmc-credentials REF] [--bmc-username USER] [--bmc-password-file PATH] [--bmc-tls-spki HEX] [--macs MAC,…] [--product-uuid UUID] [--arch ARCH] [--legacy-bios] [--hardware-profile NAME] [--config PATH] [--url URL]
```

metal enroll creates a node in --pool of --site from its BMC (--bmc-address, --bmc-driver) and queues BMC verification, inspection and commissioning. The BMC credentials are a reference to a secret of the tenant itself (--bmc-credentials secret://\<tenant>/\<id>; vault://… and env://… only as the root tenant), or --bmc-username with --bmc-password-file, which the server stores in its secret broker at once and never echoes. --bmc-tls-spki pins the BMC certificate. Decided as compute:nodes:provision.

\| Flag                       | Description                                                                                                   | Default         | Environment             |
\| -------------------------- | ------------------------------------------------------------------------------------------------------------- | --------------- | ----------------------- | --- |
\| `--tenant ID`              | The tenant acting (the caller’s organization)                                                                 |                 | `BETTER_COMPUTE_TENANT` |
\| `--site ID`                | The site the machine is in                                                                                    |                 |                         |
\| `--pool ID`                | The pool it joins                                                                                             |                 |                         |
\| `--bmc-address HOST        | URL`                                                                                                          | The BMC address |                         |     |
\| `--bmc-driver DRIVER`      | The BMC driver, e.g. redfish                                                                                  |                 |                         |
\| `--bmc-credentials REF`    | secret://\<tenant>/\<id>, a secret of the node's tenant (vault:// or env:// as root only)                     |                 |                         |
\| `--bmc-username USER`      | With --bmc-password-file, instead of a reference                                                              |                 |                         |
\| `--bmc-password-file PATH` | A file holding the BMC password                                                                               |                 |                         |
\| `--bmc-tls-spki HEX`       | SHA-256 of the BMC certificate’s public key                                                                   |                 |                         |
\| `--macs MAC,…`             | Its MAC addresses, for PXE and matching                                                                       |                 |                         |
\| `--product-uuid UUID`      | Its SMBIOS product UUID                                                                                       |                 |                         |
\| `--arch ARCH`              | Default x86\_64                                                                                                |                 |                         |
\| `--legacy-bios`            | Boots in legacy BIOS mode, not UEFI                                                                           |                 |                         |
\| `--hardware-profile NAME`  | Overrides the pool’s hardware profile                                                                         |                 |                         |
\| `--config PATH`            | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                            |                 | `BETTER_COMPUTE_CONFIG` |
\| `--url URL`                | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |                 | `BETTER_COMPUTE_URL`    |
\| `--format FORMAT`          | Output as json (indented, default), compact (one line), or table                                              |                 |                         |
\| `--query PATH`             | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                      |                 |                         |

```sh
better-compute metal enroll --site site_01k5w… --pool pool_01k5w… --bmc-address 10.20.0.14 --bmc-driver redfish --bmc-credentials secret://org_acme/bmc-r12
```

### metal manage [#metal-manage]

Re-verify a ready node’s BMC access, then re-inspect and re-commission it (metal.manage)

```sh
better-compute metal manage NODE --tenant ID [--config PATH] [--url URL]
```

Re-verify a ready node’s BMC access, then re-inspect and re-commission it (metal.manage)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal manage node_01k5w…
```

### metal inspect [#metal-inspect]

Boot a ready node into the ramdisk to report its inventory (metal.inspect)

```sh
better-compute metal inspect NODE --tenant ID [--config PATH] [--url URL]
```

Boot a ready node into the ramdisk to report its inventory (metal.inspect)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal inspect node_01k5w…
```

### metal provide [#metal-provide]

Commission a ready node: diagnostics, a quick wipe, power off (metal.provide)

```sh
better-compute metal provide NODE --tenant ID [--config PATH] [--url URL]
```

Commission a ready node: diagnostics, a quick wipe, power off (metal.provide)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal provide node_01k5w…
```

### metal rescue [#metal-rescue]

Boot a deployed node into the rescue ramdisk (metal.rescue)

```sh
better-compute metal rescue NODE --tenant ID [--config PATH] [--url URL]
```

Boot a deployed node into the rescue ramdisk (metal.rescue)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal rescue node_01k5w…
```

### metal unrescue [#metal-unrescue]

Boot a rescued node back into its installed OS (metal.unrescue)

```sh
better-compute metal unrescue NODE --tenant ID [--config PATH] [--url URL]
```

Boot a rescued node back into its installed OS (metal.unrescue)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal unrescue node_01k5w…
```

### metal recover [#metal-recover]

Move a failed node back to the last stable point for the state it failed in (metal.recover)

```sh
better-compute metal recover NODE --tenant ID [--config PATH] [--url URL]
```

Move a failed node back to the last stable point for the state it failed in (metal.recover)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal recover node_01k5w…
```

### metal retire [#metal-retire]

Retire a node: its retiring workflow erases its disks and powers it off (metal.retire)

```sh
better-compute metal retire NODE --tenant ID [--yes] [--config PATH] [--url URL]
```

metal retire starts the retiring workflow: a crypto-erase of the disks, power off, and the end of the node’s identity (certificates revoked, BMC credentials forgotten). A retired node does not come back; the machine has to be enrolled again. It needs --yes, or a yes at an interactive terminal. Decided as compute:nodes:provision.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--yes`           | Confirm: the disks are erased and the node is retired for good                                                                                            |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal retire node_01k5w… --yes
```

### metal deploy [#metal-deploy]

Deploy a provision profile version on a ready node (metal.deploy)

```sh
better-compute metal deploy NODE --tenant ID --profile ID --profile-version N [--config PATH] [--url URL]
```

Deploy a provision profile version on a ready node (metal.deploy)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile ID`        | The provision profile                                                                                                                                     |         |                         |
| `--profile-version N` | Its version                                                                                                                                               |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal deploy node_01k5w… --profile ubuntu-gpu --profile-version 3
```

### metal undeploy [#metal-undeploy]

Release a deployed node: wipe its disks and power it off (metal.undeploy)

```sh
better-compute metal undeploy NODE --tenant ID [--wipe MODE] [--yes] [--config PATH] [--url URL]
```

metal undeploy releases a deployed node: the installed system is gone, the disks are wiped as --wipe says (quick by default) and the machine is powered off, back to ready. It needs --yes, or a yes at an interactive terminal. Decided as compute:nodes:provision.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--wipe MODE`     | How the disks are wiped (default quick)                                                                                                                   |         |                         |
| `--yes`           | Confirm: the installed system and its disks are wiped                                                                                                     |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal undeploy node_01k5w… --wipe secure-erase --yes
```

### metal abort [#metal-abort]

Stop a node’s running workflow and mark it failed (metal.abort)

```sh
better-compute metal abort NODE --tenant ID [--reason TEXT] [--config PATH] [--url URL]
```

Stop a node’s running workflow and mark it failed (metal.abort)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | Recorded as its last error                                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal abort node_01k5w… --reason "wrong profile"
```

### metal discovered [#metal-discovered]

Machines a site saw PXE-booting from unknown MACs (metal.discovered)

```sh
better-compute metal discovered --tenant ID --site ID [--config PATH] [--url URL]
```

metal discovered lists what the site agent saw PXE/HTTP-boot from MACs with no boot entry: MAC, SMBIOS UUID, architecture, relay, first and last seen, and the enrolled machine it is a candidate for. Nothing in it is trusted or bound: confirm a candidate with metal confirm-mac.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--site ID`       | The site (site\_…)                                                                                                                                        |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal discovered --site site_01k5w… --format table
```

### metal confirm-mac [#metal-confirm-mac]

Bind the PXE MAC of a machine enrolled without one (metal.confirmMac)

```sh
better-compute metal confirm-mac NODE --tenant ID --mac MAC [--config PATH] [--url URL]
```

metal confirm-mac is the operator's decision that --mac is this machine's PXE MAC, usually the candidate metal discovered shows for it. Only then is the machine booted from that MAC; a MAC another machine holds is CONFLICT. The decision is audited (node.pxe-mac.confirmed).

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--mac MAC`       | The machine's PXE MAC (aa:bb:cc:dd:ee:ff)                                                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal confirm-mac node_01k5w… --mac 3c:ec:ef:12:34:56
```

### metal power [#metal-power]

Send a BMC power action through the site agent (metal.power)

```sh
better-compute metal power NODE --tenant ID --action ACTION [--yes] [--config PATH] [--url URL]
```

metal power sends one BMC power action through the site agent and returns without waiting for the BMC. The actions that cut power under a running system without asking it (off, reset, cycle) need --yes, or a yes at an interactive terminal; on, graceful-off, graceful-restart and nmi do not. Decided as compute:nodes:power.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--action ACTION` | The power action                                                                                                                                          |         |                         |
| `--yes`           | Confirm: off, reset and cycle cut power without asking the running system                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal power node_01k5w… --action graceful-restart
better-compute metal power node_01k5w… --action off --yes
```

### metal bulk [#metal-bulk]

Run one lifecycle verb on many nodes of a pool as one batch (metal.bulk)

```sh
better-compute metal bulk --tenant ID --pool ID --verb VERB [--nodes ID,…] [--nodes-file PATH] [--wipe MODE] --reason TEXT [--config PATH] [--url URL]
```

metal bulk runs --verb on the nodes in --nodes (comma-separated) or --nodes-file (one per line) of --pool as one batch; destructive verbs wait for the batch quorum’s approvers, who see --reason. Decided as compute:nodes:provision.

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`         | The pool every node is in                                                                                                                                 |         |                         |
| `--verb VERB`       | The verb every node gets                                                                                                                                  |         |                         |
| `--nodes ID,…`      | The nodes                                                                                                                                                 |         |                         |
| `--nodes-file PATH` | The nodes, one id per line                                                                                                                                |         |                         |
| `--wipe MODE`       | How the disks are wiped (default quick)                                                                                                                   |         |                         |
| `--reason TEXT`     | What the batch is for                                                                                                                                     |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute metal bulk --pool pool_01k5w… --verb retire --nodes-file retire.txt --reason "EOL rack r12"
```

### profiles list [#profiles-list]

List provision profiles: the tenant’s, public ones and the modules’ (profiles.list)

```sh
better-compute profiles list --tenant ID [--limit N] [--cursor CURSOR] [--config PATH] [--url URL]
```

List provision profiles: the tenant’s, public ones and the modules’ (profiles.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--limit N`       | Page size (at most 200)                                                                                                                                   |         |                         |
| `--cursor CURSOR` | The cursor of the previous page                                                                                                                           |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles list
```

### profiles get [#profiles-get]

Show a profile version, the latest by default (profiles.get)

```sh
better-compute profiles get PROFILE --tenant ID [--profile-version N] [--config PATH] [--url URL]
```

Show a profile version, the latest by default (profiles.get)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile-version N` | The version (default: the latest)                                                                                                                         |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles get ubuntu-gpu --profile-version 3
```

### profiles validate [#profiles-validate]

Check a profile version against the registered steps (profiles.validate)

```sh
better-compute profiles validate PROFILE --tenant ID [--profile-version N] [--config PATH] [--url URL]
```

Check a profile version against the registered steps (profiles.validate)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile-version N` | The version (default: the latest)                                                                                                                         |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles validate ubuntu-gpu --profile-version 3
```

### profiles render [#profiles-render]

Show the steps a workflow of a profile version would run (profiles.render)

```sh
better-compute profiles render PROFILE --tenant ID [--profile-version N] [--config PATH] [--url URL]
```

Show the steps a workflow of a profile version would run (profiles.render)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile-version N` | The version (default: the latest)                                                                                                                         |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles render ubuntu-gpu --profile-version 3
```

### profiles create [#profiles-create]

Store a profile version from a JSON document (profiles.create)

```sh
better-compute profiles create FILE --tenant ID [--public] [--config PATH] [--url URL]
```

Store a profile version from a JSON document (profiles.create)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--public`        | Let every tenant use it (default private)                                                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles create ./ubuntu-gpu.v3.json
```

### profiles update [#profiles-update]

Replace a profile version’s document or change its visibility (profiles.update)

```sh
better-compute profiles update PROFILE --tenant ID --profile-version N [--file PATH] [--visibility VIS] [--if-version N] [--config PATH] [--url URL]
```

Replace a profile version’s document or change its visibility (profiles.update)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile-version N` | The version to change                                                                                                                                     |         |                         |
| `--file PATH`         | A replacement document (same id and version)                                                                                                              |         |                         |
| `--visibility VIS`    | Who may use it                                                                                                                                            |         |                         |
| `--if-version N`      | Only if the row is still at this version                                                                                                                  |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles update ubuntu-gpu --profile-version 3 --visibility public
```

### profiles delete [#profiles-delete]

Hide a profile version, or every version, from new workflows (profiles.delete)

```sh
better-compute profiles delete PROFILE --tenant ID [--profile-version N] [--config PATH] [--url URL]
```

Hide a profile version, or every version, from new workflows (profiles.delete)

| Flag                  | Description                                                                                                                                               | Default | Environment             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`         | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--profile-version N` | Only this version (default: every version)                                                                                                                |         |                         |
| `--config PATH`       | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`           | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`     | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`        | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute profiles delete ubuntu-gpu --profile-version 2
```

### provider-accounts list [#provider-accounts-list]

List the tenant’s provider accounts (providerAccounts.list)

```sh
better-compute provider-accounts list --tenant ID [--config PATH] [--url URL]
```

List the tenant’s provider accounts (providerAccounts.list)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts list
```

### provider-accounts get [#provider-accounts-get]

Show a provider account (providerAccounts.get)

```sh
better-compute provider-accounts get ACCOUNT --tenant ID [--config PATH] [--url URL]
```

Show a provider account (providerAccounts.get)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts get pacct_01k5w…
```

### provider-accounts delete [#provider-accounts-delete]

Delete a provider account no pool or instance uses (providerAccounts.delete)

```sh
better-compute provider-accounts delete ACCOUNT --tenant ID [--config PATH] [--url URL]
```

Delete a provider account no pool or instance uses (providerAccounts.delete)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts delete pacct_01k5w…
```

### provider-accounts create [#provider-accounts-create]

Store a provider account; its config is checked by the provider (providerAccounts.create)

```sh
better-compute provider-accounts create --tenant ID --provider ID --name NAME [--config-file PATH] [--key KEY] [--config PATH] [--url URL]
```

provider-accounts create names an account of --provider and stores --config-file (JSON), checked against the provider’s manifest. Credentials in it are secret references only (secret://…), never values.

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--provider ID`      | The provider, e.g. provider-hetzner                                                                                                                       |         |                         |
| `--name NAME`        | The account name                                                                                                                                          |         |                         |
| `--config-file PATH` | The account config, as JSON                                                                                                                               |         |                         |
| `--key KEY`          | Idempotency key: a retry returns the same account                                                                                                         |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts create --provider provider-hetzner --name hetzner-main --config-file hetzner.json
```

### provider-accounts update [#provider-accounts-update]

Replace a provider account’s config (providerAccounts.update)

```sh
better-compute provider-accounts update ACCOUNT --tenant ID --config-file PATH [--config PATH] [--url URL]
```

Replace a provider account’s config (providerAccounts.update)

| Flag                 | Description                                                                                                                                               | Default | Environment             |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`        | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config-file PATH` | The new config, as JSON                                                                                                                                   |         |                         |
| `--config PATH`      | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`          | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`    | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`       | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts update pacct_01k5w… --config-file hetzner.json
```

### provider-accounts offers [#provider-accounts-offers]

Ask the provider which machine types the account can get (providerAccounts.offers)

```sh
better-compute provider-accounts offers ACCOUNT --tenant ID [--kind KIND] [--region REGION] [--gpu-model MODEL] [--config PATH] [--url URL]
```

Ask the provider which machine types the account can get (providerAccounts.offers)

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--kind KIND`       | Only this kind                                                                                                                                            |         |                         |
| `--region REGION`   | Only this provider region                                                                                                                                 |         |                         |
| `--gpu-model MODEL` | Only this GPU model                                                                                                                                       |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts offers pacct_01k5w… --kind metal --gpu-model H100
```

### provider-accounts check [#provider-accounts-check]

Ask the provider what the account’s credential may do, read-only (providerAccounts.check)

```sh
better-compute provider-accounts check ACCOUNT --tenant ID [--config PATH] [--url URL]
```

provider-accounts check reads the account at its provider without changing anything: the account’s state there, the permissions instances need that the credential lacks (missing) and the ones only optional features need (optional). It exits with an error when the provider refuses the credential.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts check pacct_01k5w…
```

### digitalocean check [#digitalocean-check]

Check a DigitalOcean account’s token scopes, read-only (providerAccounts.check)

```sh
better-compute digitalocean check ACCOUNT --tenant ID [--config PATH] [--url URL]
```

Check a DigitalOcean account’s token scopes, read-only (providerAccounts.check)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute digitalocean check pacct_01k5w…
```

### digitalocean offers [#digitalocean-offers]

List the Droplet sizes a DigitalOcean account can get (providerAccounts.offers)

```sh
better-compute digitalocean offers ACCOUNT --tenant ID [--region REGION] [--gpu-model MODEL] [--config PATH] [--url URL]
```

List the Droplet sizes a DigitalOcean account can get (providerAccounts.offers)

| Flag                | Description                                                                                                                                               | Default | Environment             |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`       | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--region REGION`   | Only this region (nyc3)                                                                                                                                   |         |                         |
| `--gpu-model MODEL` | Only this GPU model                                                                                                                                       |         |                         |
| `--config PATH`     | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`         | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`   | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`      | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute digitalocean offers pacct_01k5w…
```

### digitalocean discover [#digitalocean-discover]

List a DigitalOcean account’s Droplets again now (providerAccounts.discover)

```sh
better-compute digitalocean discover ACCOUNT --tenant ID [--config PATH] [--url URL]
```

List a DigitalOcean account’s Droplets again now (providerAccounts.discover)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute digitalocean discover pacct_01k5w…
```

### provider-accounts bootstrap [#provider-accounts-bootstrap]

Turn a provider machine into a bcd host of a pool (providerAccounts.bootstrap)

```sh
better-compute provider-accounts bootstrap ACCOUNT --tenant ID --pool ID [--machine ID] [--offer ID] [--key KEY] [--config PATH] [--url URL]
```

Turn a provider machine into a bcd host of a pool (providerAccounts.bootstrap)

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--pool ID`       | The vps-host or metal-host pool it joins                                                                                                                  |         |                         |
| `--machine ID`    | A machine the account already has                                                                                                                         |         |                         |
| `--offer ID`      | An offer to order a machine from                                                                                                                          |         |                         |
| `--key KEY`       | Idempotency key, to retry the same request                                                                                                                |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute provider-accounts bootstrap pacct_01k5w… --pool pool_01k5w… --offer ax102-h100
```

## Fleet [#fleet]

### fleet export [#fleet-export]

Write the tenant's platform objects as a FleetConfig document (fleet.export)

```sh
better-compute fleet export --tenant ID [--out PATH] [--config PATH] [--url URL]
```

fleet export prints (or writes with --out) the sites, provider accounts, pools with their owner quotas and offers, and VPS flavors the tenant owns, keyed by name. Objects fleet did not create are named by their id; rename them in the document and the next apply keeps them. Secrets appear only as references.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--out PATH`      | Write the document to a new file                                                                                                                          |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute fleet export --out fleet.json
```

### fleet plan [#fleet-plan]

Show what applying a FleetConfig document would change (fleet.plan)

```sh
better-compute fleet plan FILE --tenant ID [--prune] [--fail-on-drift] [--calls] [--config PATH] [--url URL]
```

fleet plan diffs the document against the tenant's current objects and prints each change: + create, \~ update (with the fields), - delete (only with --prune), and BLOCKED for changes the platform cannot make (a site's region, a pool's class or site, dangling names). It changes nothing. The printed planHash, passed to fleet apply --plan-hash, applies exactly this plan. With --fail-on-drift it exits FLEET\_DRIFTED when anything would change, for drift checks in CI; a blocked plan always exits PLAN\_BLOCKED.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--prune`         | Also delete what the document no longer lists (pools are archived)                                                                                        |         |                         |
| `--fail-on-drift` | Exit FLEET\_DRIFTED when the plan is not empty                                                                                                            |         |                         |
| `--calls`         | Also print the API calls apply would make                                                                                                                 |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute fleet plan fleet.json
better-compute fleet plan fleet.json --prune --fail-on-drift
```

### fleet status [#fleet-status]

Show the last applied document and whether the fleet drifted from it (fleet.status)

```sh
better-compute fleet status --tenant ID [--fail-on-drift] [--config PATH] [--url URL]
```

fleet status shows the planHash, author and time of the tenant's last fleet apply, and the drift the fleet.drift job found when it last re-planned that document (every 15 minutes by default): changes made outside fleet, listed like fleet plan. With --fail-on-drift it exits FLEET\_DRIFTED while there is drift, for monitoring.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--fail-on-drift` | Exit FLEET\_DRIFTED while the fleet drifted                                                                                                               |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute fleet status
better-compute fleet status --fail-on-drift
```

### fleet apply [#fleet-apply]

Apply a FleetConfig document (fleet.plan, then fleet.apply)

```sh
better-compute fleet apply FILE --tenant ID [--prune] [--plan-hash HASH] [--yes] [--config PATH] [--url URL]
```

fleet apply plans the document, prints the changes, then applies exactly that plan. Every underlying call (sites.create, pools.&#x2A;, quotas.set, flavors.*, providerAccounts.\*) is decided for you before anything runs, and then all of them run in one transaction, so a denial or failure applies nothing. If the fleet changes between the plan and the apply, nothing is applied and the command exits CONFIG\_DRIFT. Pass --plan-hash to apply a plan reviewed earlier with fleet plan; without --yes it only plans.

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--prune`          | Also delete what the document no longer lists (pools are archived)                                                                                        |         |                         |
| `--plan-hash HASH` | Apply only if the plan still has this hash (from fleet plan)                                                                                              |         |                         |
| `--yes`            | Apply without stopping after the plan                                                                                                                     |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute fleet apply fleet.json --yes
better-compute fleet apply fleet.json --prune --plan-hash 3f1c…
```

### daemon install-script [#daemon-install-script]

Write the POSIX sh script that installs bcd on a server and joins it to a pool

```sh
better-compute daemon install-script --pool ID --site ID --gateway-url URL --base-url URL [--base-path PATH] --bcd-version VERSION [--sha256 ARCH=HEX,…] [--release-manifest PATH] [--release-pins PINS] [--release-threshold N] [--plugins] [--install-plugins NAMES] [--unprivileged] [--attestor tpm] [--websocket-url URL] [--websocket-audience HOST] [--proxy-ca-file PATH] [--bmc-networks CIDRS] [--approver-set-sha256 HEX] [--instance-bridges BRIDGES] [--command-root-sha256 HEX,…] [--privileged-plugins NAMES] [--release-sequence N] [--pki PATH] [--out PATH]
```

daemon install-script emits the bcd installer (PLAN.md §3.3, §6). The script reads the join token only from BCD\_JOIN\_TOKEN or standard input, downloads the pinned bcd for the machine's architecture from \{base-url}\{base-path}/downloads/bcd/\{version}/linux-\{arch}, checks it against --sha256 (or the digests of a --release-manifest verified here under --release-pins, product bcd, version --bcd-version), installs /usr/local/bin/bcd, /etc/bcd/config.toml (mode 0600: gateway URL, site and the serving CA, join signing and join encryption key pins, in bcd's sha256/\<base64> form), the JoinService encryption key, and a hardened bcd.service, then runs bcd join and starts the service. Run on a configured host without a token, it upgrades bcd in place. The pins come from the PKI bundle's public half (pki init), so no secret is needed. The script holds no secret either: create the join token separately (joinTokens.create) and hand it over out of band. --plugins prepares the node for bcx driver plugins: a \[plugins] section (dir /usr/local/lib/bcd/plugins, a child of bcd's delegated cgroup) and a bcd.service.d drop-in that lets bcd run each plugin under its own uid in its own cgroup (CAP\_SETUID/CAP\_SETGID/CAP\_KILL, Delegate=yes); --privileged-plugins names the ones that run as root and make namespaces (Kata). --install-plugins bcx-ipmi also installs plugins a release ships: each archive from \{base-url}\{base-path}/downloads/\{name}/\{version}/linux-\{arch} is checked against its digest in the verified --release-manifest, only the plugin's own regular file is taken from it, and its binary digest becomes its line in the plugins' pins file. Any node whose config.toml has a \[plugins] section gets the drop-in, and an upgrade keeps the sections the operator configured.

| Flag                          | Description                                                                                                                                                                             | Default                    | Environment                  |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------------- |
| `--pool ID`                   | The pool the node joins (named in the script)                                                                                                                                           |                            |                              |
| `--site ID`                   | The pool's site id (config.toml site)                                                                                                                                                   |                            |                              |
| `--gateway-url URL`           | The regional gateway bcd dials, https\://…                                                                                                                                              |                            | `BETTER_COMPUTE_GATEWAY_URL` |
| `--base-url URL`              | The control plane serving the bcd downloads, https\://…                                                                                                                                 |                            | `BETTER_COMPUTE_BASE_URL`    |
| `--base-path PATH`            | The API's base path                                                                                                                                                                     | `/api/compute`             |                              |
| `--bcd-version VERSION`       | The bcd release to install, e.g. 0.1.0                                                                                                                                                  |                            | `BETTER_COMPUTE_BCD_VERSION` |
| `--sha256 ARCH=HEX,…`         | SHA-256 of each architecture’s binary: amd64=\<hex>,arm64=\<hex> (or use --release-manifest)                                                                                            |                            |                              |
| `--release-manifest PATH`     | A signed bcd release manifest: its verified digests are pinned instead                                                                                                                  |                            |                              |
| `--release-pins PINS`         | Comma-separated sha256/… pins of the release keys                                                                                                                                       |                            | `BCD_RELEASE_PINS`           |
| `--release-threshold N`       | Release keys that must have signed (default 1)                                                                                                                                          |                            |                              |
| `--plugins`                   | Prepare the node for bcx driver plugins (\[plugins], the systemd drop-in)                                                                                                               |                            |                              |
| `--install-plugins NAMES`     | Install these bcx plugins from the release, pinned by --release-manifest (bcx-ipmi; implies --plugins)                                                                                  |                            |                              |
| `--unprivileged`              | Run bcd node as the unprivileged bcd user beside a root bcd helper (privilege separation)                                                                                               |                            |                              |
| `--attestor tpm`              | Join with the host's TPM (the tpm attestor: its endorsement key is the credential; the attestor's policy decides the pool) instead of a join token                                      |                            |                              |
| `--websocket-url URL`         | Dial the gateway over WebSocket (wss\://…/v1/session) behind TLS-inspecting proxies                                                                                                     |                            |                              |
| `--websocket-audience HOST`   | The gateway's public host the WebSocket proof names (default: the URL's host)                                                                                                           |                            |                              |
| `--proxy-ca-file PATH`        | The TLS-inspecting proxy's CA (PEM), trusted for the WebSocket's TLS only                                                                                                               |                            |                              |
| `--bmc-networks CIDRS`        | A site agent's BMC networks (comma-separated): site.httpProxy never reaches into them                                                                                                   |                            |                              |
| `--approver-set-sha256 HEX`   | Pin the deployment's approver set (pki trust-version prints it): its first set, or nothing destructive                                                                                  |                            |                              |
| `--instance-bridges BRIDGES`  | With --unprivileged, on VPS and sandbox hosts: the bridges instance NICs may join (\[helper] instance\_bridges, comma-separated)                                                        |                            |                              |
| `--command-root-sha256 HEX,…` | Pin the command-signing roots (pki trust-version prints them): bcd helper applies host config units only from specs signed under them. Default: the PKI bundle’s current and next roots |                            |                              |
| `--privileged-plugins NAMES`  | Plugins that run as root without the seccomp filter (comma-separated; implies --plugins)                                                                                                |                            |                              |
| `--release-sequence N`        | bcd's release floor (\[update] release\_sequence): it never installs an older release                                                                                                   |                            |                              |
| `--pki PATH`                  | The PKI bundle written by pki init                                                                                                                                                      | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`         |
| `--out PATH`                  | Write the script to a new file (mode 0755) instead of standard output                                                                                                                   |                            |                              |

```sh
better-compute daemon install-script --pool pool_h100 --site fsn1 --gateway-url https://gw.eu.example.com --base-url https://compute.example.com --bcd-version 0.1.0 --sha256 amd64=<hex> > install-bcd.sh
BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh
```

## PKI [#pki]

### pki init [#pki-init]

Create the machine PKI: root, node, serving and internal CAs and the JoinService keys

```sh
better-compute pki init --trust-domain DOMAIN [--pki PATH] [--offline-root PATH] [--organization NAME] [--plaintext]
```

pki init creates a deployment's machine PKI (PLAN.md §13) and writes it to one JSON file with mode 0600: an offline root CA, the per-region node CA (agent client certificates), the gateway serving CA, the internal control-plane CA, and the JoinService signing and encryption keys. No key does two jobs. The private keys are sealed with AES-256-GCM under a key derived from BETTER\_COMPUTE\_SECRET; without the secret, pass --plaintext to store them as PEM instead. --offline-root writes the root key to its own file and leaves it out of the bundle: keep that file off the servers. It prints the pins install scripts need (the same as pki show) and never overwrites an existing file.

| Flag                    | Description                                                                               | Default                    | Environment                   |
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------- | ----------------------------- |
| `--trust-domain DOMAIN` | SPIFFE trust domain of agent identities, e.g. compute.example.com                         |                            | `BETTER_COMPUTE_TRUST_DOMAIN` |
| `--pki PATH`            | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`          |
| `--offline-root PATH`   | Write the root key here (sealed like the bundle) and leave it out of the bundle           |                            |                               |
| `--organization NAME`   | Subject organization of the CA certificates                                               | `better-compute`           |                               |
| `--plaintext`           | Store private keys as PEM (mode 0600) when BETTER\_COMPUTE\_SECRET is not set             |                            |                               |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                          |                            |                               |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                               |

```sh
BETTER_COMPUTE_SECRET=... better-compute pki init --trust-domain compute.example.com
better-compute pki init --trust-domain compute.test --plaintext --pki ./dev/pki.json
better-compute pki init --trust-domain compute.example.com --offline-root /media/usb/root-ca.key
```

### pki show [#pki-show]

Print the pins install scripts and bcd need, and the CA expiry dates

```sh
better-compute pki show [--pki PATH]
```

pki show prints the trust domain, the gateway serving CA's SPKI pin, the JoinService signing and encryption key pins in bcd's sha256/\<base64> form, the encryption key as a JWK, and when each CA expires. With BETTER\_COMPUTE\_SECRET the public half is checked against the seal first (authenticated: true); without it the pins come from a file nothing vouches for, and a warning says so.

| Flag              | Description                                                                               | Default                    | Environment          |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki show
better-compute pki show --query servingCaSpkiPin
```

### pki add-sandbox-ca [#pki-add-sandbox-ca]

Add the sandbox SSH CA to a bundle made before sandboxes

```sh
better-compute pki add-sandbox-ca [--pki PATH]
```

pki add-sandbox-ca creates the sandbox SSH CA (PLAN.md §3.7) in a bundle that lacks it and writes the bundle back (sealed with BETTER\_COMPUTE\_SECRET when set). sandboxes.exec signs with it (the tunnels module's sandboxSsh, pki tunnelsOptions()) and sandbox guests trust its public key (vps(\{ sandboxExec: \{ caPublicKey } })); it is separate from the break-glass CA, so a sandbox certificate never opens a node. Bundles from pki init already have one.

| Flag              | Description                                                                               | Default                    | Environment          |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki add-sandbox-ca
```

### pki reseal [#pki-reseal]

Seal the bundle in the current format, authenticating its public parts

```sh
better-compute pki reseal [--pki PATH] [--accept-unauthenticated]
```

pki reseal rewrites the bundle sealed with BETTER\_COMPUTE\_SECRET in the current format, where the seal also covers every certificate and pin, so a file changed without the secret no longer loads. Bundles sealed before that format, or left unsealed while a secret is set, are refused everywhere else. First compare the pins it prints (the same as pki show) with the ones your install scripts and join tokens use, and check `certificates` (the SHA-256 of every certificate in every chain) holds nothing you did not put there: it seals whatever the file holds now. Needs BETTER\_COMPUTE\_SECRET; the current key and certificates are kept, nothing is rotated.

| Flag                       | Description                                                                               | Default                    | Environment          |
| -------------------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--pki PATH`               | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--accept-unauthenticated` | Seal a bundle nothing authenticates, after comparing the pins printed without it          |                            |                      |
| `--format FORMAT`          | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`             | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki reseal
better-compute pki reseal --accept-unauthenticated
```

### pki rotate-command-root [#pki-rotate-command-root]

Rotate the command-signing root in three steps: add, switch, drop

```sh
better-compute pki rotate-command-root --stage STAGE [--pki PATH] [--force]
```

pki rotate-command-root performs one step of a command-signing root rotation (PLAN.md §13) on the bundle and writes it back (sealed with BETTER\_COMPUTE\_SECRET when set). --stage add creates a new root and publishes it beside the current one (the trust bundle's version goes up); workers keep signing under the current root. Roll the bundle out to every process: daemons receive the new version in Welcome.trust. --stage switch, once every daemon holds that version, makes workers sign under the new root while the old one stays trusted. --stage drop, at least 24 hours later (worker keys live at most 24 h), removes the old root (version up again). Each step prints the roots and version daemons will trust.

| Flag              | Description                                                                               | Default                    | Environment          |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--stage STAGE`   | add a new root, switch signing to it, or drop the old one                                 |                            |                      |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--force`         | switch without checking that every connected daemon has the new trust version             |                            |                      |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki rotate-command-root --stage add
better-compute pki rotate-command-root --stage switch
better-compute pki rotate-command-root --stage drop
```

### pki trust-version [#pki-trust-version]

Show the trust bundle version the deployment publishes, and the one to sign a quorum change for

```sh
better-compute pki trust-version [--pki PATH] [--config PATH]
```

pki trust-version reads the trust bundle the deployment last published (its version follows its content: a changed bundle publishes the next version, pki.trustVersion is a floor) and compares it with this configuration and PKI bundle: whether they publish the same bundle, the published approver-set epoch, whether the configured approver quorum differs (then its current approvers sign it for signQuorumForEpoch: better-compute batch sign-approvers --epoch \<n>, with pki.approvers.epoch set to it), and how many connected daemons hold that version.

| Flag              | Description                                                                               | Default                    | Environment             |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | ----------------------- |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`    |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |                            | `BETTER_COMPUTE_CONFIG` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                         |

```sh
better-compute pki trust-version
```

### pki renew-ca [#pki-renew-ca]

Renew a CA's certificate under the root, keeping its key

```sh
better-compute pki renew-ca --ca CA [--root-key PATH] [--days N] [--pki PATH]
```

pki renew-ca issues a new certificate for the node, serving or internal CA from the root CA, for the same key and subject, and writes the bundle back (sealed with BETTER\_COMPUTE\_SECRET when set). Certificates the CA issued stay valid and the serving CA's SPKI pin does not change, so nothing is re-pinned and no node rejoins; roll the bundle out to every process before the old certificate expires (doctor warns 30 days ahead). The root key comes from the bundle or, for an offline root, from --root-key; it is never written into the bundle then.

| Flag              | Description                                                                               | Default                    | Environment          |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--ca CA`         | Which CA to renew                                                                         |                            |                      |
| `--root-key PATH` | The offline root key (pki init --offline-root), when the bundle has none                  |                            |                      |
| `--days N`        | Lifetime of the new certificate in days (capped by the root)                              | `365`                      |                      |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki renew-ca --ca node
better-compute pki renew-ca --ca serving --root-key /media/usb/root-ca.key
```

### pki issue-gateway [#pki-issue-gateway]

Issue a gateway serving certificate and key from the serving CA

```sh
better-compute pki issue-gateway --san NAMES [--out DIR] [--name NAME] [--days N] [--pki PATH]
```

pki issue-gateway generates a P-256 key and issues a serving certificate for it from the bundle's gateway serving CA (PLAN.md §7.1): DNS SANs from --san (comma-separated), serverAuth, 30 days by default. It writes \<name>.key (mode 0600) and \<name>.crt (the certificate followed by the serving CA's chain) into --out, replacing earlier files atomically so a running gateway can hot-reload them. Daemons trust the certificate through the serving CA pin (pki show).

| Flag              | Description                                                                               | Default                    | Environment          |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------- |
| `--san NAMES`     | DNS names, comma-separated, e.g. gw\.eu-central.example.com                               |                            |                      |
| `--out DIR`       | Directory for the key and certificate                                                     | `.better-compute/gateway`  |                      |
| `--name NAME`     | File name stem: NAME.key and NAME.crt                                                     | `gateway`                  |                      |
| `--days N`        | Certificate lifetime in days                                                              | `30`                       |                      |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                      |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                      |

```sh
better-compute pki issue-gateway --san gw.eu-central.example.com
better-compute pki issue-gateway --san gw1.example.com,gw.example.com --out /etc/better-compute/tls --days 14
```

## Quorum [#quorum]

### batch keygen [#batch-keygen]

Create an approver key pair for destructive batches

```sh
better-compute batch keygen --out PATH
```

batch keygen creates an ECDSA P-256 approver key (PLAN.md §13): the private key (PKCS#8 PEM, mode 0600) at --out, the public key beside it at \<out>.pub. Keep the private key with its operator, off the control plane (a hardware token or an offline machine is better). Add the public key to the deployment's pki.approvers.keys, e.g. approverKeyOf(readFileSync('alice.pem.pub', 'utf8')).spkiDer from @better-compute/pki, with pki.approvers.threshold; daemons receive it in their trust bundle. The kid printed is how approvals name the key. With BETTER\_COMPUTE\_APPROVER\_PASSPHRASE set, the private key is encrypted under it (and every batch command reads it with it).

| Flag              | Description                                                                               | Default | Environment |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ----------- |
| `--out PATH`      | Where to write the private key (never overwritten)                                        |         |             |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |             |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |             |

```sh
better-compute batch keygen --out ~/.better-compute/approver.pem
```

### batch sign [#batch-sign]

Sign a batch approval offline, from its scope

```sh
better-compute batch sign --key PATH [--batch ID] [--scope PATH] [--root HEX] [--size N] [--not-after TIME] [--trust-domain DOMAIN] [--pki PATH] [--expect PATH] [--allow-unbound] [--allow-opaque] [--operations-out PATH] [--max-hours N] [--unchecked]
```

batch sign signs one approval of a destructive batch with an approver key (--key, from batch keygen) without contacting the server: for approvers on an offline machine. Pass the batch's scope (--scope, the JSON batch show prints, or an array of \{ subject, kind }): the root is recomputed from it and every subject is shown by kind before signing (--expect checks them against your own list). --root and --size, when given as well, must match what the scope gives; signing --root and --size alone, a root the CLI cannot check, needs --unchecked. --not-after is the batch's expiry as the server set it (all approvers sign the same one), at most --max-hours away. Prints the signature (\{ kid, sig }) to hand to batch approve --signature.

| Flag                    | Description                                                                                          | Default                    | Environment                   |
| ----------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------- | ----------------------------- |
| `--key PATH`            | The approver private key (PEM)                                                                       |                            |                               |
| `--batch ID`            | The batch id (default: the scope file's batchId)                                                     |                            |                               |
| `--scope PATH`          | The batch scope (JSON), to recompute the root from                                                   |                            |                               |
| `--root HEX`            | The scope root (64 hex digits), checked against --scope                                              |                            |                               |
| `--size N`              | The scope size, checked against --scope                                                              |                            |                               |
| `--not-after TIME`      | The batch's expiry: ISO-8601 or epoch milliseconds (default: the scope file's notAfter)              |                            |                               |
| `--trust-domain DOMAIN` | The deployment you approve for (default: the PKI bundle's trust domain); any other is refused        |                            | `BETTER_COMPUTE_TRUST_DOMAIN` |
| `--pki PATH`            | The PKI bundle written by pki init                                                                   | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`          |
| `--expect PATH`         | Approve only if the batch holds exactly these subjects (one per line) or \{ subject, kind } commands |                            |                               |
| `--allow-unbound`       | Approve commands whose payload the batch does not fix (shown as "any payload")                       |                            |                               |
| `--allow-opaque`        | Approve commands whose operation the batch shows only as a digest                                    |                            |                               |
| `--operations-out PATH` | Also write every distinct operation in full (JSON), with the subjects that run it                    |                            |                               |
| `--max-hours N`         | Refuse a batch that stays open longer than this many hours                                           | `72`                       |                               |
| `--unchecked`           | Sign --root and --size without a scope to check them against                                         |                            |                               |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                                     |                            |                               |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)            |                            |                               |

```sh
better-compute batch sign --key approver.pem --scope batch.json --not-after 2026-09-25T18:00:00Z
better-compute batch sign --key approver.pem --scope batch.json --expect my-nodes.txt
```

### batch sign-approvers [#batch-sign-approvers]

Sign a change of the approver set with a current approver key

```sh
better-compute batch sign-approvers --key PATH --epoch N [--approvers PATHS] [--threshold K] [--tiers SIZE:K,…] [--unbatched N] [--trust-domain DOMAIN] [--pki PATH]
```

batch sign-approvers signs, as one of the CURRENT approvers (--key), the quorum a new trust bundle will carry: its approver keys (--approvers, the .pub files of the whole new set), --threshold, --tiers and --unbatched, for approver-set --epoch (one above the current set's; better-compute pki trust-version prints it) in the deployment's --trust-domain (default: the PKI bundle's; a set signed for staging never changes prod's quorum, even with shared approver keys). Daemons that hold approver keys take a changed quorum, removal included, only with such signatures from their current threshold of current approvers (PLAN.md §13): the join key alone, which the control plane holds, cannot lift or replace it. Members of the NEW set sign it too (their signatures move daemons that already hold that set to the new epoch); give all the signatures to the deployment's pki.approvers.setSignatures with pki.approvers.epoch; they stay valid on every later trust bundle version while the set is current. Prints the set's digest and the signature.

| Flag                    | Description                                                                                   | Default                    | Environment                   |
| ----------------------- | --------------------------------------------------------------------------------------------- | -------------------------- | ----------------------------- |
| `--key PATH`            | Your CURRENT approver private key (PEM)                                                       |                            |                               |
| `--epoch N`             | The new set's epoch: one above the current set's                                              |                            |                               |
| `--approvers PATHS`     | The new set: approver public keys (.pub files, comma-separated); empty lifts the quorum       |                            |                               |
| `--threshold K`         | Approvals a batch needs in the new set                                                        |                            |                               |
| `--tiers SIZE:K,…`      | Per-size thresholds, ascending, e.g. 5:1,100:2                                                |                            |                               |
| `--unbatched N`         | Unbatched destructive commands per hour                                                       | `0`                        |                               |
| `--trust-domain DOMAIN` | The deployment you approve for (default: the PKI bundle's trust domain); any other is refused |                            | `BETTER_COMPUTE_TRUST_DOMAIN` |
| `--pki PATH`            | The PKI bundle written by pki init                                                            | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`          |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                              |                            |                               |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)     |                            |                               |

```sh
better-compute batch sign-approvers --key alice.pem --epoch 2 --approvers alice.pem.pub,bob.pem.pub,carol.pem.pub --threshold 2
```

### batch list [#batch-list]

List destructive batches, newest first

```sh
better-compute batch list --tenant ID [--state STATE] [--limit N] [--cursor ID] [--config PATH] [--url URL]
```

batch list pages commandBatches.list: each batch with its reason, state (pending, approved, expired, cancelled), scope size, the approvals it has and needs, and its member commands by state. Needs compute:batches:read.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--state STATE`   | Only batches in this state                                                                                                                                |         |                         |
| `--limit N`       | Batches per page                                                                                                                                          | `50`    |                         |
| `--cursor ID`     | The previous page's next                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute batch list --tenant org_acme --state pending
```

### batch show [#batch-show]

Show a batch and check its scope against its root

```sh
better-compute batch show --tenant ID --batch ID [--scope-out PATH] [--config PATH] [--url URL]
```

batch show reads a batch (commandBatches.get), pages its scope (commandBatches.scope) and recomputes the Merkle root from the listed (subject, kind) commands: it fails with SCOPE\_MISMATCH when they do not give the root the server asks approvers to sign. --scope-out writes the verified scope with the batch's expiry, the input of batch sign on an offline machine.

| Flag               | Description                                                                                                                                               | Default | Environment             |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`      | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--batch ID`       | The batch (batch\_…)                                                                                                                                      |         |                         |
| `--scope-out PATH` | Write the verified scope (JSON) for batch sign                                                                                                            |         |                         |
| `--config PATH`    | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`        | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute batch show --tenant org_acme --batch batch_01k5w…
better-compute batch show --tenant org_acme --batch batch_01k5w… --scope-out batch.json
```

### batch approve [#batch-approve]

Approve a destructive batch with your approver key

```sh
better-compute batch approve --tenant ID --batch ID [--key PATH] [--signature PATH] [--root HEX] [--expect PATH] [--allow-unbound] [--allow-opaque] [--operations-out PATH] [--max-hours N] [--trust-domain DOMAIN] [--pki PATH] [--yes] [--config PATH] [--url URL]
```

batch approve reads the batch, pages its scope and recomputes the root from the listed commands (refusing with SCOPE\_MISMATCH when they do not give it), shows the reason, every subject by kind (--expect checks them against your own list) and the approvals so far, and then signs the batch's root, size and expiry with --key and submits the signature (commandBatches.approve, compute:batches:approve). Without --yes it stops after showing the batch; --root approves only the root you reviewed. Once the batch holds the required approvals it is released and its commands run. With --signature, submits a signature made offline by batch sign instead, after checking it is for this batch.

| Flag                    | Description                                                                                                                                               | Default                    | Environment                   |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | ----------------------------- |
| `--tenant ID`           | The tenant acting (the caller’s organization)                                                                                                             |                            | `BETTER_COMPUTE_TENANT`       |
| `--batch ID`            | The batch (batch\_…)                                                                                                                                      |                            |                               |
| `--key PATH`            | Your approver private key (from batch keygen)                                                                                                             |                            |                               |
| `--signature PATH`      | A signature from batch sign (instead of --key)                                                                                                            |                            |                               |
| `--root HEX`            | Approve only if the batch still has this root (64 hex digits)                                                                                             |                            |                               |
| `--expect PATH`         | Approve only if the batch holds exactly these subjects (one per line) or \{ subject, kind } commands                                                      |                            |                               |
| `--allow-unbound`       | Approve commands whose payload the batch does not fix (shown as "any payload")                                                                            |                            |                               |
| `--allow-opaque`        | Approve commands whose operation the batch shows only as a digest                                                                                         |                            |                               |
| `--operations-out PATH` | Also write every distinct operation in full (JSON), with the subjects that run it                                                                         |                            |                               |
| `--max-hours N`         | Refuse a batch that stays open longer than this many hours                                                                                                | `72`                       |                               |
| `--trust-domain DOMAIN` | The deployment you approve for (default: the PKI bundle's trust domain); any other is refused                                                             |                            | `BETTER_COMPUTE_TRUST_DOMAIN` |
| `--pki PATH`            | The PKI bundle written by pki init                                                                                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`          |
| `--yes`                 | Sign and submit without stopping after the summary                                                                                                        |                            |                               |
| `--config PATH`         | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |                            | `BETTER_COMPUTE_CONFIG`       |
| `--url URL`             | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |                            | `BETTER_COMPUTE_URL`          |
| `--format FORMAT`       | Output as json (indented, default), compact (one line), or table                                                                                          |                            |                               |
| `--query PATH`          | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |                            |                               |

```sh
better-compute batch approve --tenant org_acme --batch batch_01k5w… --key ~/.better-compute/approver.pem
better-compute batch approve --tenant org_acme --batch batch_01k5w… --key approver.pem --root 3f1c…
better-compute batch approve --tenant org_acme --batch batch_01k5w… --signature alice.sig.json
```

### batch create [#batch-create]

Create a destructive batch that waits for its approvers

```sh
better-compute batch create --tenant ID --reason TEXT --scope PATH [--hours N] [--key KEY] [--config PATH] [--url URL]
```

batch create creates a batch (commandBatches.create, compute:batches:create) from --scope FILE: a JSON array of \{ subject, kind, payload } leaves. subject is the node id for kinds a node runs on itself, bmc:\<site agent node id>/\<address> for bmc.\*, node:\<target> for site.httpProxy. payload binds the leaf to exactly one operation, which approvers are shown: \{ json } (the kind's payload message as proto3 JSON, encoded here) or \{ typeUrl, value } (base64); a leaf without one admits any payload of its kind and approvers must accept that explicitly. The batch admits its commands for --hours (default 24, at most 72) once approved; run them with commands.run \{ batchId }. Prints the batch; approvers then run better-compute batch approve.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--reason TEXT`   | What the batch does, shown to approvers                                                                                                                   |         |                         |
| `--scope PATH`    | The leaves (JSON array of \{ subject, kind, payload })                                                                                                    |         |                         |
| `--hours N`       | How long it admits its commands                                                                                                                           | `24`    |                         |
| `--key KEY`       | Idempotency key: repeat it to retry the same creation                                                                                                     |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute batch create --tenant org_acme --reason "reimage rack 7" --scope rack7.json
```

### batch members [#batch-members]

List a batch's commands: node, kind, state, whether held for approval

```sh
better-compute batch members --tenant ID --batch ID [--state STATE] [--limit N] [--cursor ID] [--config PATH] [--url URL]
```

batch members pages commandBatches.members: every command enqueued into the batch with its node (the site agent for bmc: subjects), subject, kind and state; held ones are awaitingApproval until the batch is approved. Needs compute:batches:read.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--batch ID`      | The batch (batch\_…)                                                                                                                                      |         |                         |
| `--state STATE`   | Only commands in this state (pending, succeeded, …)                                                                                                       |         |                         |
| `--limit N`       | Commands per page                                                                                                                                         | `100`   |                         |
| `--cursor ID`     | The previous page's next                                                                                                                                  |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute batch members --tenant org_acme --batch batch_01k5w… --state pending
```

### batch cancel [#batch-cancel]

Cancel a pending batch and its held commands

```sh
better-compute batch cancel --tenant ID --batch ID [--config PATH] [--url URL]
```

batch cancel ends a pending batch (commandBatches.cancel, compute:batches:approve): none of its commands run, and the pending ones are cancelled with reason batch-cancelled.

| Flag              | Description                                                                                                                                               | Default | Environment             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant acting (the caller’s organization)                                                                                                             |         | `BETTER_COMPUTE_TENANT` |
| `--batch ID`      | The batch (batch\_…)                                                                                                                                      |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}                                                                         |         | `BETTER_COMPUTE_CONFIG` |
| `--url URL`       | Call a running compute server ([https://host](https://host) or [https://host/api/compute](https://host/api/compute)) instead of loading the configuration |         | `BETTER_COMPUTE_URL`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                                                                                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                 |         |                         |

```sh
better-compute batch cancel --tenant org_acme --batch batch_01k5w…
```

## IAM [#iam]

### iam setup [#iam-setup]

Print the example compute policies and create the root billing meters in better-iam

```sh
better-compute iam setup --audience URL [--gpu-model MODEL] [--regions LIST] [--max-gpus N] [--max-hours N] [--feature FLAG] [--meters] [--iam-url URL] [--iam-base-path PATH] [--root-tenant ID] [--gpu-models LIST]
```

iam setup prints the PLAN.md §12.6 example policies, gpu-user and agent-guardrails, for an organization administrator to apply: compute cannot write organization policies. The policies are parameterized by the session JWT audience (the compute API's URL), the GPU model, regions and limits of the bounded-reservation statement. With --meters it also creates the root meter catalog in better-iam: gpu.\<family>.hours and gpu.\<family>.spot.hours for every known GPU family, the provider-VM (vm.gpu.\*) and reserved GPU hours for each model in --gpu-models (default H100,H200,B200,A100,L4,L40S and other), plus node.metal.hours, the hosted and provider vCPU, memory and volume hours, reservation-grant and managed-VM hours, and the network, public-IP, disk and snapshot hours (at most 100 meters, which better-iam allows per tenant). Meters that already exist are left alone, so re-running is safe. Creating meters calls better-iam at --iam-url as BETTER\_IAM\_TOKEN, which must be a root credential, through the optional peer @better-iam/client. It sets no prices: pricing is an operator decision (billing.setPrice).

| Flag                   | Description                                                                                                                                                                                                                         | Default    | Environment                  |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------- |
| `--audience URL`       | The compute API URL, used as the session JWT audience                                                                                                                                                                               |            | `BETTER_COMPUTE_BASE_URL`    |
| `--gpu-model MODEL`    | GPU model family the bounded-reservation statement allows                                                                                                                                                                           | `H100`     |                              |
| `--regions LIST`       | Regions the bounded-reservation statement allows, comma-separated                                                                                                                                                                   |            |                              |
| `--max-gpus N`         | GPUs per reservation                                                                                                                                                                                                                | `8`        |                              |
| `--max-hours N`        | Hours per reservation, extensions included                                                                                                                                                                                          | `72`       |                              |
| `--feature FLAG`       | Tenant feature the reservation statement requires (default compute.\<model>)                                                                                                                                                        |            |                              |
| `--meters`             | Also create the root billing meters in better-iam                                                                                                                                                                                   |            |                              |
| `--iam-url URL`        | better-iam base URL for --meters                                                                                                                                                                                                    |            | `BETTER_IAM_URL`             |
| `--iam-base-path PATH` | better-iam's API base path                                                                                                                                                                                                          | `/api/iam` |                              |
| `--root-tenant ID`     | The root tenant that owns the meters (required with --meters)                                                                                                                                                                       |            | `BETTER_COMPUTE_ROOT_TENANT` |
| `--gpu-models LIST`    | GPU models to price on provider VMs (vm.gpu.\*), comma-separated (default: H100,H200,B200,A100,L4,L40S; other is always added). Hosted gpu.\<family>.hours and .spot meters exist for every known family. At most 100 meters in all |            |                              |
| `--format FORMAT`      | Output as json (indented, default), compact (one line), or table                                                                                                                                                                    |            |                              |
| `--query PATH`         | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw)                                                                                                                                           |            |                              |

```sh
better-compute iam setup --audience https://compute.example.com --query "[0]"
BETTER_IAM_TOKEN=... better-compute iam setup --audience https://compute.example.com --meters --iam-url https://iam.example.com --root-tenant ten_root
```

## Audit [#audit]

### audit checkpoint [#audit-checkpoint]

Seal committed audit events into signed per-tenant checkpoints

```sh
better-compute audit checkpoint [--pki PATH] [--limit N] [--config PATH]
```

audit checkpoint runs what the audit module's hourly job does: it seals every audit event committed since the last checkpoint into a signed, hash-chained Merkle checkpoint per tenant, signed with the PKI bundle's audit key (unsealed with BETTER\_COMPUTE\_SECRET). It repeats until nothing is left. The deployment must include auditModule() and be migrated.

| Flag              | Description                                                                               | Default                    | Environment             |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | ----------------------- |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`    |
| `--limit N`       | Events per pass (default 50000)                                                           |                            |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |                            | `BETTER_COMPUTE_CONFIG` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                         |

```sh
better-compute audit checkpoint
```

### audit verify [#audit-verify]

Verify a tenant's audit trail against its signed checkpoints

```sh
better-compute audit verify --tenant ID [--pin PIN] [--pki PATH] [--config PATH]
```

audit verify recomputes every checkpoint of --tenant from the audit events as they are now: signatures under the audit key's pin, the hash chain, contiguous ranges, sizes and Merkle roots. An edited, deleted or inserted event in a sealed range, a broken chain or a foreign signature is a problem; with any problem the command exits AUDIT\_TAMPERED after printing the report. It only reads.

| Flag              | Description                                                                               | Default                    | Environment                |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------------- |
| `--tenant ID`     | The tenant                                                                                |                            | `BETTER_COMPUTE_TENANT`    |
| `--pin PIN`       | The audit signing key pin (sha256/…); default: from the PKI bundle                        |                            | `BETTER_COMPUTE_AUDIT_PIN` |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`       |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |                            | `BETTER_COMPUTE_CONFIG`    |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                            |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                            |

```sh
better-compute audit verify --tenant org_acme
better-compute audit verify --tenant org_acme --query problems
```

### audit prove [#audit-prove]

Write an inclusion proof that one audit event is sealed

```sh
better-compute audit prove EVENT --tenant ID [--out PATH] [--config PATH]
```

audit prove writes a self-contained proof that the event is in a signed checkpoint: the event, its Merkle path and the signed checkpoint. Anyone holding the audit key pin checks it offline with audit check-proof, without database access.

| Flag              | Description                                                                               | Default | Environment             |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `--tenant ID`     | The tenant                                                                                |         | `BETTER_COMPUTE_TENANT` |
| `--out PATH`      | Write the proof to a file instead of printing it                                          |         |                         |
| `--config PATH`   | Configuration module; default: the nearest better-compute.config.\{mjs,js,ts,mts}         |         | `BETTER_COMPUTE_CONFIG` |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |                         |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |                         |

```sh
better-compute audit prove aud_01k… --tenant org_acme --out proof.json
```

### audit check-proof [#audit-check-proof]

Check an inclusion proof offline against the audit key pin

```sh
better-compute audit check-proof FILE [--pin PIN] [--pki PATH]
```

audit check-proof verifies a proof written by audit prove: the checkpoint's signature under the pin and the event's Merkle path to the signed root. It needs no database; it exits INVALID\_PROOF when the proof does not hold.

| Flag              | Description                                                                               | Default                    | Environment                |
| ----------------- | ----------------------------------------------------------------------------------------- | -------------------------- | -------------------------- |
| `--pin PIN`       | The audit signing key pin (sha256/…); default: from the PKI bundle                        |                            | `BETTER_COMPUTE_AUDIT_PIN` |
| `--pki PATH`      | The PKI bundle written by pki init                                                        | `.better-compute/pki.json` | `BETTER_COMPUTE_PKI`       |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |                            |                            |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |                            |                            |

```sh
better-compute audit check-proof proof.json --pin sha256/…
```

## Release [#release-1]

### release keygen [#release-keygen]

Create an offline release signing key and print its pin

```sh
better-compute release keygen --out PATH
```

release keygen writes a new P-256 release signing key to --out (owner-only, never overwritten) and its public half to \<out>.pub.pem, and prints the key's pin. Keep the key offline. Give the pin to installers (BCD\_RELEASE\_PINS) and daemons; give the .pub.pem to whoever builds manifests that you will cosign.

| Flag              | Description                                                                               | Default | Environment |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ----------- |
| `--out PATH`      | Where to write the key                                                                    |         |             |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |             |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |             |

```sh
better-compute release keygen --out ~/keys/release-alice.pem
```

### release sign [#release-sign]

Sign a release manifest over a directory's artifacts

```sh
better-compute release sign DIR --product NAME --version SEMVER --sequence N --key PATH [--cosigners PATHS] [--validity-days DAYS] [--out PATH]
```

release sign hashes every file in DIR (not subdirectories; \*.sha256, manifest.json and signature files are skipped), derives each artifact's platform from goreleaser names (\_linux\_amd64.), and writes a manifest signed with --key. --sequence must increase with every release of --product: installers and bcd refuse a manifest whose sequence is below the last one they accepted. List the public halves of keys that will cosign later with --cosigners so their signatures count.

| Flag                   | Description                                                                               | Default | Environment |
| ---------------------- | ----------------------------------------------------------------------------------------- | ------- | ----------- |
| `--product NAME`       | e.g. bcd                                                                                  |         |             |
| `--version SEMVER`     | e.g. 1.4.0                                                                                |         |             |
| `--sequence N`         | The rollback counter: higher than every earlier release                                   |         |             |
| `--key PATH`           | The signing key                                                                           |         |             |
| `--cosigners PATHS`    | Comma-separated .pub.pem files of keys that will cosign                                   |         |             |
| `--validity-days DAYS` | How long installers accept the manifest (default 365)                                     |         |             |
| `--out PATH`           | Where to write the manifest (default DIR/manifest.json)                                   |         |             |
| `--format FORMAT`      | Output as json (indented, default), compact (one line), or table                          |         |             |
| `--query PATH`         | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |             |

```sh
better-compute release sign dist --product bcd --version 1.4.0 --sequence 14 --key ~/keys/release-alice.pem
```

### release cosign [#release-cosign]

Add your signature to a release manifest

```sh
better-compute release cosign MANIFEST --key PATH
```

release cosign adds --key's signature to MANIFEST in place. The key must be one the manifest lists (release sign --cosigners), so a threshold of release managers can each sign on their own machine.

| Flag              | Description                                                                               | Default | Environment |
| ----------------- | ----------------------------------------------------------------------------------------- | ------- | ----------- |
| `--key PATH`      | Your signing key                                                                          |         |             |
| `--format FORMAT` | Output as json (indented, default), compact (one line), or table                          |         |             |
| `--query PATH`    | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |             |

```sh
better-compute release cosign dist/manifest.json --key ~/keys/release-bob.pem
```

### release verify [#release-verify]

Verify a release manifest (and its artifacts) against pinned keys

```sh
better-compute release verify MANIFEST --pins PINS [--threshold N] [--min-sequence N] [--product NAME] [--dir DIR]
```

release verify checks MANIFEST the way installers and bcd do, in this order: a --threshold of distinct keys among --pins signed it; its sequence is not below --min-sequence (rollback); it has not expired; and, with --dir, every listed artifact present there matches its size and SHA-256. It exits RELEASE\_REJECTED with the reason otherwise.

| Flag               | Description                                                                               | Default | Environment        |
| ------------------ | ----------------------------------------------------------------------------------------- | ------- | ------------------ |
| `--pins PINS`      | Comma-separated sha256/… pins of the release keys                                         |         | `BCD_RELEASE_PINS` |
| `--threshold N`    | Default 1                                                                                 |         |                    |
| `--min-sequence N` | The last sequence accepted before                                                         |         |                    |
| `--product NAME`   | Refuse other products                                                                     |         |                    |
| `--dir DIR`        | Also check the artifacts in DIR                                                           |         |                    |
| `--format FORMAT`  | Output as json (indented, default), compact (one line), or table                          |         |                    |
| `--query PATH`     | Print only part of the result, e.g. protocol.max or findings\[].check (strings print raw) |         |                    |

```sh
better-compute release verify dist/manifest.json --pins sha256/…,sha256/… --threshold 2 --dir dist
```

## Environment [#environment]

| Variable                      | Description                                                                                                        |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `BETTER_COMPUTE_CONFIG`       | Configuration module, like --config                                                                                |
| `BETTER_COMPUTE_TOKEN`        | IAM session token, API key or session JWT that token commands act as                                               |
| `BETTER_COMPUTE_URL`          | Compute server token commands call, like --url                                                                     |
| `BETTER_COMPUTE_DATABASE_URL` | Without a configuration file: the database (postgres\://, pglite:DIR, pglite:memory), with BETTER\_COMPUTE\_SECRET |
| `BETTER_COMPUTE_SECRET`       | Without a configuration file: the deployment secret (at least 32 characters)                                       |
| `BETTER_COMPUTE_BASE_URL`     | Without a configuration file: the public origin (default [http://localhost:3000](http://localhost:3000))           |
| `BETTER_COMPUTE_DEBUG`        | Print the stack trace of a failure                                                                                 |


# Command kinds (/docs/reference/command-kinds)

> The 21 command kinds daemons run, with lane, safety flags, timeout and payload.



Commands reach a node over its gateway session, one lane per kind. *Retry-safe* kinds are re-run after a crash between accept and completion; *self-terminating* ones end the agent (a reboot, a kexec, an upgrade) and complete when it rejoins; *attested* results are node-signed and verified before any transition; *destructive* ones must be signed.

| Kind                  | Lane      | Roles               | Retry-safe | Self-terminating | Attested | Destructive | Timeout | Payload                                          |
| --------------------- | --------- | ------------------- | ---------- | ---------------- | -------- | ----------- | ------- | ------------------------------------------------ |
| `bmc.boot`            | control   | site                | yes        |                  |          | yes         | `2m`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `bmc.firmware`        | control   | site                |            |                  |          | yes         | `1h`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `bmc.power`           | control   | site                |            |                  |          | yes         | `5m`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `bmc.query`           | control   | site                | yes        |                  |          |             | `2m`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `bmc.rotate`          | control   | site                |            |                  | yes      | yes         | `5m`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `bmc.storage`         | control   | site                |            |                  |          | yes         | `1h`    | `bettercompute.site.v1.BmcOperationRequest`      |
| `boot.cleanUp`        | control   | site                | yes        |                  |          |             | `2m`    | `bettercompute.site.v1.BootCleanUpRequest`       |
| `boot.kexec`          | control   | ramdisk             |            | yes              |          |             | `15m`   | `bettercompute.agent.v1.KexecRequest`            |
| `boot.prepareRamdisk` | control   | site                | yes        |                  |          |             | `2m`    | `bettercompute.site.v1.ProxyDhcpEntry`           |
| `daemon.upgrade`      | control   | node, site          | yes        | yes              |          |             | `15m`   | `bettercompute.agent.v1.DaemonSection`           |
| `deploy.image`        | provision | ramdisk             |            |                  |          | yes         | `1h`    | `bettercompute.agent.v1.DeployImageRequest`      |
| `diag.dcgm`           | provision | ramdisk, node       | yes        |                  | yes      |             | `1h`    | `bettercompute.agent.v1.DiagDcgmRequest`         |
| `diag.nvml`           | control   | node, ramdisk       | yes        |                  | yes      |             | `2m`    | `bettercompute.agent.v1.DiagNvmlRequest`         |
| `disk.wipe`           | provision | ramdisk             |            |                  | yes      | yes         | `6h`    | `bettercompute.agent.v1.WipeRequest`             |
| `driver.invoke`       | provision | node, site, ramdisk |            |                  |          | yes         | `10m`   | `bettercompute.agent.v1.DriverInvokeRequest`     |
| `gpu.reset`           | control   | node                |            |                  | yes      | yes         | `5m`    | `bettercompute.agent.v1.GpuResetRequest`         |
| `instance.snapshot`   | bulk      | node                |            |                  |          |             | `1h`    | `bettercompute.agent.v1.InstanceSnapshotRequest` |
| `inventory.refresh`   | control   | node, site, ramdisk | yes        |                  |          |             | `2m`    | `bettercompute.agent.v1.InventoryRefreshRequest` |
| `logs.collect`        | bulk      | node, site, ramdisk | yes        |                  |          |             | `15m`   | `bettercompute.agent.v1.LogsCollectRequest`      |
| `node.reboot`         | control   | node, site          | yes        | yes              |          | yes         | `15m`   | `bettercompute.agent.v1.NodeRebootRequest`       |
| `site.httpProxy`      | control   | site                |            |                  |          | yes         | `1m`    | `bettercompute.site.v1.HttpProxyRequest`         |


# Errors (/docs/reference/errors)

> Every stable error code the API, the SDK and the CLI surface, with its HTTP status.



Errors answer `{ "error": { "code", "message", "retryAfterMs"?, "details"? } }`. Codes are stable; messages are not.

| Code                                              | HTTP status | Meaning                                                                                                                                                                                               |
| ------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`UNAUTHENTICATED`](#unauthenticated)             | 401         | The request carries no credential, or IAM does not accept the one it carries.                                                                                                                         |
| [`ACCESS_DENIED`](#access_denied)                 | 403         | IAM decided the call's `action @ resource` and denied it.                                                                                                                                             |
| [`CONFIRMATION_REQUIRED`](#confirmation_required) | 403         | The caller is an agent acting for a person, and that person must confirm the request first.                                                                                                           |
| [`NOT_FOUND`](#not_found)                         | 404         | The route, or the object the input names, does not exist or is not visible to the caller.                                                                                                             |
| [`INVALID_INPUT`](#invalid_input)                 | 400         | The body failed validation: a missing or unknown field, a wrong type, or a value outside its constraints.                                                                                             |
| [`INVALID_STATE`](#invalid_state)                 | 409         | The object exists but is in a state that does not allow the call, such as uncordoning a node being deleted or extending a released reservation.                                                       |
| [`CONFLICT`](#conflict)                           | 409         | The call raced a concurrent change, or reuses an `idempotencyKey` for a different request, or names something that already exists.                                                                    |
| [`CAPACITY_UNAVAILABLE`](#capacity_unavailable)   | 409         | No placement satisfies the request now: there are not enough free GPUs, or none that match its constraints, in the pools it may use.                                                                  |
| [`QUOTA_EXCEEDED`](#quota_exceeded)               | 409         | The request would take the tenant or project past a quota, or past a per-tenant limit such as the number of webhooks.                                                                                 |
| [`SPEND_LIMIT_REACHED`](#spend_limit_reached)     | 402         | An IAM budget covering this request is spent, and the pool refuses new work when a budget is exceeded.                                                                                                |
| [`UNSUPPORTED`](#unsupported)                     | 422         | The request is valid, but the driver, provider or deployment it lands on cannot do it: a provider with no power control, an instance that cannot be resized in place, or a deployment with no SSH CA. |
| [`UPGRADE_REQUIRED`](#upgrade_required)           | 426         | The peer speaks a protocol version older than the server's minimum.                                                                                                                                   |
| [`RATE_LIMITED`](#rate_limited)                   | 429         | Too many requests of this kind, from the caller or to a provider behind the call.                                                                                                                     |
| [`DRIVER_ERROR`](#driver_error)                   | 502         | A driver, a node command or a site agent failed, or answered with something compute cannot trust: a BMC operation that failed, an unattested result, or firmware and BIOS settings that did not take. |
| [`PROVIDER_ERROR`](#provider_error)               | 502         | A VPS or bare-metal provider's API refused or failed the call.                                                                                                                                        |
| [`INVALID_CONFIG`](#invalid_config)               | 500         | The deployment is configured wrongly, for example an option out of range or a module that needs another one.                                                                                          |
| [`NOT_IMPLEMENTED`](#not_implemented)             | 501         | The method or feature is planned for a later phase and not built in this release.                                                                                                                     |
| [`CSRF_REJECTED`](#csrf_rejected)                 | 403         | A POST lacks `content-type: application/json` or the `X-Better-Compute: 1` header, or a cookie-authenticated request has no `Origin`.                                                                 |
| [`UNTRUSTED_ORIGIN`](#untrusted_origin)           | 403         | A cookie-authenticated request, or a tunnel's WebSocket upgrade, came from an `Origin` other than the deployment's own.                                                                               |
| [`METHOD_NOT_ALLOWED`](#method_not_allowed)       | 405         | The HTTP method is not accepted on this path.                                                                                                                                                         |
| [`PAYLOAD_TOO_LARGE`](#payload_too_large)         | 413         | The request body is larger than the server accepts.                                                                                                                                                   |
| [`INTERNAL_ERROR`](#internal_error)               | 500         | An unexpected failure on the server.                                                                                                                                                                  |
| [`MISSING_DEPENDENCY`](#missing_dependency)       | 500         | An optional package or service the deployment needs for this call is not installed or not configured, such as PGlite for an embedded database or a remote IAM endpoint.                               |
| [`CONFIG_DRIFT`](#config_drift)                   | 409         | The fleet changed after the plan was made, so `fleet.apply` refused a plan whose `planHash` no longer matches.                                                                                        |
| [`UNAVAILABLE`](#unavailable)                     | 503         | A dependency the request needs is unreachable for now: the secret store's KMS, a node's gateway, or remote IAM.                                                                                       |
| [`TENANT_SUSPENDED`](#tenant_suspended)           | 403         | The IAM tenant is suspended, so its mutations are refused, or deleted, so every call is refused.                                                                                                      |

## UNAUTHENTICATED [#unauthenticated]

HTTP 401.

The request carries no credential, or IAM does not accept the one it carries.

Send a valid IAM session or API key. An expired key or a revoked session answers the same way; the message does not say which.

## ACCESS\_DENIED [#access_denied]

HTTP 403.

IAM decided the call's `action @ resource` and denied it.

Callers outside the input tenant (and outside the root tenant) are refused before any database access. Grant the action in IAM, or call as an identity that holds it. See [Actions and resources](/docs/guides/iam/actions-and-resources).

## CONFIRMATION\_REQUIRED [#confirmation_required]

HTTP 403.

The caller is an agent acting for a person, and that person must confirm the request first.

`details` carries `confirmationId`, the `action` and `resource`, and `expiresAt` when IAM set one. Nothing was changed. Once the person confirms, retry with the same `idempotencyKey`. See [Delegated confirmation](/docs/guides/iam/delegated-confirmation).

## NOT\_FOUND [#not_found]

HTTP 404.

The route, or the object the input names, does not exist or is not visible to the caller.

Lookups are scoped to the input `tenantId`, so an id that belongs to another tenant is not found.

## INVALID\_INPUT [#invalid_input]

HTTP 400.

The body failed validation: a missing or unknown field, a wrong type, or a value outside its constraints.

Unknown fields are refused, never ignored. The Input table of each method in the [API reference](/docs/reference/api) lists every accepted field.

## INVALID\_STATE [#invalid_state]

HTTP 409.

The object exists but is in a state that does not allow the call, such as uncordoning a node being deleted or extending a released reservation.

Read the object again and act on its current state. Retrying unchanged fails the same way.

## CONFLICT [#conflict]

HTTP 409.

The call raced a concurrent change, or reuses an `idempotencyKey` for a different request, or names something that already exists.

A concurrent change is safe to retry after reading the object again. A reused key with a different body is not: send a new key for a new request.

## CAPACITY\_UNAVAILABLE [#capacity_unavailable]

HTTP 409.

No placement satisfies the request now: there are not enough free GPUs, or none that match its constraints, in the pools it may use.

Retry later, relax the constraints, or book a later window. See [Placement](/docs/guides/capacity/placement).

## QUOTA\_EXCEEDED [#quota_exceeded]

HTTP 409.

The request would take the tenant or project past a quota, or past a per-tenant limit such as the number of webhooks.

Release something, or raise the quota with `quotas.set`. See [Quotas and usage](/docs/guides/capacity/quotas-and-usage).

## SPEND\_LIMIT\_REACHED [#spend_limit_reached]

HTTP 402.

An IAM budget covering this request is spent, and the pool refuses new work when a budget is exceeded.

`details` names the budget (`budgetId`, `name`) when IAM reports it. See [Billing and usage](/docs/guides/iam/billing-and-usage).

## UNSUPPORTED [#unsupported]

HTTP 422.

The request is valid, but the driver, provider or deployment it lands on cannot do it: a provider with no power control, an instance that cannot be resized in place, or a deployment with no SSH CA.

## UPGRADE\_REQUIRED [#upgrade_required]

HTTP 426.

The peer speaks a protocol version older than the server's minimum.

A daemon below `proto_min` is welcomed but receives only `daemon.upgrade` until it runs a supported version. See [Protocol](/docs/daemon/protocol).

## RATE\_LIMITED [#rate_limited]

HTTP 429.

Too many requests of this kind, from the caller or to a provider behind the call.

Wait `retryAfterMs` when the error carries it, then retry. The TypeScript client does this once when created with `retryRateLimited` (waits up to 5 seconds by default).

## DRIVER\_ERROR [#driver_error]

HTTP 502.

A driver, a node command or a site agent failed, or answered with something compute cannot trust: a BMC operation that failed, an unattested result, or firmware and BIOS settings that did not take.

The message names the step; `details` may carry the driver's own code (`driverCode`).

## PROVIDER\_ERROR [#provider_error]

HTTP 502.

A VPS or bare-metal provider's API refused or failed the call.

The message carries the provider's reason. Background work such as instance bootstraps retries it; a direct call does not.

## INVALID\_CONFIG [#invalid_config]

HTTP 500.

The deployment is configured wrongly, for example an option out of range or a module that needs another one.

This describes the server, not the request, so it answers 500. Fix the configuration and restart.

## NOT\_IMPLEMENTED [#not_implemented]

HTTP 501.

The method or feature is planned for a later phase and not built in this release.

`details` names the `feature` and the `phase` that delivers it. See the [roadmap](/docs/reference/roadmap).

## CSRF\_REJECTED [#csrf_rejected]

HTTP 403.

A POST lacks `content-type: application/json` or the `X-Better-Compute: 1` header, or a cookie-authenticated request has no `Origin`.

Browsers cannot send that header cross-site without a preflight, so it stops cross-site request forgery. Clients built on the SDK send it.

## UNTRUSTED\_ORIGIN [#untrusted_origin]

HTTP 403.

A cookie-authenticated request, or a tunnel's WebSocket upgrade, came from an `Origin` other than the deployment's own.

## METHOD\_NOT\_ALLOWED [#method_not_allowed]

HTTP 405.

The HTTP method is not accepted on this path.

API methods are `POST` only; `GET` serves `/health` and `/.well-known/better-compute`; tunnels need a WebSocket upgrade.

## PAYLOAD\_TOO\_LARGE [#payload_too_large]

HTTP 413.

The request body is larger than the server accepts.

## INTERNAL\_ERROR [#internal_error]

HTTP 500.

An unexpected failure on the server.

The server logs it with the request id (`x-request-id`); quote that id when reporting it.

## MISSING\_DEPENDENCY [#missing_dependency]

HTTP 500.

An optional package or service the deployment needs for this call is not installed or not configured, such as PGlite for an embedded database or a remote IAM endpoint.

Like `INVALID_CONFIG`, it describes the deployment, so it answers 500.

## CONFIG\_DRIFT [#config_drift]

HTTP 409.

The fleet changed after the plan was made, so `fleet.apply` refused a plan whose `planHash` no longer matches.

Plan again and review the new plan. `--fail-on-drift` in the CLI uses the same code. See [Fleet configuration](/docs/guides/fleet-config).

## UNAVAILABLE [#unavailable]

HTTP 503.

A dependency the request needs is unreachable for now: the secret store's KMS, a node's gateway, or remote IAM.

Nothing was changed. Retry after `retryAfterMs`.

## TENANT\_SUSPENDED [#tenant_suspended]

HTTP 403.

The IAM tenant is suspended, so its mutations are refused, or deleted, so every call is refused.

`details` carries the `tenantId` and its `state`. A suspension or deletion also covers the tenant's descendants, and root-tenant callers are exempt. Reads still work in a suspended tenant. See [Tenancy](/docs/guides/iam/tenancy).


# Events (/docs/reference/events)

> The 52 event families the control plane publishes to webhooks, the SSE watch stream and sinks.



Every event is `{ id, type, tenantId, subject, data, time }`, delivered at least once in commit order. Subscriptions name a type or a family (`node.*`); a pattern nothing in this catalog can match is refused, and `webhooks.eventTypes` returns the same list.

| Type                               | Subject                       | Module            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ---------------------------------- | ----------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `site.*`                           | `site/{id}`                   | `server`          | Sites created, updated and deleted                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `project.*`                        | `project/{id}`                | `server`          | Projects created, changed and archived (project.archived: \{ projectId, name })                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `tenant.*`                         | `tenant/{id}`                 | `server`          | IAM tenant mirrored as suspended, deleted or active again (tenant.state-changed: \{ tenantId, state, previous, reason }); a deleted tenant fully released by every module (tenant.torn-down)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `pool.*`                           | `pool/{id}`                   | `capacity`        | Pools created, updated, offered, unoffered and archived                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `pool.machines.*`                  | `pool/{id}`                   | `vps`             | Discovered provider machines bound to or unbound from a provider pool (pools.bindMachines, pools.unbindMachines): bound and unbound \{ poolId, accountId, externalIds }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `quota.updated`                    | `pool/{id}`                   | `capacity`        | A tenant quota on a pool changed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `quota.budget-exceeded`            | `pool/{id}`                   | `capacity`        | A tenant's IAM budget was spent and the pool offer's onBudgetExceeded applied (notify, stop-new, drain or suspend)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `quota.budget-suspended`           | `pool/{id}`                   | `capacity`        | A tenant's IAM budget was spent and the pool offer's onBudgetExceeded: 'suspend' suspended the offer: \{ poolId, budgetId }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `reservation.*`                    | `reservation/{id}`            | `capacity`        | Reservation lifecycle (placed, active, released, ended, …) and awaiting-confirmation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `reservation.reassigned`           | `reservation/{id}`            | `capacity`        | An offboarded identity's reservation moved to its successor (§12.7): \{ reservationId, from, to, reason }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `node.*`                           | `node/{id}`                   | `server`          | Node lifecycle and connectivity: joined, acceptance.\*, adopted, connected, disconnected, unreachable, lost, drained, deleted, status, command.queued, ssh-cert.issued, report-failed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `node.command.cancel-requested`    | `node/{id}`                   | `server`          | Cancelling a command was requested (\{ nodeId, commandId, kind, state })                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `node.condition.changed`           | `node/{id}`                   | `inventory`       | A hygiene or pressure condition was raised or cleared (DriverMismatch, CertExpiring, DaemonOutdated, Cpu/Memory/DiskPressure): \{ condition, status, reason, noSchedule }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `node.remediation.*`               | `node/{id}`                   | `provision`       | Health remediation started, succeeded or escalated to an operator                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `pool.remediation.paused`          | `node/{id}`                   | `provision`       | The remediation circuit breaker held a pool back (too many nodes turned unhealthy)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `node.inventory.changed`           | `node/{id}`                   | `inventory`       | A node reported a different inventory                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `node.gpu.*`                       | `node/{id}`                   | `inventory`       | A GPU appeared or went missing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `node.mig.*`                       | `node/{id}`                   | `inventory`       | A MIG layout changed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `node.disk.*`                      | `node/{id}`                   | `inventory`       | A disk was replaced                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `node.driver.*`                    | `node/{id}`                   | `inventory`       | A driver changed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `node.enrolled`                    | `node/{id}`                   | `metal`           | A bare-metal machine enrolled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `node.pxe-mac.discovered`          | `node/{id}`                   | `metal`           | PXE discovery found the MAC of a machine enrolled without MACs, first seen under its product UUID right after metal power-cycled it: a candidate for metal.confirmMac, or bound at once on a pool that trusts its provisioning network (\{ nodeId, mac, productUuid, siteId, agentNodeId, bound })                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `node.pxe-mac.confirmed`           | `node/{id}`                   | `metal`           | An operator bound a machine's PXE MAC with metal.confirmMac (\{ nodeId, mac, discovered, requestId })                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `node.provision.*`                 | `node/{id}`                   | `metal`           | Provisioning state transitions of a bare-metal node                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `command.settled`                  | `command/{id}`                | `gateway`         | A command reached a terminal state                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `workflow.*`                       | `workflow/{id}`               | `provision`       | Provisioning workflow transitions                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `instance.*`                       | `instance/{id}`               | `vps`             | VPS instance lifecycle, and updated (\{ changed }: the name, metadata, labels or issuedBy fields instances.update changed)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `provider.machine.*`               | `provider-account/{id}`       | `vps`             | Machines a provider account can see (vps.discovery): discovered \{ accountId, externalId, machineType, gpus }, changed \{ accountId, externalId, changes }, gone \{ accountId, externalId, instanceId? }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `sandbox.*`                        | `sandbox/{id}`                | `vps`             | Sandbox lifecycle (§3.7): created, ready (its instance first runs), extended, expiring (once, before expiry: \{ expiresAt }), expired (before the delete), withdrawn (its agent or delegation ended, or its owner left without a successor: \{ reason }), reassigned (\{ from, to, reason }), updated (\{ changed }: name, metadata or labels), deleted; data \{ templateId, instanceId, poolId }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `sandbox.snapshot.*`               | `sandbox/{id}`                | `vps`             | A sandbox snapshot settled: ready \{ snapshotId, includeMemory, sha256, sizeBytes } or failed \{ snapshotId, reason }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `provider.reservation.*`           | `provider-account/{id}`       | `vps`             | Provider capacity reservations seen by the vps.reservations worker: detected \{ accountId, reservationId, machineType, gpus?, total, state, startsAt? }, changed (+ free, inUse, changes), ready (a scheduled or creating reservation became ready), gone (+ poolId, flavorId when it was bound)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `provider.purchase.*`              | `provider-account/{id}`       | `vps`             | Provider capacity purchases (AWS A3): requested \{ accountId, purchaseId, offeringId }, completed (+ reservationId, state), failed (+ code), unknown (+ code: the attempts ran out while one may have bought; check the account); proposed \{ accountId, proposalId, reservationId, poolId, offeringId, startsAt, upfrontPrice, currency } (AWS A10: capacity proposed 48 h before a pending reservation of a provider pool starts; nothing is bought)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `billing.*`                        | `provider-account/{id}`       | `vps`             | Provider costs read by the vps.costs worker (D10): cost-ingested once per pass per account \{ accountId, from, to, currency, rows, inserted, corrected, unchanged, stale, costMicros, creditsMicros, unattributedRows, currencyMismatchRows, passedThroughRows, passedThroughMicros, leaks }; unattributed once per stored row with no tenant \{ accountId, sourceRowId, service, sku, resourceName?, usageStart, costMicros, creditsMicros, currency }; cost-corrected when a re-exported row changed its cost or credits \{ accountId, sourceRowId, previousMicros, costMicros, previousCreditsMicros, creditsMicros, currency, passedThrough }; leak at most once per instance and UTC day of a platform account when the distinct hour slots the provider billed compute for exceed those a running interval of the instance covers by more than leakToleranceHours (slots, not summed quantities; instances billed none are not checked) \{ accountId, instanceId, tenantId, day, providerHours, meteredHours } |
| `reservation-grant.*`              | `reservation-grant/{id}`      | `vps`             | Reservation grants (root commits reservation slots to a tenant), published in the grantee tenant: created (+ accountId, reservationId), active, ended, cancelled (+ reason: reservation-gone when the provider stopped reporting the reservation); data \{ grantId, poolId, projectId?, slots, startsAt, endsAt }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `reservation-grant.capacity-short` | `provider-account/{id}`       | `vps`             | A reservations scan saw a reservation's total drop below the slots its live grants hold (at their peak overlap), published in the account's (platform) tenant once per change; the grants keep billing \{ accountId, reservationId, total, grantedSlots, grantIds }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `network.*`                        | `network/{id}`                | `vps`             | Tenant networks (VPCs): created, adopted (an existing provider VPC, networks.adopt: \{ externalNetwork, sharedTagScoped }), ready, error, deleting, deleted; network.subnet.ready and network.rule.ready/error when a subnet or firewall rule converges; network.filesystem.created/ready/error/deleting/deleted for its shared filesystems \{ filesystemId, ... }; (edge) network.natGateway.ready/error, network.peering.ready/error and network.route.ready/error \{ edgeId, subnetId }; data \{ networkId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `providerVolume.*`                 | `provider-volume/{id}`        | `vps`             | Provider volumes (DO P2): created \{ poolId, providerAccountId, name, region, sizeGiB, sourceSnapshotId?, instanceId? }, updated (an attach, detach or resize asked: \{ attach \| detach \| sizeGiB, previousSizeGiB }), ready, attached and detached \{ instanceId }, resized \{ sizeGiB }, error \{ message }, deleting, deleted; data \{ volumeId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `providerVolume.snapshot.*`        | `provider-snapshot/{id}`      | `vps`             | Provider volume snapshots: created \{ volumeId, name }, ready, error \{ message }, deleting, deleted; data \{ snapshotId, volumeId }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `providerAddress.*`                | `provider-address/{id}`       | `vps`             | Reserved provider addresses (DO P2): created \{ poolId, providerAccountId, name, region, family }, updated (\{ assign \| unassign }), ready \{ address }, assigned \{ instanceId }, unassigned, error \{ message }, deleting, deleted; data \{ addressId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `providerCertificate.*`            | `provider-certificate/{id}`   | `vps`             | TLS certificates on a provider account (DO P4): created \{ poolId, providerAccountId, name, kind }, ready, error \{ message }, deleting, deleted; data \{ certificateId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `providerLoadBalancer.*`           | `provider-load-balancer/{id}` | `vps`             | Load balancers on a provider account (DO P4): created \{ poolId, providerAccountId, name, scope, region? }, updated \{ changed }, ready \{ address? }, error \{ message }, deleting, deleted; data \{ loadBalancerId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `providerDnsZone.*`                | `provider-dns-zone/{id}`      | `vps`             | DNS zones on a provider account (DO P4): created \{ poolId, providerAccountId, zone, adopt }, ready, error \{ message }, deleting, deleted; data \{ zoneId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `providerDnsRecord.*`              | `provider-dns-record/{id}`    | `vps`             | DNS records (DO P4): created \{ zoneId, name, type, instanceId? }, updated \{ changed }, ready, error \{ message }, deleting \{ instanceId? }, deleted; data \{ recordId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `kubernetes.cluster.*`             | `kubernetes-cluster/{id}`     | `kubernetes`      | Managed Kubernetes clusters: created \{ poolId, providerAccountId, name, mode, location }, ready the first time \{ endpoint?, version? }, updated when ready again after a change or on another version \{ version?, previousVersion? }, error \{ message }, deleting (+ reason: tenant-deleted), deleted; credentials-issued when a kubeconfig is minted \{ role, subject, identityId, ttlSeconds, expiresAt } (never the token); data \{ clusterId, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `kubernetes.nodepool.*`            | `kubernetes-cluster/{id}`     | `kubernetes`      | Node pools of managed Kubernetes clusters, published on their cluster: created, ready, error \{ message }, deleted; data \{ clusterId, nodePoolId, name, ... }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `cloud.resource.*`                 | `cloud-resource/{id}`         | `cloud-resources` | Managed data and platform services (databases, apps, functions, …): created \{ poolId, providerAccountId, family, name, region? }, ready the first time \{ outputs, secretOutputs } (output names only, never a value), updating \{ fields }, updated when ready again after a change, failed \{ message }, deleting \{ reason: requested \| parent-deleted \| tenant-deleted }, deleted; adopted \{ poolId, providerAccountId, family, name, providerId } (cloudResources.adopt), releasing and released (cloudResources.release: left running, unmanaged); data \{ resourceId, kind, ... }                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `kubernetes.substrate.*`           | `pool/{id}`                   | `kubernetes`      | Pool substrates (policy.substrate: the managed Kubernetes cluster behind a pool): created \{ providerAccountId, mode }, ready \{ clusterId, nodes }, error \{ message }, token-issued \{ tokenId, maxUses, ttlSeconds } and token-revoked \{ tokenId, reason } for the hosted join token bcd joins with (never the token itself), node-removed \{ nodeId, kubernetesNode, released, requeued, retired } when a pool node's Kubernetes Node is gone, its reservations ended and the node was retired, draining \{ version, bookings } and upgrading \{ version, drained } for upgrades: drain, deleting, deleted \{ retiredNodes }; data \{ poolId, ... }                                                                                                                                                                                                                                                                                                                                                             |
| `fleet.*`                          | `fleet/{id}`                  | `fleet`           | FleetConfig converged or drifted                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `rollout.*`                        | `rollout/{id}`                | `rollout`         | Daemon rollouts started, paused, resumed, aborted, completed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `tunnel.*`                         | `tunnel/{id}`                 | `server`          | Tunnels issued and closed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `security.*`                       | `varies`                      | `server`          | Join tokens, site tokens, refused joins, revoked certificates, secrets                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `webhook.ping`                     | `webhook/{id}`                | `events`          | A test delivery from webhooks.ping                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |


# Glossary (/docs/reference/glossary)

> Every term the Better Compute docs use, from atoms and bookings to leases, NodeSpecs and exec tickets, with a link to the page that explains it.



Better Compute borrows words from Kubernetes, OpenStack Ironic and cloud APIs, and gives a few of them a narrower
meaning. "Lease", for example, is never used alone: it is an allocation lease, a liveness lease, or a shard, pool or
site lease, and each is a different thing. This page defines every term the guides mark as a glossary term, one or
two sentences each, with a link to the page that explains it in depth.

A dotted underline in any guide marks such a term: hover or tap it for the definition, then follow **Glossary** to
land on its entry here. Each entry's anchor is its name in lower case with hyphens, for example
`#allocation-lease`.

### Admission

The request-as-resource path every `create` of a reservation or instance takes: the request is written as a draft, Better IAM decides on that draft with its attributes, and only an allowed draft is placed. A retry with the same idempotency key replays the stored result.

See [Admission](/docs/guides/capacity/reservations).

### Admission context

The IAM decision context recorded with each allowed mutation in `audit_events`: identity, session kind, agent, delegation and request id. Better IAM does not audit allows, and asynchronous work acts on this record, never on a stored token.

See [Admission context](/docs/guides/iam).

### Allocation

A placement on one node, owned by a reservation or an instance, listing the atoms and scalar amounts it holds. A gang has one allocation per node.

See [Allocation](/docs/guides/capacity/reservations).

### Allocation lease

A renewable hold on a reservation: one made with `leaseSeconds` is released unless `reservations.renew` is called within every period.

See [Allocation lease](/docs/guides/capacity/reservations).

### Atom

A device with `bookable: true`, the only kind ever booked: a whole GPU, a MIG instance, a time-slice or MPS slot, a NIC virtual function, a disk or a whole node. A partitioned GPU's parent is not an atom.

See [Atom](/docs/guides/capacity).

### Attestor

A plug-in of JoinService that checks one kind of join evidence and decides whether a machine may join and as which node: `join-token`, `x509pop`, `provisioned`, `discovery`, `cloud-iid` or `tpm`.

See [Attestor](/docs/guides/nodes/attestation).

### Authz projection

Compute's own rows describing every `compute-*` resource Better IAM may decide on (type, id, tenant, parent, owner and attributes), written in the same transaction as the change and held in memory. The resolver answers IAM from it, and list queries filter on the same rows.

See [Authz projection](/docs/guides/iam/actions-and-resources).

### bcd

The Better Compute daemon: one Go binary that runs on servers (`node`), site agents (`site`) and inside the PXE ramdisk (`ramdisk`), dials out to a gateway, runs commands and converges its NodeSpec. None of it is released. Its v0.1 node agent (`bcd join`, `bcd node`, `bcd status`) is tested end to end; `bcd site` and `bcd ramdisk` are written and covered by Go unit tests only; `bcd sim` runs many real node agents in one process on fake-hardware builds only; a host build with `virsh` runs VPS instances on the `libvirt-kvm` runtime, tested against a fake `virsh` only; self-update with A/B slots and the WebSocket fallback are written; and the TypeScript `FakeDaemon` stands in where bcd is missing.

See [bcd](/docs/daemon).

### BMC

The baseboard management controller: the part of a server that works when the server does not, used to power it, choose its boot device, mount media and read hardware. Metal reaches every BMC through a site agent, never from the control plane.

See [BMC](/docs/guides/bare-metal/bmc).

### Booking

One row `(device_id, allocation_id, during)` in `device_bookings`. An `EXCLUDE` constraint allows one live booking per atom per instant, which is why double-booking cannot happen.

See [Booking](/docs/guides/concepts/consistency).

### BookingRepair

The check that re-places a calendar reservation whose booked hardware went bad (cordoned, quarantined or unreachable), run 15 minutes before its start and again at the start.

See [BookingRepair](/docs/guides/capacity/gang-and-calendar).

### Calendar reservation

A reservation with a `start` in the future. Its atoms are booked at admission for the window from `start` to `start + duration`, so nobody else can take them for that window.

See [Calendar reservation](/docs/guides/capacity/gang-and-calendar).

### Capacity

What the inventory makes bookable, per pool and resource: allocatable, allocated, reserved and available. Only schedulable nodes count as allocatable.

See [Capacity](/docs/guides/capacity).

### CDI

The Container Device Interface. For a `hold` reservation the node writes a CDI spec naming exactly the booked GPUs, which a workload manager on the node (containerd, Kubernetes, Slurm) consumes.

See [CDI](/docs/guides/capacity/reservations).

### Command

A durable, idempotent one-shot message to one node, with a per-node `seq`, `not_before`, `not_after`, an optional `min_generation` and the flags of its kind. Commands are for actions; lasting state belongs in the NodeSpec.

See [Command](/docs/guides/nodes/commands).

### Command kind

The registered type of a command, such as `node.reboot` or `diag.nvml`: its lane, the agent roles that accept it, a timeout, a payload message, and the flags retry-safe, self-terminating, attested and destructive.

See [Command kind](/docs/reference/command-kinds).

### compute_now()

The SQL function every query uses for the current time instead of `now()`, so the whole control plane reads one database clock. Tests point it at a simulated clock, and a repository lint refuses bare `now()`.

See [compute_now()](/docs/guides/concepts/consistency).

### Condition

A typed statement about a node or GPU, such as `GpuCountMismatch`, `NodeUnreachable` or `ClockSkew`, written by the health, inventory and liveness controllers. A condition with a NoSchedule effect keeps the node out of placement.

See [Condition](/docs/guides/nodes/inventory-and-health).

### Config unit

One concern of a machine's host configuration in its NodeSpec `host` section, such as `nvidia-driver`, `mig-layout` or `hugepages`: a unit name and validated settings. Profiles' `hostConfig` and pool policies produce them; bcd's host converger applies them in a fixed order.

See [Config unit](/docs/daemon/host-config).

### Cordon

A flag that stops new placements on a node and changes nothing else: existing allocations keep running and commands still flow. `nodes.uncordon` clears it.

See [Cordon](/docs/guides/nodes/operating-nodes).

### Delegated confirmation

When an agent acting under a Better IAM delegation is denied an action the person held back, compute answers `CONFIRMATION_REQUIRED` instead of a plain denial; the person approves in Better IAM and the agent retries the same request.

See [Delegated confirmation](/docs/guides/iam/delegated-confirmation).

### Destructive-batch quorum

The v1.0 control against a compromised api or worker: destructive commands carry a `batch_id` and run only when enough operator approver keys, pinned in the trust bundle and held off the control plane, signed a `BatchApproval` over the Merkle root of the batch's (subject, kind) scope, which the machine checks with the command's inclusion proof. bcd and the FakeDaemon enforce it, the server stores, approves and holds batches (`commandBatches`, `enqueue({ batchId })`), the gateway attaches each batched command's approval and proof, and the built-in workflows, `metal.bulk` and `metal.power` create their own batches.

See [Destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10).

### Device

A unit of hardware with typed attributes, in the style of Kubernetes dynamic resource allocation: class `gpu`, `gpu.mig`, `gpu.slot`, `nic.vf`, `disk` or `node`, with a stable id such as a GPU UUID, disk WWN or MAC.

See [Device](/docs/guides/nodes/inventory-and-health).

### Draft

The first state of a reservation or instance, written before IAM decides so that policies can read exactly what is asked for. Its id derives from the tenant, the caller's identity and the idempotency key, so a retry finds the same row.

See [Draft](/docs/guides/capacity/reservations).

### Drain

A cordon that also waits for the node to empty: `nodes.drain` cordons the node, adds the `draining` NoSchedule reason and hands it to the Drain controller.

See [Drain](/docs/guides/nodes/operating-nodes).

### Drift

A difference between the fleet and the FleetConfig document last applied. The `fleet.drift` reconciler re-plans the applied document every 15 minutes by default and publishes `fleet.drifted` or `fleet.converged`.

See [Drift](/docs/guides/fleet-config).

### Driver

An implementation of one or more hardware concerns in the Ironic sense: `power`, `management`, `boot`, `deploy`, `inspect`, `console`, `virtual-media`, `bios`, `raid`, `firmware`, `config` or `runtime`. A node binds one driver per concern, inherited from its pool's hardware profile unless overridden.

See [Driver](/docs/extending/drivers).

### Epoch

A counter that fences a lease or session: every takeover increments it, and writes carrying an older epoch are refused, so a stalled worker, site agent or gateway cannot act after losing ownership.

See [Epoch](/docs/daemon/sessions).

### EXCLUDE constraint

The PostgreSQL constraint `EXCLUDE USING gist (device_id WITH =, during WITH &&)` on `device_bookings`. It lets at most one live booking exist per atom per instant, whatever the placer or the API does.

See [EXCLUDE constraint](/docs/guides/concepts/consistency).

### Exec ticket

A single-use ES256 JWT (`typ: bc-exec+jwt`) that `tunnels.open` issues for one node, one tunnel and one caller, valid for at most 60 seconds. The API, the gateway and the daemon each verify it before any byte flows.

See [Exec ticket](/docs/guides/access/tunnels).

### FakeDaemon

The TypeScript implementation of the daemon side of the protocol in `@better-compute/testing`, with the same journal rules, acceptance checks and lanes as bcd. It runs in tests, the fleet simulator and `better-compute dev --sim N`, and covers what bcd does not implement yet: VM hosts and their metadata service, site agents, self-update and the WebSocket fallback.

See [FakeDaemon](/docs/operations/testing).

### Flavor

A named instance shape such as `g1.small`, defined by a pool's owner and picked by name when creating an instance. In a hosted pool it becomes claims on one host; in a provider pool it names the provider's offer.

See [Flavor](/docs/guides/vps/flavors).

### FleetConfig

The fleet's shape (sites, provider accounts, pools, quotas, offers and flavors) as one document keyed by name, exported, planned and applied in one transaction with `fleet.export`, `fleet.plan` and `fleet.apply`.

See [FleetConfig](/docs/guides/fleet-config).

### Gang

A reservation of 2 to 64 whole nodes that is placed all or nothing, optionally on one value of a topology label such as `fabric.domain`. It has one allocation per node and counts as one request for quota and IAM.

See [Gang](/docs/guides/capacity/gang-and-calendar).

### Gateway

The process that holds daemons' long-lived gRPC sessions over mutual TLS, delivers desired state and commands, and writes results and events to PostgreSQL. It keeps only soft state, so losing one loses open streams and nothing else.

See [Gateway](/docs/daemon/gateway).

### Generation

The version number of a node's NodeSpec, incremented only when the spec's canonical JSON changes; adoption writes generation 1. The daemon reports `observedGeneration`, and a node is schedulable only once it has converged at least once.

See [Generation](/docs/guides/concepts/desired-state-and-commands).

### Hardware key

The SHA-256 of a machine's normalized product UUID, serial number, permanent MACs and GPU UUIDs, used to recognize the same machine across joins. It is trusted only when at least two independent identifiers survive the placeholder blocklist.

See [Hardware key](/docs/guides/nodes/joining).

### IamPort

The interface compute calls for every IAM question: `authenticate`, `require`, `authorize`, `authorizeMany`, `requestConfirmation`, `billingCheck`, `recordUsage` and `featureEnabled`. The IAM bridge implements it over an embedded or a remote Better IAM.

See [IamPort](/docs/guides/iam).

### Image

A signed, content-addressed OS or ramdisk artifact (`name@version`, `raw`, `qcow2` or `iso`, with a SHA-256). It is `verified` once its signature checks out against a signer allowlist, and unsigned images stay private.

See [Image](/docs/guides/bare-metal/profiles-and-images).

### Instance

A VM, microVM or single-tenant container in a VPS pool, with a `desiredState` (`running`, `stopped` or `deleted`) and an observed `state` that the `vps.instances` reconciler drives toward it.

See [Instance](/docs/guides/vps/instances).

### Inventory

The node's `NodeInventory v1` document: hardware facts in sections, each hashed so the daemon sends only what changed, stored as append-only snapshots. It produces the node's devices, labels and conditions.

See [Inventory](/docs/guides/nodes/inventory-and-health).

### Join token

A secret of the form `bcjt_<id>.<secret>` that lets a machine join one pool, for a limited time (`ttlSeconds`) and a limited number of machines (`maxUses`, default 1). It is shown once and handed to the machine out of band.

See [Join token](/docs/guides/nodes/joining).

### JoinService

The control-plane service that admits machines: it runs the attestors on the join evidence, creates or reclaims the node row, and signs a short-lived node certificate, all in one transaction.

See [JoinService](/docs/guides/nodes/joining).

### Journal

The daemon's local, bounded record of every command it accepted and every result and event not yet acknowledged, keyed by a `journal_id`. It makes sure no side effect runs twice; a lost journal starts a new id, and the server fails in-flight commands instead of replaying them.

See [Journal](/docs/daemon/delivery).

### Lane

One of the three delivery channels a command kind belongs to, `control`, `provision` or `bulk`, each with its own credits and a fixed pool of workers on the daemon, so bulk work never blocks control commands.

See [Lane](/docs/daemon/delivery).

### Liveness lease

A session's 40-second time to live, extended by heartbeats. When it lapses the node becomes unreachable, but nothing it holds is released.

See [Liveness lease](/docs/daemon/sessions).

### Maintenance

A node flag, separate from cordon, set when provisioning fails, remediation gives up or an operator asks. It keeps the node out of placement until cleared.

See [Maintenance](/docs/guides/nodes/operating-nodes).

### MIG

NVIDIA Multi-Instance GPU, which splits one GPU into isolated instances. A GPU in MIG mode becomes an unbookable parent plus one `gpu.mig` atom per instance, and only `mig` pools accept requests for them.

See [MIG](/docs/guides/capacity/fractional-gpus).

### Node

A machine Better Compute manages: a server running bcd, a site agent, or a provider machine. Its state is several independent axes (provision state, maintenance, cordon, power, connectivity and conditions), not one field.

See [Node](/docs/guides/nodes).

### Node certificate

The short-lived ECDSA P-256 client certificate JoinService signs for a daemon, valid for at most 24 hours (2 hours for a ramdisk), named by a SPIFFE ID and carrying a role: `node`, `site`, `ramdisk` or `node-tenant`.

See [Node certificate](/docs/guides/nodes/joining).

### NodeSpec

A node's generation-numbered desired state, in sections the daemon converges in a fixed order: `host`, `devices`, `instances`, `access`, `telemetry` and `daemon`. Modules contribute sections; the gateway delivers the whole spec or a patch.

See [NodeSpec](/docs/guides/concepts/desired-state-and-commands).

### Offering

A `pool_offers` row that makes a pool visible to another tenant, with a quota per resource and an optional priority. A resource missing from the quota grants zero.

See [Offering](/docs/guides/capacity/pools).

### Outbox

The table every event is written to, in the same transaction as the change it describes, so an event exists if and only if the change committed. Webhooks, the live stream, the history API and in-process subscribers all read it.

See [Outbox](/docs/guides/events).

### Placer

The component that decides where a request lands: which node, which atoms and which scalar amounts. The default inline placer places during admission; the pool-lease placer queues requests and places them in batches.

See [Placer](/docs/guides/capacity/placement).

### Pool

A scheduling and policy domain in one site, owned by one tenant, with exactly one GPU model, a class (`metal-host`, `metal-dedicated`, `vps-host` or `provider`), a sharing mode and policies such as acceptance checks and a disruption budget.

See [Pool](/docs/guides/capacity/pools).

### Profile

A versioned provisioning template for metal or VPS targets, with driver bindings, an image, user data, steps, host configuration and acceptance checks. It is immutable once a workflow has used it.

See [Profile](/docs/guides/bare-metal/profiles-and-images).

### Project

An application-owned grouping inside a tenant (the `projects` table), which reservations and instances can name and policies can read as `projectId`.

See [Project](/docs/guides/iam/tenancy).

### Provider

An external capacity API, such as Hetzner Cloud, Hetzner Robot or Proxmox VE, behind a pool of class `provider`. Instances in such a pool are created through the provider rather than on your own hosts.

See [Provider](/docs/guides/vps/providers).

### Provider account

A configured account with a provider, whose credentials are stored only as secret references, never as values. A provider pool is backed by one account.

See [Provider account](/docs/guides/vps/providers).

### Provision state

A node's lifecycle axis, from `adopting` or `enrolling` through `ready` and `deployed` to `retired`. The legal moves live in one table in `@better-compute/core`, and every verb, method and reconciler checks against it.

See [Provision state](/docs/guides/bare-metal).

### PXE

Network boot. On a bare-metal site the site agent answers proxy DHCP and serves TFTP and iPXE, so an enrolled machine boots into the ramdisk without anyone touching it.

See [PXE](/docs/guides/bare-metal/deploying).

### Quota

A cap on how much of each resource a tenant may hold in one pool at the same time, set by the pool's offering or `quotas.set`, counted in compute's database and charged all or nothing at admission.

See [Quota](/docs/guides/capacity/quotas-and-usage).

### Ramdisk

The PXE-booted initramfs in which bcd runs as `bcd ramdisk` to inspect, burn in, wipe and image a machine. It takes the gateway, pins, a single-use discovery token and the site agent's signed assertion from its kernel command line, joins with the `discovery` attestor, and keeps its identity in tmpfs, so every boot is a new one. Its certificate lasts at most 2 hours and allows only the commissioning command kinds (`disk.wipe`, `deploy.image`, `boot.kexec`, `diag.*`, `inventory.refresh`, `logs.collect`).

See [Ramdisk](/docs/guides/bare-metal/deploying).

### Reconciler

A level-triggered controller that reads the current state, compares it with the desired state and acts on the difference. It is woken through the work queue, sharded across workers, and never relies on having seen every event.

See [Reconciler](/docs/guides/concepts/desired-state-and-commands).

### Region

A geographic label in Better IAM's region-name grammar, stored on a site as `iamRegion` so policies can decide on where capacity is.

See [Region](/docs/guides/concepts/domain-model).

### Release

A manifest of one bcd version's files, signed offline by keys the control plane knows only by their `sha256/...` pins. Rollouts move nodes from one release to another.

See [Release](/docs/guides/rollouts).

### Reservation

A tenant's claim on devices of one pool for a time window: devices with counts, selectors and topology, an optional `start`, a `duration` and a `binding` of `hold`, `instance` or `node`.

See [Reservation](/docs/guides/capacity/reservations).

### Resolver

The function Better IAM calls to learn about a resource it does not store. Compute's resolver answers every `compute-*` type from the authz projection in memory and hands other types to the host's own resolver.

See [Resolver](/docs/guides/iam/actions-and-resources).

### Retry pushback

The delay a busy gateway attaches to a `RESOURCE_EXHAUSTED` refusal as the gRPC trailer `grpc-retry-pushback-ms` (and `retry-after` on the WebSocket path), jittered and scaled to the number of daemons it is refusing. bcd and the FakeDaemon wait exactly that long instead of their own back-off, so a power-on herd spreads over the window the gateway can admit.

See [Retry pushback](/docs/daemon/sessions#opening).

### Rollout

The controlled move of the fleet to one signed bcd release in waves (1%, 10%, 50%, 100% by default), within each pool's disruption budget, pausing by itself when too many nodes of a wave fail.

See [Rollout](/docs/guides/rollouts).

### Sandbox

An isolated place to run untrusted code, served by `sandboxes.create`: one hosted VPS instance made from a per-tenant template (pool, flavor, image), with a time-to-live after which it is deleted, and optionally claimed from the template's warm pool of paused (or stopped) instances. Instance methods refuse a sandbox's instance. It can be snapshotted and restored, and `sandboxes.exec` reaches it over SSH through a tunnel, with no agent in the guest.

See [Sandbox](/docs/guides/vps/sandboxes).

### Sandbox SSH CA

The deployment's SSH certificate authority for `sandboxes.exec`, separate from the break-glass CA: it signs certificates of at most 15 minutes for the principal `<account>@<instanceId>`, and every sandbox instance's user data makes its sshd trust it for its own principal only.

See [Sandbox SSH CA](/docs/guides/vps/sandboxes#run-code-inside).

### Sandbox template

A tenant's recipe for sandboxes (`sandboxTemplates.*`): a hosted pool, a flavor, an image and the pool's runtime, optional user data, `env` as `secret://` references, `network` (`egress`, `private` or `none`), TTL bounds and a warm pool size.

See [Sandbox template](/docs/guides/vps/sandboxes#templates).

### Session

The long-lived gRPC stream a daemon holds to one gateway, opened with `Hello` and answered with `Welcome`. Each new session increments the node's epoch, which fences the old one.

See [Session](/docs/daemon/sessions).

### Shard lease

A lease that gives one worker or agent ownership of something, with a fencing epoch: `shard:<n>` for a slice of reconciler work, and likewise `pool:<id>`, `site:<id>:boot`, `singleton:<job>` and `gateway:<id>`.

See [Shard lease](/docs/guides/concepts/consistency).

### Site

A boot and failure domain served by at least one site agent, of kind `shared`, `dedicated` or `custom` (a tenant's own hardware). Pools, nodes and join tokens each belong to exactly one site.

See [Site](/docs/guides/bare-metal/sites).

### Site agent

bcd running as `bcd site` on a site's provisioning and BMC networks, stored as a node with `role: 'site'` and joined with a site token. It runs BMC operations (Redfish only today) and `site.httpProxy`, serves network boot (proxy DHCP, TFTP, iPXE scripts with signed discovery assertions) only to prepared machines while it holds the site's boot lease, and keeps a digest-verified image cache for the site's ramdisks.

See [Site agent](/docs/guides/bare-metal/sites).

### SiteBoot lease

The lease `site:<id>:boot` that lets exactly one connected site agent serve network boot for its site; the standby is promoted only after it expires, under a new epoch.

See [SiteBoot lease](/docs/guides/bare-metal/sites).

### SPIFFE ID

The URI name in every node certificate, `spiffe://<trustDomain>/site/<siteId>/node/<nodeId>`. Names come only from JoinService, never from the certificate request, and no tenant appears in them.

See [SPIFFE ID](/docs/guides/nodes/joining).

### Spot

A reservation made with `spot: true`: placed like any other and billed as `gpu.<model>.spot.hours`, but preempted with a short notice (2 minutes by default) when a regular request needs its GPUs and nothing else fits.

See [Spot](/docs/guides/capacity/spot-and-preemption).

### Step

One persisted unit of a workflow, run on the `control-plane`, a `site` agent, the `node` or the `ramdisk`. An agent step is exactly one command, and `await: 'rejoin'` marks steps that end the agent's boot.

See [Step](/docs/guides/bare-metal/workflows).

### Taint

A named NoSchedule reason an operator sets on a node (`taint:<key>`), shown next to the reasons controllers write, to keep the node out of placement for a reason that should stay visible.

See [Taint](/docs/guides/nodes/operating-nodes).

### Tenant

A Better IAM `tenantId`: the root tenant is the platform, organizations are customers. Every tenant-scoped record carries its tenant, and a caller outside a request's tenant and the root tenant is refused before any database access.

See [Tenant](/docs/guides/iam/tenancy).

### Tunnel

A two-way byte stream between a caller and a shell, console, SSH server or TCP port on a node, carried over the session the node already holds to its gateway, so the node needs no inbound port.

See [Tunnel](/docs/guides/access/tunnels).

### Usage window

One row per allocation, hour and meter in `usage_windows`, such as `gpu.h100.hours`, written by the usage reconcilers every 5 minutes and sent to Better IAM billing with an idempotency key, so every retry is safe.

See [Usage window](/docs/guides/capacity/quotas-and-usage).

### Warm pool

The instances a sandbox template keeps booted and then `paused` (or `stopped` on a runtime that cannot pause), so `sandboxes.create` can claim one after its own IAM decision without a new placement. Warm instances are charged to the tenant's quota and metered as `vps.sandbox.warm` until claimed.

See [Warm pool](/docs/guides/vps/sandboxes#warm-pools).

### Work queue

The table of `(kind, key)` wake-ups for reconcilers, written in the same transaction as the change that causes them. Enqueueing a key that is already queued bumps its generation instead of adding a row, so no wake-up is lost.

See [Work queue](/docs/guides/concepts/desired-state-and-commands).

### Workflow

A profile expanded for one target into persisted steps, run by the provision engine with retries, timeouts, failure and cleanup steps. A node runs at most one active workflow at a time.

See [Workflow](/docs/guides/bare-metal/workflows).


# Host config units (/docs/reference/host-units)

> The 12 config units a node's spec may set, their settings and what bcd does with them.



Host config units are what a profile's `hostConfig` and a pool's policy turn into the NodeSpec `host` section; provision validates their settings against the manifests below, and bcd's host converger applies them, level-triggered: drift is repaired at the next pass (at least every 10 minutes). bcd converges them in this order, each seeing what the ones before it did: `kernel-modules`, `sysctl`, `hugepages`, `cpu`, `iommu`, `nvidia-driver`, `fabric-manager`, `gpu-settings`, `mig-layout`, `gpu-sharing`, `container-toolkit`, `k8s-join`. A unit that fails never holds back the others; the host section reports every failure, retried while any is. A unit whose state takes effect only after a reboot reports node condition `RebootRequired`: bcd never reboots on its own. Under `bcd helper`, the node plans each unit and the helper applies it as root, only as a spec signed under the pinned roots asks. With `[host] units = false` (bcd as a Kubernetes DaemonSet) no unit is applied. See [Host configuration](/docs/daemon/host-config).

## nvidia-driver [#nvidia-driver]

The NVIDIA driver branch (and the open kernel modules or not).

Installs the driver branch (Ubuntu's server packages, NVIDIA's CUDA repository, DNF module streams, SUSE's G06 packages) and loads it. A different driver already loaded is swapped only while no process holds the GPUs; otherwise the unit reports `RebootRequired`. It keeps `nvidia-persistenced` running where it is installed.

| Field    | Type    | Required | Constraints          |
| -------- | ------- | -------- | -------------------- |
| `branch` | string  | yes      | matches `^[0-9]{3}$` |
| `open`   | boolean |          |                      |

## container-toolkit [#container-toolkit]

The NVIDIA container toolkit, generating CDI specs.

Installs the NVIDIA container toolkit, configures containerd and Docker with its runtime (with `cdi`, CDI enabled), and regenerates `/etc/cdi/nvidia.json` (JSON, GPUs and MIG devices named by UUID; it replaces the `/etc/cdi/nvidia.yaml` older versions wrote) whenever the driver or the GPU and MIG device list changes. A hold's CDI spec copies its devices' nodes, mounts and hooks from it, so `bcd.dev/gpu=<id>` alone is enough. With `cdi` (the default), a container gets GPUs only by the CDI names bcd hands out. The unit sets `accept-nvidia-visible-devices-envvar-when-unprivileged` and `accept-nvidia-visible-devices-as-volume-mounts` to false in `/etc/nvidia-container-runtime/config.toml`. Otherwise an unprivileged container could choose GPUs other tenants hold with `NVIDIA_VISIBLE_DEVICES` (the packaged config leaves that key at its permissive default). On a Kubernetes node that is the `ForeignGpuExposure` condition.

| Field | Type    | Required | Constraints    |
| ----- | ------- | -------- | -------------- |
| `cdi` | boolean |          | default `true` |

## mig-layout [#mig-layout]

A MIG layout: a named profile per GPU, or the pool's layout.

Partitions every MIG-capable GPU to the layout: `all-disabled`, `all-<profile>` (as many as fit, e.g. `all-1g.10gb`) or `mixed:<n>x<profile>+<profile>…` (per GPU, e.g. `mixed:3g.40gb+2x2g.20gb`), with a compute instance in each GPU instance. A GPU holding allocations or running compute processes is left as it is until they end (under `bcd helper`, the helper sees that itself, never taking the node's word). A MIG mode change that needs a GPU reset, which fails, reports `RebootRequired`.

| Field    | Type                     | Required | Constraints           |
| -------- | ------------------------ | -------- | --------------------- |
| `from`   | one of `pool`, `profile` |          | default `"profile"`   |
| `layout` | string                   |          | at most 64 characters |

## fabric-manager [#fabric-manager]

NVIDIA Fabric Manager for NVSwitch systems.

On NVSwitch systems only (nothing to do elsewhere): installs and runs NVIDIA Fabric Manager; node condition `FabricManagerDown`, which makes the node NoSchedule, while it is not running.

Takes an empty object: `{}`.

## sysctl [#sysctl]

Kernel parameters.

Free-form: each key a sysctl name with a scalar value, written to `/etc/sysctl.d/90-bcd.conf` and set live. A key the kernel does not have yet (its module is not loaded) is retried. Keys that would run programs as root or lower the host's defenses are refused (`kernel.core_pattern`, `kernel.modprobe`, `kernel.usermodehelper.*`, `fs.binfmt_misc.*`, `kernel.randomize_va_space`, `kernel.yama.*`, `user.*` and the like), and a few take only their hardened values: `dev.tty.legacy_tiocsti` and `dev.tty.ldisc_autoload` only 0, `kernel.io_uring_disabled` only 1 or 2, `net.ipv4.ip_unprivileged_port_start` only 1024 and above.

Takes free-form keys: sysctl names (`^[a-z0-9_]+(?:\.[a-z0-9_-]+)+$`) with scalar values.

## hugepages [#hugepages]

Huge pages: 2 MiB pages allocated at runtime, 1 GiB pages through the kernel command line (effective after a reboot), and the transparent huge page policy.

2 MiB pages are allocated at runtime; 1 GiB pages at runtime when the kernel finds the memory, else through the boot loader (`hugepagesz=1G hugepages=N`) with `RebootRequired`. With `perNuma` the counts are per NUMA node. 1 GiB pages are mounted at `/dev/hugepages1G` (a systemd mount unit), 2 MiB pages at systemd's `/dev/hugepages`.

| Field         | Type                                | Required | Constraints |
| ------------- | ----------------------------------- | -------- | ----------- |
| `pages2m`     | integer                             |          | 0-4194304   |
| `pages1g`     | integer                             |          | 0-16384     |
| `transparent` | one of `always`, `madvise`, `never` |          |             |
| `perNuma`     | boolean                             |          |             |

## gpu-sharing [#gpu-sharing]

How a shared pool's GPUs are split between clients: MPS (bcd-mps.service, 100/replicas % of the SMs each) or time-slicing (MPS stopped).

`mps` runs the MPS control daemon (`bcd-mps.service`, each client limited to 100/`replicas` percent of the SMs, node condition `MPSDown` while it is down); `timeslice` stops it.

| Field      | Type                      | Required | Constraints |
| ---------- | ------------------------- | -------- | ----------- |
| `mode`     | one of `mps`, `timeslice` | yes      |             |
| `replicas` | integer                   |          | 1-48        |

## cpu [#cpu]

Host CPUs: the cpufreq governor (live), and CPUs isolated from the kernel scheduler for pinned VMs and holds (isolcpus/nohz\_full/rcu\_nocbs, effective after a reboot).

Sets the frequency governor on every CPU that has one, and isolates CPUs from the scheduler for pinned vCPUs (`isolcpus`, `nohz_full`, `rcu_nocbs` through the boot loader, `RebootRequired` until the kernel shows them isolated). CPU 0 is never isolated.

Takes an empty object: `{}`.

## iommu [#iommu]

The IOMMU for GPU passthrough to VMs and sandboxes: intel\_iommu=on or amd\_iommu=on (by CPU vendor) and iommu=pt on the kernel command line, effective after a reboot.

Turns the IOMMU on for GPUs passed through to VMs and sandboxes (`intel_iommu=on` or `amd_iommu=on`, and `iommu=pt` unless `passthrough` is false) through the boot loader, with `RebootRequired` until an IOMMU is active.

| Field         | Type    | Required | Constraints    |
| ------------- | ------- | -------- | -------------- |
| `passthrough` | boolean |          | default `true` |

## gpu-settings [#gpu-settings]

Per-GPU settings kept by bcd: the power cap (nvidia-smi -pl, re-applied on drift) and ECC (effective after a reboot).

Caps each GPU's board power (applied again whenever it drifts; outside a GPU's range the unit fails) and sets the ECC mode, which changes at the GPU's next reset (`RebootRequired` until then).

| Field             | Type    | Required | Constraints |
| ----------------- | ------- | -------- | ----------- |
| `powerLimitWatts` | integer |          | 1-10000     |
| `ecc`             | boolean |          |             |

## kernel-modules [#kernel-modules]

Kernel modules loaded at boot and now (modules-load.d), with their options (modprobe.d): kvm\_intel/kvm\_amd, vhost\_net and tun for VM hosts.

Modules loaded now and at every boot (`/etc/modules-load.d/bcd.conf`) with their options (`/etc/modprobe.d/bcd.conf`). A loaded module whose options differ is reloaded only when nothing uses it; otherwise they apply when it next loads. Sandbox hosts load `kvm_intel` or `kvm_amd`, `vhost_net`, `tun` and, for GPU passthrough, `vfio-pci`. Options that turn off DMA isolation for passed-through devices (`vfio` `enable_unsafe_noiommu_mode`, `vfio_iommu_type1` `allow_unsafe_interrupts`) are refused.

Takes an empty object: `{}`.

## k8s-join [#k8s-join]

Joins the node to a Kubernetes cluster as a worker (kubeadm join with a bootstrap token fetched by reference) and keeps its kubelet, labels and taints; GPUs through containerd's NVIDIA runtime and CDI.

Joins the machine to a Kubernetes cluster as a worker with a bootstrap token fetched through the secret broker, only to clusters root pins in `[kubernetes] allowed_clusters` under `bcd helper`. See [Kubernetes](/docs/daemon/kubernetes).

Takes an empty object: `{}`.


# Reference (/docs/reference)

> Generated reference for the API, the CLI, command kinds, error codes, the daemon protocol, events, modules and packages, from a live control plane and the .proto files.



Most of this section is generated from the repository by `apps/docs/scripts/generate.mjs`, so it matches the code
it documents. The generator does not parse source files. It imports the built packages and constructs a real control
plane with `betterCompute()` and **every** module registered: the built-in node controllers, `inventory` and
`capacity`, plus `provision` with `metal`, `vps`, `events`, `fleet`, `rollout` and `audit`. That control plane runs
on a PGlite executor that is never opened and an `IamPort` that refuses every call, so nothing is migrated and no
decision is made. What it reads is the registry a full deployment serves, plus the `.proto` files for the protocol
page:

| Page                                           | Generated from                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [API](/docs/reference/api)                     | The method registry, joined with PLAN.md §9.2 (core's `apiSurface`), so every method shows as `registered` (served by this build) or `planned` with its phase                                                                                                                                                                                                                                                                                                                                                        |
| [CLI](/docs/reference/cli)                     | `cliManifest()` of `@better-compute/cli`, the same data `better-compute help --json` prints                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [Command kinds](/docs/reference/command-kinds) | `registry.commandKinds`: lane, roles, retry-safe, self-terminating, attested and destructive flags, timeout and payload message                                                                                                                                                                                                                                                                                                                                                                                      |
| [Errors](/docs/reference/errors)               | Core's `errorCodes` and `defaultStatus`                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| [Protocol](/docs/reference/protocol)           | The `.proto` sources under `proto/bettercompute`, read by `apps/docs/scripts/protocol.mjs` (5cafb1e): every service, RPC, message and enum of `commands.proto`, `inventory.proto`, `session.proto`, `common.proto`, `driver.proto` and `site.proto`, with field numbers, labels, types linked to their message or enum, and each declaration's own comment. It reads the files rather than the compiled descriptors, which drop comments, and was checked against them (157 messages, 25 enums, equal field numbers) |
| [Events](/docs/reference/events)               | `eventCatalog` of `@better-compute/events`: every event family with its type, subject and module, the same list `webhooks.eventTypes` returns                                                                                                                                                                                                                                                                                                                                                                        |
| [Metrics](/docs/reference/metrics)             | `metricCatalog()` of `@better-compute/observability`: every Prometheus family with its type, labels, help and buckets, grouped by the function that defines it                                                                                                                                                                                                                                                                                                                                                       |
| [Modules](/docs/reference/modules)             | The registry's modules, their API groups and migrations, and every job in `compute.jobs`                                                                                                                                                                                                                                                                                                                                                                                                                             |
| [Packages](/docs/reference/packages)           | Every `packages/*/package.json`: name, description and export subpaths                                                                                                                                                                                                                                                                                                                                                                                                                                               |

The prose on the API and error pages is hand-written and kept beside the generator in `apps/docs/descriptions/`:
`api/<group>.md` holds an overview paragraph and one `## method` section per method, whose first paragraph becomes
the summary, and `errors.md` holds one `## CODE` section per error code. The generator checks those files against
the registry, so a description of a method or code that no longer exists fails the run.

The rest of the section is written by hand: the [glossary](/docs/reference/glossary) (from
`apps/docs/data/glossary.json`, which also feeds the hover cards on every term in the guides), how to
[use these docs with AI](/docs/reference/ai), the [roadmap](/docs/reference/roadmap), and the
[changelog](/docs/reference/changelog), which includes the repository's `CHANGELOG.md` at build time.

  - [API](/docs/reference/api): Every `compute.api` group and method: HTTP route, IAM action and resource, module, phase and status.

  - [CLI](/docs/reference/cli): The `better-compute` command line: `dev`, `serve`, `pki`, `fleet`, `sim`, `audit` and the rest, with flags and environment variables.

  - [Command kinds](/docs/reference/command-kinds): The typed one-shot operations daemons run, with their lanes and safety flags.

  - [Errors](/docs/reference/errors): Every stable error code with its HTTP status and meaning.

  - [Protocol](/docs/reference/protocol): Every service, message and enum of the daemon protocol, with field numbers and types.

  - [Events](/docs/reference/events): Every event type the outbox publishes, for webhooks and the watch stream.

  - [Metrics](/docs/reference/metrics): Every Prometheus series: type, labels, help text and buckets.

  - [Modules](/docs/reference/modules): What each module registers: API groups, migrations and jobs.

  - [Packages](/docs/reference/packages): Every package in the repository and its entry points.

  - [Glossary](/docs/reference/glossary): Atoms, bookings, leases, NodeSpecs and every other term the guides use.

  - [Use the docs with AI](/docs/reference/ai): The docs MCP server, `llms.txt` and per-page Markdown.

  - [Roadmap](/docs/reference/roadmap): The version plan from P0 to v1.0 and the honest state of each item.

  - [Changelog](/docs/reference/changelog): Every notable change, newest first.

## Machine-readable docs [#machine-readable-docs]

The site serves its content to tools and language models as well as to people:

| URL                                | Content                                                                                                                      |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `/llms.txt`                        | An index of every page with its URL and description, headed by a one-line summary of Better Compute                          |
| `/llms-full.txt`                   | Every page as Markdown in one file                                                                                           |
| `/docs/<page>.md`                  | One page as Markdown. A request for `/docs/<page>` with `Accept: text/markdown` gets the same answer                         |
| `/llms.mdx/docs/<page>/content.md` | The route both of the above are rewritten to, with a `Link: rel="canonical"` header pointing at the HTML page                |
| `/api/search?query=...`            | The search index behind the search dialog, as JSON                                                                           |
| `/api/mcp`                         | An MCP server (Streamable HTTP) with search, page and reference lookup tools; see [use the docs with AI](/docs/reference/ai) |

`/llms.txt` and `/llms-full.txt` are sent with `X-Robots-Tag: noindex`, so search engines index the pages
themselves.

## Regenerating [#regenerating]

The generator imports the packages' `dist/` output, so build them first. It stops with the list of missing builds
when one is absent.

```bash
node node_modules/typescript/bin/tsc -b          # build every package the generator imports
node apps/docs/scripts/generate.mjs              # write apps/docs/generated/*.json and the generated pages
node apps/docs/scripts/generate.mjs --check      # fail when anything it would write differs (CI)
node apps/docs/scripts/check-content.mjs         # frontmatter, meta.json entries, links and anchors
```

`generate.mjs` writes `apps/docs/generated/{api,errors,protocol,cli,command-kinds,events,metrics,modules,packages}.json`
next to the pages.
The site reads them at build time: the MCP server's `lookup_api_method`, `lookup_error_code` and
`lookup_cli_command` tools answer from them. In `--check` mode, pages in `reference/api/` that the generator would no
longer write count as drift too; without `--check` it deletes them.

`tests/docs-generate.test.ts` checks the committed output without a build: the descriptions name only real groups,
methods and codes, every API group has its page and `meta.json` entry, the committed `generated/protocol.json` equals
what `protocol.mjs` reads from the `.proto` files now and the protocol page has a heading for every message
(2adc7a3), and `check-content.mjs` reports no problems.
With the packages built and `DOCS_GENERATE_CHECK=1` it also runs `generate.mjs --check`, so a registry change
without regenerated docs fails the suite.

> **Edit the source, not the page.** 
  Every generated page starts with a comment naming its source. Change the code, or the matching file in
  `apps/docs/descriptions/`, and regenerate; a hand edit to a generated page is overwritten on the next run and fails
  `--check` until then.



# Metrics (/docs/reference/metrics)

> The 57 Prometheus metric families better-compute exposes on /metrics, with their labels.



Every family carries the `better_compute_` prefix and none has a tenant label (series stay bounded by `maxSeries`). Histograms also expose `_bucket`, `_sum` and `_count` series. Families are grouped by the function that registers them: `standardMetrics` on every process, the others where a host wires them.

## standardMetrics [#standardmetrics]

| Metric                                      | Type      | Labels          | Description                                                                                                                                                                                 |
| ------------------------------------------- | --------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `better_compute_gateway_sessions`           | gauge     | `role`          | Open daemon sessions on this gateway                                                                                                                                                        |
| `better_compute_gateway_handshakes_total`   | counter   | `outcome`       | Session handshakes                                                                                                                                                                          |
| `better_compute_command_latency_seconds`    | histogram | `kind`, `lane`  | Command enqueue to terminal result. Buckets (s): 0.05, 0.1, 0.25, 0.5, 1, 1.5, 2.5, 5, 10, 30, 60                                                                                           |
| `better_compute_commands_swept_total`       | counter   | `to`            | Commands the CommandSweeper moved                                                                                                                                                           |
| `better_compute_work_queue_depth`           | gauge     | `kind`          | Due work\_queue rows                                                                                                                                                                        |
| `better_compute_reconcile_duration_seconds` | histogram | `reconciler`    | One reconciler run. Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30                                                                                                |
| `better_compute_generation_lag_nodes`       | gauge     |                 | Nodes whose observed generation is behind                                                                                                                                                   |
| `better_compute_placements_total`           | counter   | `outcome`       | Placement attempts                                                                                                                                                                          |
| `better_compute_booking_conflicts_total`    | counter   |                 | Booking conflicts that forced a replan                                                                                                                                                      |
| `better_compute_pool_gpus`                  | gauge     | `state`         | GPUs by state across pools                                                                                                                                                                  |
| `better_compute_node_connectivity`          | gauge     | `state`         | Nodes by connectivity                                                                                                                                                                       |
| `better_compute_backlog`                    | gauge     | `queue`         | Rows waiting in a queue table                                                                                                                                                               |
| `better_compute_usage_dead_letters_total`   | counter   |                 | Usage events dead-lettered                                                                                                                                                                  |
| `better_compute_iam_inbox_spill_total`      | counter   |                 | IAM events spilled because the inbox was down                                                                                                                                               |
| `better_compute_iam_calls_total`            | counter   | `op`, `outcome` | IamPort calls                                                                                                                                                                               |
| `better_compute_iam_call_duration_seconds`  | histogram | `op`            | IamPort call latency (each op takes ≥1 IAM lock acquisition). Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30                                                      |
| `better_compute_resolver_lookups_total`     | counter   | `result`        | Authz resolver lookups                                                                                                                                                                      |
| `better_compute_tenant_states`              | gauge     | `state`         | IAM tenants compute refuses (§12.8), by state: suspended, deleted                                                                                                                           |
| `better_compute_tenant_teardowns_pending`   | gauge     |                 | Deleted tenants whose teardown has not finished (a module still releasing)                                                                                                                  |
| `better_compute_sandbox_ready_seconds`      | histogram | `source`        | Sandbox create to its instance first running (§3.7), by source: a warm claim or a cold create; the counts give the warm-pool hit ratio. Buckets (s): 0.5, 1, 2, 5, 10, 20, 30, 60, 120, 300 |
| `better_compute_sandbox_expiry_lag_seconds` | histogram |                 | How late a sandbox was expired after its expiresAt. Buckets (s): 1, 5, 15, 30, 60, 120, 300, 900                                                                                            |
| `better_compute_sandbox_warm_instances`     | gauge     | `state`         | Unclaimed warm-pool instances, by instance state                                                                                                                                            |
| `better_compute_sandbox_warm_deficit`       | gauge     |                 | Warm instances missing below the templates’ warm.min, summed over live templates                                                                                                            |
| `better_compute_sandbox_warm_refills_total` | counter   |                 | Warm-pool instances created to refill templates                                                                                                                                             |
| `better_compute_cert_expiry_seconds`        | gauge     | `kind`          | Seconds until the soonest expiry                                                                                                                                                            |
| `better_compute_clock_offset_seconds`       | gauge     |                 | Process clock minus compute\_now()                                                                                                                                                          |
| `better_compute_security_events_total`      | counter   | `type`          | Security events recorded                                                                                                                                                                    |

## createObserver [#createobserver]

| Metric                                 | Type      | Labels                            | Description                                                                                                |
| -------------------------------------- | --------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `better_compute_spans_total`           | counter   | `kind`, `name`, `outcome`, `code` | Units of work by kind, name, outcome and code                                                              |
| `better_compute_span_duration_seconds` | histogram | `kind`                            | Latency of units of work by kind. Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 |

## meteredAttestor [#meteredattestor]

| Metric                                         | Type      | Labels                | Description                                                                                          |
| ---------------------------------------------- | --------- | --------------------- | ---------------------------------------------------------------------------------------------------- |
| `better_compute_attestor_verifications_total`  | counter   | `attestor`, `outcome` | Attestor verifications by outcome                                                                    |
| `better_compute_attestor_verification_seconds` | histogram | `attestor`            | Attestor verification time. Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 |

## meteredGatewayServices [#meteredgatewayservices]

| Metric                                       | Type      | Labels              | Description                                                                                           |
| -------------------------------------------- | --------- | ------------------- | ----------------------------------------------------------------------------------------------------- |
| `better_compute_gateway_service_calls_total` | counter   | `method`, `outcome` | Join, Renew and Secret calls by outcome                                                               |
| `better_compute_gateway_service_seconds`     | histogram | `method`            | Join, Renew and Secret time. Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 |

## meteredProvider [#meteredprovider]

| Metric                                 | Type      | Labels                          | Description                                                                      |
| -------------------------------------- | --------- | ------------------------------- | -------------------------------------------------------------------------------- |
| `better_compute_provider_calls_total`  | counter   | `provider`, `method`, `outcome` | Provider API calls by outcome                                                    |
| `better_compute_provider_call_seconds` | histogram | `provider`, `method`            | Provider API call time. Buckets (s): 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60 |

## placementObserver [#placementobserver]

| Metric                             | Type      | Labels   | Description                                                                                                |
| ---------------------------------- | --------- | -------- | ---------------------------------------------------------------------------------------------------------- |
| `better_compute_placement_seconds` | histogram | `placer` | One placement attempt, by placer. Buckets (s): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 |

## gatewayCollector [#gatewaycollector]

| Metric                                             | Type  | Labels            | Description                                                                    |
| -------------------------------------------------- | ----- | ----------------- | ------------------------------------------------------------------------------ |
| `better_compute_gateway_open_sessions`             | gauge | `gateway`, `role` | Open agent sessions by role                                                    |
| `better_compute_gateway_stream_headroom`           | gauge | `gateway`         | Streams this gateway can still admit                                           |
| `better_compute_gateway_draining`                  | gauge | `gateway`         | Whether the gateway is draining (1) or not (0)                                 |
| `better_compute_gateway_lease_held`                | gauge | `gateway`         | Whether the gateway holds its lease (1) or is fenced (0)                       |
| `better_compute_gateway_commands_in_flight`        | gauge | `gateway`         | Commands dispatched and not yet settled                                        |
| `better_compute_gateway_sessions_upgrade_required` | gauge | `gateway`         | Sessions told to upgrade                                                       |
| `better_compute_gateway_loop_consecutive_failures` | gauge | `gateway`, `loop` | Failed runs of a gateway loop since its last clean one (alert on a stuck loop) |
| `better_compute_gateway_loop_failures`             | gauge | `gateway`, `loop` | Failed runs of a gateway loop since start                                      |
| `better_compute_gateway_loop_last_ok_age_seconds`  | gauge | `gateway`, `loop` | Seconds since a gateway loop last ran cleanly (-1 before its first clean run)  |

## revocationFeedCollector [#revocationfeedcollector]

| Metric                                                | Type  | Labels | Description                                                                                    |
| ----------------------------------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------- |
| `better_compute_revocation_feed_consecutive_failures` | gauge | `feed` | Revocation feed polls failed in a row: revoked certificates keep their sessions while it fails |

## eventCounter [#eventcounter]

| Metric                        | Type    | Labels             | Description                     |
| ----------------------------- | ------- | ------------------ | ------------------------------- |
| `better_compute_events_total` | counter | `family`, `module` | Outbox events by catalog family |

## externalMetricsCollector [#externalmetricscollector]

| Metric                                                           | Type    | Labels               | Description                                                                                                   |
| ---------------------------------------------------------------- | ------- | -------------------- | ------------------------------------------------------------------------------------------------------------- |
| `better_compute_external_host_cpu_utilization_ratio`             | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (cpu\_utilization\_ratio)               |
| `better_compute_external_host_memory_used_bytes`                 | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (memory\_used\_bytes)                   |
| `better_compute_external_host_memory_total_bytes`                | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (memory\_total\_bytes)                  |
| `better_compute_external_host_load1`                             | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (load1)                                 |
| `better_compute_external_host_filesystem_used_bytes`             | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (filesystem\_used\_bytes)               |
| `better_compute_external_host_filesystem_size_bytes`             | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (filesystem\_size\_bytes)               |
| `better_compute_external_host_network_receive_bytes_per_second`  | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (network\_receive\_bytes\_per\_second)  |
| `better_compute_external_host_network_transmit_bytes_per_second` | gauge   | `source`, `instance` | A host metric of a machine that runs no bcd, pulled from its provider (network\_transmit\_bytes\_per\_second) |
| `better_compute_external_metrics_pull_failures_total`            | counter | `source`, `code`     | Pulls of an external metrics source that failed                                                               |
| `better_compute_external_metrics_last_pull_timestamp_seconds`    | gauge   | `source`             | When an external metrics source last answered (epoch seconds)                                                 |


# Modules (/docs/reference/modules)

> What each module registers in a full deployment: API groups, migrations, and the jobs the runtime runs.



The built-ins (the node controllers, `inventory`, `capacity`) are always registered; the rest are passed as `betterCompute({ modules })`.

| Module             | API groups                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Migrations                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node-controllers` |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `site-boot`        |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `inventory`        | [`inventory`](/docs/reference/api/inventory)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `capacity`         | [`capacity`](/docs/reference/api/capacity), [`pools`](/docs/reference/api/pools), [`quotas`](/docs/reference/api/quotas), [`reservations`](/docs/reference/api/reservations)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `capacity/0001_device_links`, `capacity/0002_lease_health`, `capacity/0003_device_counts`, `capacity/0004_mig_layouts`, `capacity/0005_extend_holds`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `tenancy`          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `tenancy/0001_tenant_states`, `tenancy/0002_teardown_attempts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `confirmations`    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `confirmations/0001_bindings`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `quorum`           | [`commandBatches`](/docs/reference/api/command-batches)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `quorum/0001_batches`, `quorum/0002_payload_digest`, `quorum/0003_trust_bundle_state`, `quorum/0004_trust_bundle_approver_set`, `quorum/0005_batch_request`, `quorum/0006_trust_bundle_history`, `quorum/0007_leaf_payload`, `quorum/0008_batch_agent`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `pki`              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `pki/0001_sealed_secrets`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `tunnels`          | [`nodes`](/docs/reference/api/nodes), [`sandboxes`](/docs/reference/api/sandboxes), [`tunnels`](/docs/reference/api/tunnels)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `provision`        | [`images`](/docs/reference/api/images), [`profiles`](/docs/reference/api/profiles), [`workflows`](/docs/reference/api/workflows)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `provision/0001_workflows`, `provision/0002_images`, `provision/0003_workflow_owner`, `provision/0004_gated_mig_layout`, `provision/0005_image_pins`, `provision/0006_workflow_batch`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `metal`            | [`metal`](/docs/reference/api/metal)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `metal/0001_nodes`, `metal/0002_addresses`, `metal/0003_tenant_root`, `metal/0004_bmc_credentials`, `metal/0005_batch`, `metal/0006_bmc_driver_options`, `metal/0007_discovered`, `metal/0008_mac_source`, `metal/0009_power_cut`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `vps`              | [`billing`](/docs/reference/api/billing), [`certificates`](/docs/reference/api/certificates), [`dnsRecords`](/docs/reference/api/dns-records), [`dnsZones`](/docs/reference/api/dns-zones), [`filesystems`](/docs/reference/api/filesystems), [`firewallRules`](/docs/reference/api/firewall-rules), [`flavors`](/docs/reference/api/flavors), [`images`](/docs/reference/api/images), [`instances`](/docs/reference/api/instances), [`loadBalancers`](/docs/reference/api/load-balancers), [`networks`](/docs/reference/api/networks), [`pools`](/docs/reference/api/pools), [`providerAccounts`](/docs/reference/api/provider-accounts), [`providerAddresses`](/docs/reference/api/provider-addresses), [`providerSnapshots`](/docs/reference/api/provider-snapshots), [`providerVolumes`](/docs/reference/api/provider-volumes), [`reservationGrants`](/docs/reference/api/reservation-grants), [`sandboxSnapshots`](/docs/reference/api/sandbox-snapshots), [`sandboxTemplates`](/docs/reference/api/sandbox-templates), [`sandboxVolumes`](/docs/reference/api/sandbox-volumes), [`sandboxes`](/docs/reference/api/sandboxes) | `vps/0001_instances`, `vps/0002_sealed_user_data`, `vps/0003_provider_bootstraps`, `vps/0004_sandboxes`, `vps/0005_sandbox_snapshots`, `vps/0006_warm_claims`, `vps/0007_template_env`, `vps/0008_template_network`, `vps/0009_snapshot_fixes`, `vps/0010_sandbox_metric_indexes`, `vps/0011_host_certificates`, `vps/0012_host_cert_validity`, `vps/0013_sandbox_expiring`, `vps/0014_sandbox_extending`, `vps/0015_snapshot_agent`, `vps/0016_warm_secret_versions`, `vps/0017_identity_rebind`, `vps/0018_placement`, `vps/0019_ip_pool_dns`, `vps/0020_volumes`, `vps/0021_volume_resize_hold`, `vps/0022_volume_storage`, `vps/0023_provider_machines`, `vps/0024_instance_attach`, `vps/0025_provider_reservations`, `vps/0026_networks`, `vps/0027_account_exclusive_key`, `vps/0030_instance_usage`, `vps/0031_reservation_grants`, `vps/0032_network_metering`, `vps/0033_provider_costs`, `vps/0042_pool_machines`, `vps/0043_instance_labels`, `vps/0044_network_adopt`, `vps/0046_instance_identity`, `vps/0048_quota_counts`, `vps/0084_network_zones`, `vps/0085_subnet_private_access`, `vps/0087_provider_purchases`, `vps/0088_provider_purchases_unknown`, `vps/0089_provider_purchases_attempts`, `vps/0090_provider_events`, `vps/0091_provider_interruptions`, `vps/0096_instance_aws_role`, `vps/0111_instance_placement`, `vps/0112_filesystems`, `vps/0113_purchase_proposals`, `vps/0114_provider_volumes`, `vps/0115_provider_snapshots`, `vps/0116_provider_addresses`, `vps/0117_provider_shared_volumes`, `vps/0118_provider_certificates`, `vps/0119_provider_load_balancers`, `vps/0120_provider_edge`, `vps/0121_quota_bookings`, `vps/0122_instance_external_metrics`, `vps/0123_provider_connect_secrets`, `vps/0124_instance_create_attempted` |
| `kubernetes`       | [`kubernetesClusters`](/docs/reference/api/kubernetes-clusters), [`kubernetesNodePools`](/docs/reference/api/kubernetes-node-pools), [`kubernetesSubstrates`](/docs/reference/api/kubernetes-substrates)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `kubernetes/0001_clusters`, `kubernetes/0002_substrates`, `kubernetes/0003_substrate_nodes`, `kubernetes/0004_substrate_ownership`, `kubernetes/0005_cluster_claims`, `kubernetes/0006_delete_backoff`, `kubernetes/0007_substrate_join_wait`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `cloud-resources`  | [`cloudResources`](/docs/reference/api/cloud-resources)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `cloud-resources/0001_resources`, `cloud-resources/0002_metering`, `cloud-resources/0003_identity`, `cloud-resources/0004_ensure_sent`, `cloud-resources/0005_network_ids`, `cloud-resources/0006_retry_backoff`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `events`           | [`webhooks`](/docs/reference/api/webhooks)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `events/0001_webhooks`, `events/0002_delivery_retention`, `events/0003_webhooks_by_tenant`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `fleet`            | [`fleet`](/docs/reference/api/fleet)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `fleet/0001_names`, `fleet/0002_applied`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `rollout`          | [`releases`](/docs/reference/api/releases), [`rollouts`](/docs/reference/api/rollouts)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `rollout/0001_rollouts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `audit`            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `audit/0001_checkpoints`, `audit/0002_archives`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

## Jobs [#jobs]

Each is one reconciler or runtime loop; `compute.jobs[name].tick()` runs one pass.

* `audit.checkpoint`
* `capacity.hosts`
* `capacity.mig`
* `capacity.reservations`
* `capacity.rollup`
* `capacity.usage`
* `cloud.resources`
* `cloud.resources.usage`
* `command-sweeper`
* `drain`
* `event-subscribers`
* `events.deliver`
* `events.fanout`
* `events.prune`
* `fleet.drift`
* `health`
* `inventory`
* `inventory.hygiene`
* `kubernetes.clusters`
* `kubernetes.substrates`
* `kubernetes.usage`
* `liveness`
* `metal`
* `node`
* `nodespec`
* `outbox-dispatch`
* `provision.site-cache`
* `quorum.batches`
* `rollout`
* `site-boot`
* `tenancy`
* `tenancy.resync`
* `tunnels.ssh-certificates-gc`
* `tunnels.ssh-certificates-reauthorize`
* `vps.ancillary-usage`
* `vps.bootstraps`
* `vps.costs`
* `vps.discovery`
* `vps.filesystems`
* `vps.grants`
* `vps.hosts`
* `vps.instances`
* `vps.networks`
* `vps.provider-events`
* `vps.provider-resources`
* `vps.provider-usage`
* `vps.purchase-proposals`
* `vps.purchases`
* `vps.reservations`
* `vps.sandboxes`
* `vps.suspension`
* `vps.usage`
* `vps.volumes`
* `workflow`


# Packages (/docs/reference/packages)

> Every package of the repository, its version, whether a release publishes it, what it is for and its entry points.



A release publishes 31 of these 35 packages to npm, all at one version; the release allowlist in `scripts/release-publish.mjs` decides which.

| Package                                 | Version | Release                                                                                  | Description                                                                                                                                                                                          | Entry points                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------- | ------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@better-compute/audit`                 | 0.3.0   | released                                                                                 | Tamper-evident audit for better-compute: signed hourly per-tenant Merkle checkpoints over audit\_events, inclusion proofs and verification.                                                          | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/aws`                   | 0.3.0   | not released: not ready: the AWS platform services are still in progress                 | AWS platform services behind better-compute seams: Secrets Manager, Parameter Store, KMS, S3, EventBridge, SNS, SQS, Kinesis, Firehose, ACM Private CA, RDS IAM auth and CloudWatch EMF. No AWS SDK. | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/aws-core`              | 0.3.0   | not released: not ready: only the unreleased AWS provider uses it                        | Zero-dependency AWS plumbing for better-compute: SigV4 signing, endpoints, query and JSON protocol clients, a hardened XML reader and credential sources.                                            | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `better-compute`                        | 0.3.0   | released                                                                                 | Manage GPU and bare-metal capacity and provision nodes in many ways: one package with every better-compute subpath.                                                                                  | `.`, `./core`, `./protocol`, `./inventory`, `./capacity`, `./provision`, `./metal`, `./vps`, `./iam`, `./iam/plugin`, `./iam/testing`, `./events`, `./fleet`, `./rollout`, `./kubernetes`, `./cloud-resources`, `./audit`, `./observability`, `./server`, `./gateway`, `./pki`, `./postgres`, `./pglite`, `./client`, `./cli`, `./mcp`, `./testing` |
| `@better-compute/capacity`              | 0.3.0   | released                                                                                 | Capacity for better-compute: atoms, bookings, quotas, rollups and the placement pipeline.                                                                                                            | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/cli`                   | 0.3.0   | released                                                                                 | The better-compute command line.                                                                                                                                                                     | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/client`                | 0.3.0   | released                                                                                 | Typed, browser-safe client for the better-compute API.                                                                                                                                               | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/cloud-resources`       | 0.3.0   | released                                                                                 | Managed data and platform services (databases, apps, functions) on provider accounts for better-compute: one governed, audited, metered resource API.                                                | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/console-kit`           | 0.3.0   | released                                                                                 | Framework-free view models for better-compute consoles: resource stores with pagination, live event refresh and permission-aware actions.                                                            | `.`, `./react`                                                                                                                                                                                                                                                                                                                                      |
| `@better-compute/core`                  | 0.3.0   | released                                                                                 | Browser-safe contracts for better-compute: domain types, errors, state tables, extension contracts and the IAM catalog.                                                                              | `.`, `./conformance`                                                                                                                                                                                                                                                                                                                                |
| `@better-compute/driver-sdk`            | 0.3.0   | released                                                                                 | bcx, the better-compute out-of-process driver protocol: serveDriver runs a defineDriver object as a plugin, externalDriver hosts one over auto-mTLS gRPC.                                            | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/events`                | 0.3.0   | released                                                                                 | Event delivery for better-compute: per-tenant signed webhooks with retries and dead letters, and the SSE /watch stream, read from the outbox.                                                        | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/fleet`                 | 0.3.0   | released                                                                                 | FleetConfig as code for better-compute: export, plan and apply sites, pools, offers, quotas, flavors and provider accounts.                                                                          | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/gateway`               | 0.3.0   | released                                                                                 | The better-compute gateway: daemon sessions over gRPC/HTTP/2 mTLS, liveness, command dispatch and Join/Renew/Secret forwarding.                                                                      | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/gcp-common`            | 0.3.0   | released                                                                                 | Shared Google Cloud plumbing for better-compute: token sources, workload identity, REST client, errors, paging, operations                                                                           | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/iam`                   | 0.3.0   | released                                                                                 | The better-iam bridge for better-compute: IAM plugin and catalog, event inbox, authz projection and resolver, the embedded IamPort, usage sink and flag cache.                                       | `.`, `./plugin`, `./testing`                                                                                                                                                                                                                                                                                                                        |
| `@better-compute/inventory`             | 0.3.0   | released                                                                                 | Hardware inventory for better-compute: section hashing, diffs, GPU model mapping, labels and health conditions.                                                                                      | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/kubernetes`            | 0.3.0   | released                                                                                 | Managed Kubernetes clusters (GKE and the like) on provider accounts for better-compute: the provider runs the control plane.                                                                         | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/mcp`                   | 0.3.0   | released                                                                                 | Model Context Protocol tools for better-compute: pools, capacity, GPU reservations and node status for AI agents, gated by better-iam.                                                               | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/metal`                 | 0.3.0   | released                                                                                 | Bare-metal lifecycle for better-compute: power, management, boot, deploy and inspect drivers.                                                                                                        | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/observability`         | 0.3.0   | released                                                                                 | Metrics for better-compute: a bounded Prometheus registry, the standard better*compute*\* series, a metered IamPort and backlog gauges.                                                              | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/pki`                   | 0.3.0   | released                                                                                 | Machine PKI for better-compute: local CAs, SPIFFE node certificates, sealed join tokens, signed join responses and renew proofs.                                                                     | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/postgres`              | 0.3.0   | released                                                                                 | PostgreSQL and PGlite storage for better-compute: executor, migrations and store conformance.                                                                                                        | `.`, `./pglite`, `./conformance`                                                                                                                                                                                                                                                                                                                    |
| `@better-compute/protocol`              | 0.3.0   | released                                                                                 | Wire contracts between the bcd daemon and the better-compute control plane (protobuf-es generated).                                                                                                  | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-aws`          | 0.3.0   | not released: not ready: the AWS program is still in progress (A1, emulator-tested only) | Amazon EC2 provider for better-compute: instances through provider pools, offers, attach, and the IAM and trust policies an AWS account needs.                                                       | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-digitalocean` | 0.3.0   | not released: not ready: the DigitalOcean program is still in progress                   | DigitalOcean provider for better-compute: Droplets as VPS instances and offers, VPC networks, power, reinstall and discovery.                                                                        | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-gcp`          | 0.3.0   | released                                                                                 | Google Compute Engine provider for better-compute: VMs as VPS instances and offers, attach, reservations and tenant networks.                                                                        | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-hetzner`      | 0.3.0   | released                                                                                 | Hetzner providers for better-compute: Hetzner Cloud servers as VPS instances and offers.                                                                                                             | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-kubernetes`   | 0.3.0   | released                                                                                 | Kubernetes as a capacity source for better-compute: bcd as a DaemonSet adopts a cluster's nodes and GPUs into a pool.                                                                                | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provider-proxmox`      | 0.3.0   | released                                                                                 | Proxmox VE provider for better-compute: QEMU VMs cloned from templates as VPS instances.                                                                                                             | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/provision`             | 0.3.0   | released                                                                                 | Provisioning for better-compute: profiles, steps and the workflow engine.                                                                                                                            | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/rollout`               | 0.3.0   | released                                                                                 | Signed bcd releases and wave rollouts (A/B self-update) for better-compute, within each pool's disruption budget.                                                                                    | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/server`                | 0.3.0   | released                                                                                 | The better-compute control plane: betterCompute(), the operation envelope, router and runtimes.                                                                                                      | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/testing`               | 0.3.0   | released                                                                                 | Testing kit for better-compute: fixtures, simulated daemons, sim clock and conformance runners.                                                                                                      | `.`                                                                                                                                                                                                                                                                                                                                                 |
| `@better-compute/vps`                   | 0.3.0   | released                                                                                 | VPS instances for better-compute: runtimes, flavors and external providers.                                                                                                                          | `.`                                                                                                                                                                                                                                                                                                                                                 |


# Protocol (/docs/reference/protocol)

> Every service, message and enum of the daemon protocol, from the .proto sources, with field numbers and types.



The wire contract between bcd, the gateway and drivers. It is frozen: changes are additive only (`buf breaking`), and messages and fields marked with a phase are part of the frozen shape before they are used. For the session flow, framing, renewal and the WebSocket fallback, see [Protocol](/docs/daemon/protocol).

## commands.proto [#commandsproto]

Package `bettercompute.agent.v1`, `proto/bettercompute/agent/v1/commands.proto`.

Payloads and results of the built-in v0.1 command kinds (PLAN.md §7.3). A Command carries the request in CommandBody.payload (google.protobuf.Any); the CommandUpdate carries the serialized result in `result`. The kind table (lane and flags) lives in @better-compute/protocol (`commandKinds`) and in the daemon's dispatcher; both must agree.

### DiagNvmlRequest [#diagnvmlrequest]

| Field                | Number | Type              | Description                                                    |
| -------------------- | ------ | ----------------- | -------------------------------------------------------------- |
| `gpu_uuids`          | 1      | repeated `string` | GPUs to check; empty checks every GPU.                         |
| `expected_gpu_count` | 2      | `uint32`          | When non-zero, fewer visible GPUs fails the check "gpu.count". |

### DiagNvmlResult [#diagnvmlresult]

| Field            | Number | Type                                           | Description                                       |
| ---------------- | ------ | ---------------------------------------------- | ------------------------------------------------- |
| `passed`         | 1      | `bool`                                         | Every check on every GPU passed.                  |
| `gpu_count`      | 2      | `uint32`                                       | GPUs visible to NVML.                             |
| `driver_version` | 3      | `string`                                       |                                                   |
| `checks`         | 4      | repeated [`DiagnosticCheck`](#diagnosticcheck) | Node-level checks, e.g. "nvml.init", "gpu.count". |
| `gpus`           | 5      | repeated [`GpuDiagnostic`](#gpudiagnostic)     | Sorted by uuid.                                   |

### GpuDiagnostic [#gpudiagnostic]

| Field    | Number | Type                                           | Description                                                                                    |
| -------- | ------ | ---------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `uuid`   | 1      | `string`                                       |                                                                                                |
| `passed` | 2      | `bool`                                         |                                                                                                |
| `checks` | 3      | repeated [`DiagnosticCheck`](#diagnosticcheck) | e.g. "ecc.uncorrectable", "row-remap.pending", "row-remap.failure", "pcie.link", "xid.recent". |

### DiagnosticCheck [#diagnosticcheck]

| Field     | Number | Type     | Description |
| --------- | ------ | -------- | ----------- |
| `name`    | 1      | `string` |             |
| `passed`  | 2      | `bool`   |             |
| `message` | 3      | `string` |             |

### InventoryRefreshRequest [#inventoryrefreshrequest]

| Field      | Number | Type              | Description                                                                  |
| ---------- | ------ | ----------------- | ---------------------------------------------------------------------------- |
| `sections` | 1      | repeated `string` | Section names to send; empty sends every section.                            |
| `rescan`   | 2      | `bool`            | Re-collect from the hardware first instead of resending the last collection. |

### InventoryRefreshResult [#inventoryrefreshresult]

| Field            | Number | Type                       | Description                                            |
| ---------------- | ------ | -------------------------- | ------------------------------------------------------ |
| `section_hashes` | 1      | map of `string` to `bytes` | Section name → SHA-256 of the section content as sent. |

### LogsCollectRequest [#logscollectrequest]

| Field              | Number | Type                                 | Description                                                                   |
| ------------------ | ------ | ------------------------------------ | ----------------------------------------------------------------------------- |
| `upload_url`       | 1      | `string`                             | Presigned HTTPS PUT URL.                                                      |
| `upload_headers`   | 2      | repeated [`HttpHeader`](#httpheader) | Headers the presigned request requires.                                       |
| `units`            | 3      | repeated `string`                    | systemd units; empty means bcd's own units.                                   |
| `lookback_seconds` | 4      | `uint32`                             | How far back to collect.                                                      |
| `max_bytes`        | 5      | `uint64`                             | Upload cap; the bundle is truncated (oldest first) beyond it.                 |
| `support_bundle`   | 6      | `bool`                               | Include the support bundle (bcd status, doctor, journal metadata, inventory). |

### HttpHeader [#httpheader]

| Field   | Number | Type     | Description |
| ------- | ------ | -------- | ----------- |
| `name`  | 1      | `string` |             |
| `value` | 2      | `string` |             |

### LogsCollectResult [#logscollectresult]

| Field            | Number | Type     | Description                     |
| ---------------- | ------ | -------- | ------------------------------- |
| `uploaded_bytes` | 1      | `uint64` |                                 |
| `sha256`         | 2      | `bytes`  | SHA-256 of the uploaded object. |
| `truncated`      | 3      | `bool`   |                                 |
| `content_type`   | 4      | `string` | e.g. "application/zstd".        |

### NodeRebootRequest [#noderebootrequest]

| Field      | Number | Type     | Description                                                               |
| ---------- | ------ | -------- | ------------------------------------------------------------------------- |
| `reason`   | 1      | `string` |                                                                           |
| `delay_ms` | 2      | `uint32` | Wait before rebooting (lets the CommandUpdate RUNNING reach the gateway). |

### NodeRebootResult [#noderebootresult]

| Field              | Number | Type     | Description                                                           |
| ------------------ | ------ | -------- | --------------------------------------------------------------------- |
| `previous_boot_id` | 1      | `string` | The boot that ran the command; CommandUpdate.boot\_id is the new one. |

### GpuResetRequest [#gpuresetrequest]

| Field       | Number | Type              | Description                                              |
| ----------- | ------ | ----------------- | -------------------------------------------------------- |
| `gpu_uuids` | 1      | repeated `string` | At least one GPU; an empty list fails "invalid-payload". |

### GpuResetResult [#gpuresetresult]

| Field  | Number | Type                                           | Description     |
| ------ | ------ | ---------------------------------------------- | --------------- |
| `gpus` | 1      | repeated [`GpuResetOutcome`](#gpuresetoutcome) | Sorted by uuid. |

### GpuResetOutcome [#gpuresetoutcome]

| Field     | Number | Type     | Description                                   |
| --------- | ------ | -------- | --------------------------------------------- |
| `uuid`    | 1      | `string` |                                               |
| `reset`   | 2      | `bool`   |                                               |
| `healthy` | 3      | `bool`   | ECC and row-remap state read after the reset. |
| `message` | 4      | `string` |                                               |

### DiagDcgmRequest [#diagdcgmrequest]

diag.dcgm: lane provision, retry-safe, attested. Commissioning (level 2) and acceptance (level 1).

| Field       | Number | Type              | Description                                                            |
| ----------- | ------ | ----------------- | ---------------------------------------------------------------------- |
| `level`     | 1      | `uint32`          | dcgmi diag -r level: 1 (seconds), 2 (\~2 min), 3 (\~15 min), 4 (long). |
| `gpu_uuids` | 2      | repeated `string` | GPUs to test; empty tests every GPU.                                   |

### DiagDcgmResult [#diagdcgmresult]

| Field    | Number | Type                                           | Description                                                 |
| -------- | ------ | ---------------------------------------------- | ----------------------------------------------------------- |
| `passed` | 1      | `bool`                                         |                                                             |
| `level`  | 2      | `uint32`                                       |                                                             |
| `tests`  | 3      | repeated [`DiagnosticCheck`](#diagnosticcheck) | Node-level tests, e.g. "dcgm.deployment", "pcie.bandwidth". |
| `gpus`   | 4      | repeated [`GpuDiagnostic`](#gpudiagnostic)     | Per-GPU tests, sorted by uuid.                              |

### WipeRequest [#wiperequest]

wipe: lane provision, NOT retry-safe, attested, destructive (signed from v0.2). The §11.1
invariant "ready only after a verified wipe" rests on this result's node signature.

| Field   | Number | Type                    | Description                                                            |
| ------- | ------ | ----------------------- | ---------------------------------------------------------------------- |
| `mode`  | 1      | [`WipeMode`](#wipemode) |                                                                        |
| `disks` | 2      | repeated `string`       | Disks to wipe, by WWN or serial; empty wipes every non-removable disk. |

### WipeResult [#wiperesult]

| Field   | Number | Type                             | Description   |
| ------- | ------ | -------------------------------- | ------------- |
| `mode`  | 1      | [`WipeMode`](#wipemode)          |               |
| `disks` | 2      | repeated [`DiskWipe`](#diskwipe) | Sorted by id. |

### DiskWipe [#diskwipe]

| Field      | Number | Type     | Description                                                                                   |
| ---------- | ------ | -------- | --------------------------------------------------------------------------------------------- |
| `id`       | 1      | `string` | WWN, or the serial when the disk has no WWN.                                                  |
| `method`   | 2      | `string` | How the disk was wiped, e.g. "nvme-format", "ata-secure-erase", "blkdiscard", "crypto-erase". |
| `verified` | 3      | `bool`   | A read-back check confirmed the result.                                                       |
| `message`  | 4      | `string` |                                                                                               |

### DeployImageRequest [#deployimagerequest]

deploy.image: lane provision, NOT retry-safe, attested, destructive. Streams the image from
the site cache (HTTP range reads) through decompression straight to the root disk, checking the
SHA-256 inline (§3.5 step 4).

| Field            | Number | Type                                    | Description                                                                                                                                            |
| ---------------- | ------ | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url`            | 1      | `string`                                | HTTPS URL, normally the site agent's image cache.                                                                                                      |
| `sha256`         | 2      | `bytes`                                 | SHA-256 of the image as stored (before decompression).                                                                                                 |
| `format`         | 3      | [`ImageFormat`](#imageformat)           |                                                                                                                                                        |
| `compression`    | 4      | [`ImageCompression`](#imagecompression) |                                                                                                                                                        |
| `root_disk`      | 5      | [`RootDiskHint`](#rootdiskhint)         |                                                                                                                                                        |
| `user_data_ref`  | 6      | `string`                                | Secret ref of the rendered user data (it embeds a single-use `provisioned` join token); the agent fetches it with SecretRequest, never in the command. |
| `user_data_kind` | 7      | `string`                                | "cloud-init", "ignition" or "talos".                                                                                                                   |
| `uefi`           | 8      | `bool`                                  | Write for UEFI boot (EFI system partition) rather than legacy BIOS.                                                                                    |

### RootDiskHint [#rootdiskhint]

The disk to install to; the first non-empty field wins, and a hint matching no disk fails the step.

| Field            | Number | Type     | Description                                                    |
| ---------------- | ------ | -------- | -------------------------------------------------------------- |
| `wwn`            | 1      | `string` |                                                                |
| `serial`         | 2      | `string` |                                                                |
| `device`         | 3      | `string` | e.g. "/dev/nvme0n1"; last resort, device names are not stable. |
| `min_size_bytes` | 4      | `uint64` | Smallest disk at least this big.                               |

### DeployImageResult [#deployimageresult]

| Field           | Number | Type     | Description                                              |
| --------------- | ------ | -------- | -------------------------------------------------------- |
| `written_bytes` | 1      | `uint64` |                                                          |
| `sha256`        | 2      | `bytes`  | SHA-256 verified while streaming (equals the request's). |
| `root_disk`     | 3      | `string` | WWN or serial of the disk written.                       |
| `duration_ms`   | 4      | `uint32` |                                                          |

### KexecRequest [#kexecrequest]

boot.kexec: lane control, retry-safe, self-terminating. Boots the installed OS from the ramdisk
without firmware POST; completes when the agent rejoins with a new boot\_id and role `node`.

| Field            | Number | Type              | Description                                                                                      |
| ---------------- | ------ | ----------------- | ------------------------------------------------------------------------------------------------ |
| `kernel_path`    | 1      | `string`          | Kernel and initrd paths on the installed root filesystem; empty means the distribution defaults. |
| `initrd_path`    | 2      | `string`          |                                                                                                  |
| `kernel_cmdline` | 3      | repeated `string` | Appended to the installed kernel command line.                                                   |

### KexecResult [#kexecresult]

| Field              | Number | Type     | Description                                                                     |
| ------------------ | ------ | -------- | ------------------------------------------------------------------------------- |
| `previous_boot_id` | 1      | `string` | The ramdisk boot that ran the command; the rejoining agent reports the new one. |

### DriverInvokeRequest [#driverinvokerequest]

| Field            | Number | Type                     | Description                                                                                                              |
| ---------------- | ------ | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `workflow_id`    | 1      | `string`                 |                                                                                                                          |
| `step_id`        | 2      | `string`                 |                                                                                                                          |
| `attempt`        | 3      | `uint32`                 |                                                                                                                          |
| `target_node_id` | 4      | `string`                 | Set when a site agent runs it for another node.                                                                          |
| `args`           | 5      | `google.protobuf.Struct` | The expanded step: \{concern, operation, args, driver?, config?}; config holds secret:// refs only, never secret values. |

### DriverInvokeResult [#driverinvokeresult]

| Field         | Number | Type     | Description                                                    |
| ------------- | ------ | -------- | -------------------------------------------------------------- |
| `result_json` | 1      | `string` | The operation's JSON result (bcx InvokeResponse.result\_json). |

### InstanceSnapshotRequest [#instancesnapshotrequest]

| Field            | Number | Type                                 | Description                                                                 |
| ---------------- | ------ | ------------------------------------ | --------------------------------------------------------------------------- |
| `instance_id`    | 1      | `string`                             |                                                                             |
| `upload_url`     | 2      | `string`                             | Presigned HTTPS PUT URL.                                                    |
| `upload_headers` | 3      | repeated [`HttpHeader`](#httpheader) |                                                                             |
| `image_id`       | 4      | `string`                             | The snapshot image id the control plane registers the upload under.         |
| `include_memory` | 5      | `bool`                               | Include the memory state (a restorable running snapshot); false: disk only. |

### InstanceSnapshotResult [#instancesnapshotresult]

| Field            | Number | Type     | Description                                                                                            |
| ---------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------ |
| `uploaded_bytes` | 1      | `uint64` |                                                                                                        |
| `sha256`         | 2      | `bytes`  | SHA-256 of the uploaded object; restoring verifies it (InstanceSpec.restore\_from.sha256).             |
| `runtime`        | 3      | `string` | The runtime that wrote it; a restore needs the same runtime.                                           |
| `cpu_model`      | 4      | `string` | The host CPU model the memory state depends on (the inventory label `cpu.model`), empty for disk only. |

### WipeMode [#wipemode]

| Value                    | Number | Description                                                        |
| ------------------------ | ------ | ------------------------------------------------------------------ |
| `WIPE_MODE_UNSPECIFIED`  | 0      |                                                                    |
| `WIPE_MODE_QUICK`        | 1      | Partition tables, signatures and the first/last MiB of every disk. |
| `WIPE_MODE_SECURE_ERASE` | 2      | ATA secure erase / NVMe format with user-data erase.               |
| `WIPE_MODE_CRYPTO_ERASE` | 3      | Rotate the self-encrypting drive's media key.                      |

### ImageFormat [#imageformat]

| Value                      | Number | Description |
| -------------------------- | ------ | ----------- |
| `IMAGE_FORMAT_UNSPECIFIED` | 0      |             |
| `IMAGE_FORMAT_RAW`         | 1      |             |
| `IMAGE_FORMAT_QCOW2`       | 2      |             |

### ImageCompression [#imagecompression]

| Value                           | Number | Description |
| ------------------------------- | ------ | ----------- |
| `IMAGE_COMPRESSION_UNSPECIFIED` | 0      |             |
| `IMAGE_COMPRESSION_NONE`        | 1      |             |
| `IMAGE_COMPRESSION_ZSTD`        | 2      |             |
| `IMAGE_COMPRESSION_GZIP`        | 3      |             |

## inventory.proto [#inventoryproto]

Package `bettercompute.agent.v1`, `proto/bettercompute/agent/v1/inventory.proto`.

NodeInventory v1 (PLAN.md §10.1; docs/research/prior-art.md §4).

### NodeInventory [#nodeinventory]

An assembled snapshot: what the control plane stores (append-only) and what an inspect driver
returns. Never hashed as a whole.

| Field            | Number | Type                              | Description                                                  |
| ---------------- | ------ | --------------------------------- | ------------------------------------------------------------ |
| `schema_version` | 1      | `uint32`                          | 1 for this schema.                                           |
| `collected_at`   | 2      | `google.protobuf.Timestamp`       |                                                              |
| `section_hashes` | 3      | map of `string` to `bytes`        | Section name → SHA-256 of that section's canonical encoding. |
| `system`         | 10     | [`SystemSection`](#systemsection) |                                                              |
| `cpu`            | 11     | [`CpuSection`](#cpusection)       |                                                              |
| `memory`         | 12     | [`MemorySection`](#memorysection) |                                                              |
| `disks`          | 13     | [`DisksSection`](#diskssection)   |                                                              |
| `nics`           | 14     | [`NicsSection`](#nicssection)     |                                                              |
| `gpus`           | 15     | [`GpusSection`](#gpussection)     |                                                              |
| `bmc`            | 16     | [`BmcSection`](#bmcsection)       |                                                              |
| `os`             | 17     | [`OsSection`](#ossection)         |                                                              |
| `images`         | 18     | [`ImagesSection`](#imagessection) | (v0.3) Section "images".                                     |

### CloudSection [#cloudsection]

| Field           | Number | Type     | Description                                                                            |
| --------------- | ------ | -------- | -------------------------------------------------------------------------------------- |
| `provider`      | 1      | `string` | "aws".                                                                                 |
| `instance_id`   | 2      | `string` | e.g. "i-0123456789abcdef0".                                                            |
| `instance_type` | 3      | `string` | e.g. "p5.48xlarge", "c7i.metal-48xl".                                                  |
| `region`        | 4      | `string` | e.g. "us-east-1".                                                                      |
| `zone`          | 5      | `string` | The zone name, e.g. "us-east-1a" (names map to physical zones per account).            |
| `zone_id`       | 6      | `string` | The zone id, e.g. "use1-az4" (the same physical zone in every account).                |
| `lifecycle`     | 7      | `string` | "on-demand", "spot" or "capacity-block"; empty when the metadata names something else. |
| `efa_devices`   | 8      | `uint32` | Elastic Fabric Adapter devices (/sys/class/infiniband/\* bound to the efa driver).     |

### ImagesSection [#imagessection]

| Field            | Number | Type                                   | Description                                                                 |
| ---------------- | ------ | -------------------------------------- | --------------------------------------------------------------------------- |
| `images`         | 1      | repeated [`CachedImage`](#cachedimage) | Sorted by sha256; at most 1,000 entries (the most recently used).           |
| `capacity_bytes` | 2      | `uint64`                               | Cache capacity and use in bytes, when the runtime knows them (0 = unknown). |
| `used_bytes`     | 3      | `uint64`                               |                                                                             |

### CachedImage [#cachedimage]

| Field        | Number | Type                                    | Description                                                                         |
| ------------ | ------ | --------------------------------------- | ----------------------------------------------------------------------------------- |
| `image_id`   | 1      | `string`                                | The catalog image id (img\_…) the node pulled it for, when it knows it; else empty. |
| `ref`        | 2      | `string`                                | The reference it was pulled by (https URL or OCI ref).                              |
| `sha256`     | 3      | `bytes`                                 |                                                                                     |
| `size_bytes` | 4      | `uint64`                                |                                                                                     |
| `state`      | 5      | [`CachedImageState`](#cachedimagestate) |                                                                                     |
| `last_used`  | 6      | `google.protobuf.Timestamp`             | Last use, truncated to the hour so the section hash does not churn on every use.    |

### SystemSection [#systemsection]

| Field            | Number | Type                              | Description                                                                                                                                                                                                                                                |
| ---------------- | ------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vendor`         | 1      | `string`                          |                                                                                                                                                                                                                                                            |
| `product`        | 2      | `string`                          |                                                                                                                                                                                                                                                            |
| `serial`         | 3      | `string`                          |                                                                                                                                                                                                                                                            |
| `product_uuid`   | 4      | `string`                          | Lowercase.                                                                                                                                                                                                                                                 |
| `board_vendor`   | 5      | `string`                          |                                                                                                                                                                                                                                                            |
| `board_product`  | 6      | `string`                          |                                                                                                                                                                                                                                                            |
| `board_serial`   | 7      | `string`                          |                                                                                                                                                                                                                                                            |
| `chassis_serial` | 8      | `string`                          |                                                                                                                                                                                                                                                            |
| `bios_vendor`    | 9      | `string`                          |                                                                                                                                                                                                                                                            |
| `bios_version`   | 10     | `string`                          |                                                                                                                                                                                                                                                            |
| `boot_mode`      | 11     | [`BootMode`](#bootmode)           |                                                                                                                                                                                                                                                            |
| `secure_boot`    | 12     | optional `bool`                   | Absent when unknown.                                                                                                                                                                                                                                       |
| `tpm`            | 13     | [`Tpm`](#tpm)                     | Absent when no TPM is present.                                                                                                                                                                                                                             |
| `cloud`          | 14     | [`CloudInstance`](#cloudinstance) | (v0.3) The cloud instance this system is, from the provider's metadata service: absent off a cloud, and when the service did not answer. Inventory only: it is not signed, so nothing ever uses it as an identity (hardware\_key and node join ignore it). |

### CloudInstance [#cloudinstance]

What a cloud's instance metadata service reports about the instance (DigitalOcean:
169.254.169.254/metadata/v1.json, read only on a Droplet). Never user data, vendor data, keys
or tokens.

| Field         | Number | Type                                         | Description                                              |
| ------------- | ------ | -------------------------------------------- | -------------------------------------------------------- |
| `provider`    | 1      | `string`                                     | "digitalocean".                                          |
| `instance_id` | 2      | `string`                                     | The provider's instance id (the Droplet id, in decimal). |
| `region`      | 3      | `string`                                     | Region slug, e.g. "nyc3".                                |
| `tags`        | 4      | repeated `string`                            | Sorted, without duplicates.                              |
| `interfaces`  | 5      | repeated [`CloudInterface`](#cloudinterface) | Sorted by network, then MAC.                             |

### CloudInterface [#cloudinterface]

| Field                | Number | Type     | Description                      |
| -------------------- | ------ | -------- | -------------------------------- |
| `network`            | 1      | `string` | "public" or "private" (the VPC). |
| `mac`                | 2      | `string` | Lowercase.                       |
| `ipv4_address`       | 3      | `string` |                                  |
| `ipv4_netmask`       | 4      | `string` |                                  |
| `ipv4_gateway`       | 5      | `string` |                                  |
| `ipv6_address`       | 6      | `string` |                                  |
| `ipv6_prefix_length` | 7      | `uint32` | 0 = no IPv6.                     |
| `ipv6_gateway`       | 8      | `string` |                                  |

### Tpm [#tpm]

| Field            | Number | Type     | Description |
| ---------------- | ------ | -------- | ----------- |
| `version`        | 1      | `string` | e.g. "2.0". |
| `manufacturer`   | 2      | `string` |             |
| `ek_cert_issuer` | 3      | `string` |             |

### CpuSection [#cpusection]

| Field           | Number | Type                             | Description                                                                                                                                                                                                      |
| --------------- | ------ | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vendor`        | 1      | `string`                         |                                                                                                                                                                                                                  |
| `model`         | 2      | `string`                         |                                                                                                                                                                                                                  |
| `architecture`  | 3      | `string`                         | "x86\_64", "aarch64".                                                                                                                                                                                            |
| `sockets`       | 4      | `uint32`                         |                                                                                                                                                                                                                  |
| `cores`         | 5      | `uint32`                         |                                                                                                                                                                                                                  |
| `threads`       | 6      | `uint32`                         |                                                                                                                                                                                                                  |
| `flags`         | 7      | repeated `string`                | Sorted, e.g. "amx\_tile", "avx512f".                                                                                                                                                                             |
| `numa_nodes`    | 8      | repeated [`NumaNode`](#numanode) | Sorted by id.                                                                                                                                                                                                    |
| `isolated_cpus` | 9      | `string`                         | (v1.1 §3.8) CPUs the kernel isolates from the scheduler (/sys/devices/system/cpu/isolated, cpulist form; empty when none): bcd pins VMs and holds there first, and never counts them for the host's own reserve. |

### NumaNode [#numanode]

| Field          | Number | Type                                     | Description                                                                                                                                          |
| -------------- | ------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`           | 1      | `uint32`                                 |                                                                                                                                                      |
| `cpus`         | 2      | `string`                                 | Kernel cpulist form, e.g. "0-31,64-95".                                                                                                              |
| `memory_bytes` | 3      | `uint64`                                 |                                                                                                                                                      |
| `hugepages`    | 4      | repeated [`HugepagePool`](#hugepagepool) | (v1.1 §3.8) The node's hugepage pools (/sys/devices/system/node/nodeN/hugepages), sorted by page size: NUMA-local hugepage backing books from these. |

### MemorySection [#memorysection]

| Field         | Number | Type                                     | Description                                          |
| ------------- | ------ | ---------------------------------------- | ---------------------------------------------------- |
| `total_bytes` | 1      | `uint64`                                 |                                                      |
| `hugepages`   | 2      | repeated [`HugepagePool`](#hugepagepool) | Sorted by page size.                                 |
| `dimms`       | 3      | repeated [`Dimm`](#dimm)                 | Sorted by slot; empty when dmidecode is unavailable. |
| `edac`        | 4      | [`EdacCounters`](#edaccounters)          | Absent when EDAC is unavailable.                     |

### HugepagePool [#hugepagepool]

| Field             | Number | Type     | Description                        |
| ----------------- | ------ | -------- | ---------------------------------- |
| `page_size_bytes` | 1      | `uint64` | 2097152 (2Mi) or 1073741824 (1Gi). |
| `total`           | 2      | `uint64` |                                    |

### Dimm [#dimm]

| Field          | Number | Type     | Description  |
| -------------- | ------ | -------- | ------------ |
| `slot`         | 1      | `string` |              |
| `size_bytes`   | 2      | `uint64` |              |
| `speed_mts`    | 3      | `uint32` |              |
| `type`         | 4      | `string` | e.g. "DDR5". |
| `ecc`          | 5      | `bool`   |              |
| `manufacturer` | 6      | `string` |              |
| `part_number`  | 7      | `string` |              |
| `serial`       | 8      | `string` |              |

### EdacCounters [#edaccounters]

| Field           | Number | Type     | Description |
| --------------- | ------ | -------- | ----------- |
| `correctable`   | 1      | `uint64` |             |
| `uncorrectable` | 2      | `uint64` |             |

### DisksSection [#diskssection]

| Field   | Number | Type                     | Description     |
| ------- | ------ | ------------------------ | --------------- |
| `disks` | 1      | repeated [`Disk`](#disk) | Sorted by name. |

### Disk [#disk]

| Field        | Number | Type                              | Description                       |
| ------------ | ------ | --------------------------------- | --------------------------------- |
| `name`       | 1      | `string`                          | Kernel name, e.g. "nvme0n1".      |
| `model`      | 2      | `string`                          |                                   |
| `serial`     | 3      | `string`                          |                                   |
| `wwn`        | 4      | `string`                          | Stable device id when available.  |
| `size_bytes` | 5      | `uint64`                          |                                   |
| `rotational` | 6      | `bool`                            |                                   |
| `transport`  | 7      | [`DiskTransport`](#disktransport) |                                   |
| `smart`      | 8      | [`SmartStatus`](#smartstatus)     | Absent when SMART is unavailable. |
| `removable`  | 9      | `bool`                            |                                   |

### SmartStatus [#smartstatus]

| Field          | Number | Type              | Description |
| -------------- | ------ | ----------------- | ----------- |
| `healthy`      | 1      | `bool`            |             |
| `wear_percent` | 2      | optional `uint32` |             |
| `media_errors` | 3      | optional `uint64` |             |

### NicsSection [#nicssection]

| Field  | Number | Type                   | Description                        |
| ------ | ------ | ---------------------- | ---------------------------------- |
| `nics` | 1      | repeated [`Nic`](#nic) | Physical NICs only, sorted by MAC. |

### Nic [#nic]

| Field             | Number | Type                      | Description                                      |
| ----------------- | ------ | ------------------------- | ------------------------------------------------ |
| `name`            | 1      | `string`                  |                                                  |
| `mac`             | 2      | `string`                  | Permanent MAC, lowercase colon form.             |
| `driver`          | 3      | `string`                  |                                                  |
| `firmware`        | 4      | `string`                  |                                                  |
| `speed_mbps`      | 5      | `uint32`                  |                                                  |
| `pci_address`     | 6      | `string`                  | e.g. "0000:17:00.0".                             |
| `numa_node`       | 7      | optional `uint32`         | Absent when the device reports no NUMA affinity. |
| `sriov_total_vfs` | 8      | `uint32`                  |                                                  |
| `rdma`            | 9      | [`RdmaType`](#rdmatype)   |                                                  |
| `link`            | 10     | [`LinkState`](#linkstate) |                                                  |
| `mtu`             | 11     | `uint32`                  |                                                  |

### GpusSection [#gpussection]

| Field                 | Number | Type                   | Description                                          |
| --------------------- | ------ | ---------------------- | ---------------------------------------------------- |
| `driver_version`      | 1      | `string`               | Kernel driver version, e.g. "580.65.06" (node-wide). |
| `cuda_driver_version` | 2      | `string`               | e.g. "13.0".                                         |
| `nvml_version`        | 3      | `string`               |                                                      |
| `gpus`                | 4      | repeated [`Gpu`](#gpu) | Sorted by index.                                     |

### Gpu [#gpu]

| Field                           | Number | Type                                   | Description                                                                      |
| ------------------------------- | ------ | -------------------------------------- | -------------------------------------------------------------------------------- |
| `index`                         | 1      | `uint32`                               |                                                                                  |
| `uuid`                          | 2      | `string`                               | "GPU-…".                                                                         |
| `pci_bus_id`                    | 3      | `string`                               | e.g. "00000000:18:00.0".                                                         |
| `name`                          | 4      | `string`                               | Raw NVML name, e.g. "NVIDIA H100 80GB HBM3".                                     |
| `architecture`                  | 5      | `string`                               | e.g. "hopper", "blackwell".                                                      |
| `vram_bytes`                    | 6      | `uint64`                               |                                                                                  |
| `serial`                        | 7      | `string`                               |                                                                                  |
| `vbios_version`                 | 8      | `string`                               |                                                                                  |
| `board_part_number`             | 9      | `string`                               |                                                                                  |
| `numa_node`                     | 10     | optional `uint32`                      |                                                                                  |
| `power_limit_mw`                | 11     | `uint32`                               |                                                                                  |
| `persistence_mode`              | 12     | `bool`                                 |                                                                                  |
| `compute_mode`                  | 13     | `string`                               | e.g. "default", "exclusive-process".                                             |
| `cuda_compute_capability_major` | 14     | `uint32`                               |                                                                                  |
| `cuda_compute_capability_minor` | 15     | `uint32`                               |                                                                                  |
| `mig`                           | 16     | [`GpuMig`](#gpumig)                    |                                                                                  |
| `nvlinks`                       | 17     | repeated [`NvlinkLink`](#nvlinklink)   | Sorted by link.                                                                  |
| `topology`                      | 18     | repeated [`GpuPeerLink`](#gpupeerlink) | Topology matrix row for this GPU: one entry per other GPU, sorted by peer\_uuid. |
| `fabric`                        | 19     | [`GpuFabric`](#gpufabric)              | Absent unless the GPU is part of a multi-node NVLink domain (e.g. GB200 NVL72).  |
| `ecc`                           | 20     | [`GpuEcc`](#gpuecc)                    |                                                                                  |
| `row_remap`                     | 21     | [`GpuRowRemap`](#gpurowremap)          | Absent on GPUs without row remapping.                                            |

### GpuMig [#gpumig]

| Field       | Number | Type                                   | Description                                                |
| ----------- | ------ | -------------------------------------- | ---------------------------------------------------------- |
| `capable`   | 1      | `bool`                                 |                                                            |
| `current`   | 2      | [`MigMode`](#migmode)                  |                                                            |
| `pending`   | 3      | [`MigMode`](#migmode)                  | Set when a mode change waits for a GPU reset.              |
| `profiles`  | 4      | repeated [`MigProfile`](#migprofile)   | GPU instance profiles, sorted by id.                       |
| `instances` | 5      | repeated [`MigInstance`](#miginstance) | Current GPU/compute instances, sorted by (gi\_id, ci\_id). |

### MigProfile [#migprofile]

| Field           | Number | Type     | Description                   |
| --------------- | ------ | -------- | ----------------------------- |
| `id`            | 1      | `uint32` | NVML GPU instance profile id. |
| `name`          | 2      | `string` | e.g. "1g.10gb".               |
| `max_instances` | 3      | `uint32` |                               |
| `memory_bytes`  | 4      | `uint64` |                               |

### MigInstance [#miginstance]

| Field     | Number | Type     | Description |
| --------- | ------ | -------- | ----------- |
| `gi_id`   | 1      | `uint32` |             |
| `ci_id`   | 2      | `uint32` |             |
| `profile` | 3      | `string` |             |
| `uuid`    | 4      | `string` | "MIG-…".    |

### NvlinkLink [#nvlinklink]

| Field     | Number | Type                      | Description                                              |
| --------- | ------ | ------------------------- | -------------------------------------------------------- |
| `link`    | 1      | `uint32`                  |                                                          |
| `state`   | 2      | [`LinkState`](#linkstate) |                                                          |
| `peer`    | 3      | `string`                  | A GPU UUID, or "nvswitch:\<pci bus id>" behind NVSwitch. |
| `version` | 4      | `uint32`                  |                                                          |

### GpuPeerLink [#gpupeerlink]

One cell of the nvidia-smi topo -m matrix.

| Field       | Number | Type                          | Description                                                                   |
| ----------- | ------ | ----------------------------- | ----------------------------------------------------------------------------- |
| `peer_uuid` | 1      | `string`                      |                                                                               |
| `link`      | 2      | [`GpuLinkType`](#gpulinktype) |                                                                               |
| `nvlinks`   | 3      | `uint32`                      | Number of NVLinks for GPU\_LINK\_TYPE\_NV (the "NV\<n>" figure); 0 otherwise. |

### GpuFabric [#gpufabric]

MNNVL / IMEX domain membership.

| Field          | Number | Type     | Description |
| -------------- | ------ | -------- | ----------- |
| `cluster_uuid` | 1      | `string` |             |
| `clique_id`    | 2      | `uint32` |             |

### GpuEcc [#gpuecc]

| Field              | Number | Type                      | Description                                |
| ------------------ | ------ | ------------------------- | ------------------------------------------ |
| `enabled`          | 1      | `bool`                    |                                            |
| `pending_enabled`  | 2      | optional `bool`           | Set when a mode change waits for a reboot. |
| `volatile_counts`  | 3      | [`EccCounts`](#ecccounts) | Since the last driver load.                |
| `aggregate_counts` | 4      | [`EccCounts`](#ecccounts) | Lifetime.                                  |

### EccCounts [#ecccounts]

| Field           | Number | Type     | Description       |
| --------------- | ------ | -------- | ----------------- |
| `correctable`   | 1      | `uint64` | Single-bit (SBE). |
| `uncorrectable` | 2      | `uint64` | Double-bit (DBE). |

### GpuRowRemap [#gpurowremap]

| Field           | Number | Type     | Description                             |
| --------------- | ------ | -------- | --------------------------------------- |
| `correctable`   | 1      | `uint32` |                                         |
| `uncorrectable` | 2      | `uint32` |                                         |
| `pending`       | 3      | `bool`   | A remap waits for a GPU reset (XID 63). |
| `failure`       | 4      | `bool`   | Remapping failed (XID 64): RMA.         |

### BmcSection [#bmcsection]

| Field              | Number | Type     | Description |
| ------------------ | ------ | -------- | ----------- |
| `ip`               | 1      | `string` |             |
| `mac`              | 2      | `string` |             |
| `vendor`           | 3      | `string` |             |
| `firmware_version` | 4      | `string` |             |
| `redfish`          | 5      | `bool`   |             |
| `ipmi`             | 6      | `bool`   |             |

### OsSection [#ossection]

| Field                  | Number | Type                              | Description                                                                                                                                                                                                                                                                                                                                          |
| ---------------------- | ------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `distro`               | 1      | `string`                          | os-release ID, e.g. "ubuntu".                                                                                                                                                                                                                                                                                                                        |
| `distro_version`       | 2      | `string`                          | os-release VERSION\_ID, e.g. "24.04".                                                                                                                                                                                                                                                                                                                |
| `kernel`               | 3      | `string`                          | uname -r.                                                                                                                                                                                                                                                                                                                                            |
| `cgroup`               | 4      | [`CgroupVersion`](#cgroupversion) |                                                                                                                                                                                                                                                                                                                                                      |
| `kvm`                  | 5      | `bool`                            |                                                                                                                                                                                                                                                                                                                                                      |
| `iommu`                | 6      | `bool`                            |                                                                                                                                                                                                                                                                                                                                                      |
| `runtimes`             | 7      | repeated `string`                 | Sorted, e.g. "containerd", "docker", "libvirt".                                                                                                                                                                                                                                                                                                      |
| `hostname`             | 8      | `string`                          |                                                                                                                                                                                                                                                                                                                                                      |
| `kubernetes_node_name` | 9      | `string`                          | (v1.1) The Kubernetes Node this machine is, when bcd runs on it as a DaemonSet pod: the pod's spec.nodeName (NODE\_NAME). Empty anywhere else. It is the Node's name whatever the kernel hostname says (some EKS AMIs keep a short hostname while the Node is the instance's DNS name), so a pool substrate matches the pool node to its Node by it. |

### CachedImageState [#cachedimagestate]

| Value                            | Number | Description |
| -------------------------------- | ------ | ----------- |
| `CACHED_IMAGE_STATE_UNSPECIFIED` | 0      |             |
| `CACHED_IMAGE_STATE_READY`       | 1      |             |
| `CACHED_IMAGE_STATE_PULLING`     | 2      |             |
| `CACHED_IMAGE_STATE_EVICTING`    | 3      |             |

### BootMode [#bootmode]

| Value                   | Number | Description |
| ----------------------- | ------ | ----------- |
| `BOOT_MODE_UNSPECIFIED` | 0      |             |
| `BOOT_MODE_UEFI`        | 1      |             |
| `BOOT_MODE_BIOS`        | 2      |             |

### DiskTransport [#disktransport]

| Value                        | Number | Description |
| ---------------------------- | ------ | ----------- |
| `DISK_TRANSPORT_UNSPECIFIED` | 0      |             |
| `DISK_TRANSPORT_NVME`        | 1      |             |
| `DISK_TRANSPORT_SATA`        | 2      |             |
| `DISK_TRANSPORT_SAS`         | 3      |             |
| `DISK_TRANSPORT_VIRTIO`      | 4      |             |
| `DISK_TRANSPORT_USB`         | 5      |             |

### RdmaType [#rdmatype]

| Value                   | Number | Description |
| ----------------------- | ------ | ----------- |
| `RDMA_TYPE_UNSPECIFIED` | 0      |             |
| `RDMA_TYPE_NONE`        | 1      |             |
| `RDMA_TYPE_INFINIBAND`  | 2      |             |
| `RDMA_TYPE_ROCE`        | 3      |             |

### LinkState [#linkstate]

| Value                    | Number | Description |
| ------------------------ | ------ | ----------- |
| `LINK_STATE_UNSPECIFIED` | 0      |             |
| `LINK_STATE_UP`          | 1      |             |
| `LINK_STATE_DOWN`        | 2      |             |

### MigMode [#migmode]

| Value                  | Number | Description |
| ---------------------- | ------ | ----------- |
| `MIG_MODE_UNSPECIFIED` | 0      |             |
| `MIG_MODE_DISABLED`    | 1      |             |
| `MIG_MODE_ENABLED`     | 2      |             |

### GpuLinkType [#gpulinktype]

Best to worst: NV > PIX > PXB > PHB > NODE > SYS (PLAN.md §10.3 topology scorer).

| Value                       | Number | Description                                         |
| --------------------------- | ------ | --------------------------------------------------- |
| `GPU_LINK_TYPE_UNSPECIFIED` | 0      |                                                     |
| `GPU_LINK_TYPE_NV`          | 1      | Connected by NVLink (directly or through NVSwitch). |
| `GPU_LINK_TYPE_PIX`         | 2      | At most one PCIe bridge.                            |
| `GPU_LINK_TYPE_PXB`         | 3      | Multiple PCIe bridges, no host bridge.              |
| `GPU_LINK_TYPE_PHB`         | 4      | A PCIe host bridge.                                 |
| `GPU_LINK_TYPE_NODE`        | 5      | Across host bridges within one NUMA node.           |
| `GPU_LINK_TYPE_SYS`         | 6      | Across NUMA nodes (SMP interconnect).               |

### CgroupVersion [#cgroupversion]

| Value                        | Number | Description |
| ---------------------------- | ------ | ----------- |
| `CGROUP_VERSION_UNSPECIFIED` | 0      |             |
| `CGROUP_VERSION_V1`          | 1      |             |
| `CGROUP_VERSION_V2`          | 2      |             |

## session.proto [#sessionproto]

Package `bettercompute.agent.v1`, `proto/bettercompute/agent/v1/session.proto`.

The daemon ↔ control-plane protocol (PLAN.md §7): join, renew, the bidi Session stream and its frames, desired state (NodeSpec) and tunnels.

### AgentGateway [#agentgateway]

The gateway-facing service every bcd mode (node, site, ramdisk) talks to.

| RPC       | Request                              | Response                             | Description                                                                                                                                            |
| --------- | ------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Join`    | [`JoinRequest`](#joinrequest)        | [`JoinResponse`](#joinresponse)      | Server-authenticated TLS (serving CA pinned). Forwarded to the api JoinService; the gateway cannot read the sealed token or forge the signed response. |
| `Renew`   | [`RenewRequest`](#renewrequest)      | [`RenewResponse`](#renewresponse)    | mTLS, at \~60% of the certificate lifetime. Verified by the api JoinService, not the gateway.                                                          |
| `Session` | stream [`AgentFrame`](#agentframe)   | stream [`ServerFrame`](#serverframe) | mTLS. One live session per node; a newer session supersedes an older one (ABORTED).                                                                    |
| `Tunnel`  | stream [`TunnelFrame`](#tunnelframe) | stream [`TunnelFrame`](#tunnelframe) | (v0.3) mTLS. Opened by the daemon after it verified an OpenTunnel ticket.                                                                              |

### JoinRequest [#joinrequest]

| Field            | Number | Type                                | Description                                                                                                                                                                     |
| ---------------- | ------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sealed_token`   | 1      | `bytes`                             | UTF-8 compact JWE of the join token ("bcjt\_\<id>.\<secret>") to the pinned JoinService encryption key. The token is read from BCD\_JOIN\_TOKEN or stdin only and never stored. |
| `csr`            | 2      | `bytes`                             | DER PKCS#10 CSR for a fresh P-256 key (TPM-resident when available).                                                                                                            |
| `claims`         | 3      | [`HardwareClaims`](#hardwareclaims) |                                                                                                                                                                                 |
| `attestor`       | 4      | `string`                            | Attestor that verifies this join: "join-token" (v0.1); "x509pop", "provisioned", "discovery" (v0.2); "cloud-iid" (v0.3); "tpm" (v0.5). Empty means "join-token".                |
| `attestation`    | 5      | `bytes`                             | Attestor-specific evidence (v0.2+), e.g. a site agent's SignedDiscoveryAssertion.                                                                                               |
| `daemon_version` | 6      | `string`                            |                                                                                                                                                                                 |
| `role`           | 7      | [`AgentRole`](#agentrole)           | The role the agent runs as. The token decides what is allowed; a mismatch is rejected.                                                                                          |

### HardwareClaims [#hardwareclaims]

What the daemon reports about its hardware at join. JoinService derives `hardware_key` =
sha256(product\_uuid ‖ product\_serial ‖ sorted permanent MACs ‖ sorted GPU UUIDs) after dropping
placeholder values, and enforces the token's `expect` binding and reclaim rules.

| Field            | Number | Type              | Description                                                     |
| ---------------- | ------ | ----------------- | --------------------------------------------------------------- |
| `product_uuid`   | 1      | `string`          | /sys/class/dmi/id/product\_uuid, lowercase.                     |
| `product_serial` | 2      | `string`          | /sys/class/dmi/id/product\_serial.                              |
| `board_serial`   | 3      | `string`          |                                                                 |
| `chassis_serial` | 4      | `string`          |                                                                 |
| `sys_vendor`     | 5      | `string`          |                                                                 |
| `product_name`   | 6      | `string`          |                                                                 |
| `permanent_macs` | 7      | repeated `string` | Permanent MAC addresses of physical NICs, lowercase colon form. |
| `gpu_uuids`      | 8      | repeated `string` |                                                                 |
| `bmc_mac`        | 9      | `string`          | BMC MAC when readable in-band.                                  |
| `hostname`       | 10     | `string`          |                                                                 |
| `boot_id`        | 11     | `string`          |                                                                 |

### JoinResponse [#joinresponse]

| Field  | Number | Type                      | Description                                                             |
| ------ | ------ | ------------------------- | ----------------------------------------------------------------------- |
| `body` | 1      | `bytes`                   | Serialized JoinResponseBody.                                            |
| `sig`  | 2      | [`Signature`](#signature) | Pinned JoinService signing key. Purpose "join-response", parts \[body]. |

### JoinResponseBody [#joinresponsebody]

| Field          | Number | Type                            | Description                                                                                                                                                                                                                                                                                                                    |
| -------------- | ------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `node_id`      | 1      | `string`                        |                                                                                                                                                                                                                                                                                                                                |
| `cert_chain`   | 2      | repeated `bytes`                | DER certificates, leaf first. The leaf has URI SAN spiffe://\<td>/site/\<site>/node/\<node\_id> and a role OID; no tenant.                                                                                                                                                                                                     |
| `trust`        | 3      | [`TrustBundle`](#trustbundle)   | Command/spec signing trust (empty roots in v0.1).                                                                                                                                                                                                                                                                              |
| `csr_sha256`   | 4      | `bytes`                         | SHA-256 of JoinRequest.csr: the daemon rejects a response that is not for its own CSR.                                                                                                                                                                                                                                         |
| `trust_domain` | 5      | `string`                        |                                                                                                                                                                                                                                                                                                                                |
| `site_id`      | 6      | `string`                        |                                                                                                                                                                                                                                                                                                                                |
| `role`         | 7      | [`AgentRole`](#agentrole)       |                                                                                                                                                                                                                                                                                                                                |
| `issued_at`    | 8      | `google.protobuf.Timestamp`     | Server time at issue, for a first clock-offset estimate.                                                                                                                                                                                                                                                                       |
| `deferral`     | 9      | [`JoinDeferral`](#joindeferral) | (v0.2) Set when the attestor could not decide yet: node\_id and cert\_chain are empty, nothing was issued and no token was spent. The daemon re-sends Join with the SAME csr (csr\_sha256 still binds this response to it): after `pending.retry_after_seconds`, or with its answer to `challenge` in JoinRequest.attestation. |

### JoinDeferral [#joindeferral]

A two-round join (§3.3): an operator approval (discovery → joinRequests) or an attestor challenge
(tpm credential activation).

| Field       | Number | Type                                             | Description |
| ----------- | ------ | ------------------------------------------------ | ----------- |
| `pending`   | 1      | [`JoinPending`](#joinpending) (oneof `kind`)     |             |
| `challenge` | 2      | [`JoinChallenge`](#joinchallenge) (oneof `kind`) |             |

### JoinPending [#joinpending]

| Field                 | Number | Type     | Description                                                         |
| --------------------- | ------ | -------- | ------------------------------------------------------------------- |
| `request_id`          | 1      | `string` | The join request an operator approves or rejects (joinRequests.\*). |
| `retry_after_seconds` | 2      | `uint32` |                                                                     |

### JoinChallenge [#joinchallenge]

| Field       | Number | Type                        | Description                                                                         |
| ----------- | ------ | --------------------------- | ----------------------------------------------------------------------------------- |
| `attestor`  | 1      | `string`                    | The attestor that answers it, as in JoinRequest.attestor.                           |
| `payload`   | 2      | `bytes`                     | Attestor-specific (tpm: credential blob, encrypted secret and opaque server state). |
| `not_after` | 3      | `google.protobuf.Timestamp` | Answers after this are refused.                                                     |

### RenewRequest [#renewrequest]

| Field             | Number | Type                                      | Description                                                                                                                                                                                                                                                                    |
| ----------------- | ------ | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `csr`             | 1      | `bytes`                                   | DER PKCS#10 CSR (a new key, or the current one).                                                                                                                                                                                                                               |
| `chain`           | 2      | repeated `bytes`                          | The current certificate chain, leaf first.                                                                                                                                                                                                                                     |
| `client_time`     | 3      | `google.protobuf.Timestamp`               |                                                                                                                                                                                                                                                                                |
| `pop_sig`         | 4      | `bytes`                                   | Proof of possession by the CURRENT node key. Purpose "renew-pop", parts \[sha256(csr), node\_id, client\_time]. Accepted within ±60 s of server time; 120 s replay cache.                                                                                                      |
| `key_attestation` | 5      | [`TpmKeyAttestation`](#tpmkeyattestation) | Required when the current certificate names a TPM attestation key (URI SAN urn:bc:tpm-ak:\<nameAlg>:\<digest>): that AK certifies the CSR's key as one the TPM holds. The new certificate carries the statement forward from the old one; without this the renewal is refused. |

### TpmKeyAttestation [#tpmkeyattestation]

TPM2\_Certify of a node key by the AK a certificate names, qualifying data sha256(csr).

| Field          | Number | Type    | Description                                                                                  |
| -------------- | ------ | ------- | -------------------------------------------------------------------------------------------- |
| `ak_public`    | 1      | `bytes` | TPMT\_PUBLIC of the AK: its TPM name must be the one in the certificate.                     |
| `key_public`   | 2      | `bytes` | TPMT\_PUBLIC of the node key: the CSR's key, fixedTPM \| fixedParent \| sensitiveDataOrigin. |
| `certify_info` | 3      | `bytes` | The TPMS\_ATTEST (TPM\_ST\_ATTEST\_CERTIFY) and the AK's TPMT\_SIGNATURE over it.            |
| `signature`    | 4      | `bytes` |                                                                                              |

### RenewResponse [#renewresponse]

| Field        | Number | Type                        | Description                   |
| ------------ | ------ | --------------------------- | ----------------------------- |
| `cert_chain` | 1      | repeated `bytes`            | DER certificates, leaf first. |
| `not_after`  | 2      | `google.protobuf.Timestamp` |                               |

### AgentFrame [#agentframe]

| Field                    | Number | Type                                                          | Description                                                                        |
| ------------------------ | ------ | ------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `hello`                  | 1      | [`Hello`](#hello) (oneof `msg`)                               |                                                                                    |
| `heartbeat`              | 2      | [`Heartbeat`](#heartbeat) (oneof `msg`)                       |                                                                                    |
| `update`                 | 3      | [`CommandUpdate`](#commandupdate) (oneof `msg`)               |                                                                                    |
| `events`                 | 4      | [`EventBatch`](#eventbatch) (oneof `msg`)                     |                                                                                    |
| `status`                 | 5      | [`StatusReport`](#statusreport) (oneof `msg`)                 |                                                                                    |
| `inventory`              | 6      | [`InventorySection`](#inventorysection) (oneof `msg`)         |                                                                                    |
| `spec_request`           | 7      | [`SpecRequest`](#specrequest) (oneof `msg`)                   |                                                                                    |
| `secret_request`         | 8      | [`SecretRequest`](#secretrequest) (oneof `msg`)               |                                                                                    |
| `workload_token_request` | 9      | [`WorkloadTokenRequest`](#workloadtokenrequest) (oneof `msg`) | (v0.5) A hosted instance's IMDS asks for its workload identity token (§12.2, §13). |

### ServerFrame [#serverframe]

| Field            | Number | Type                                                            | Description                                  |
| ---------------- | ------ | --------------------------------------------------------------- | -------------------------------------------- |
| `welcome`        | 1      | [`Welcome`](#welcome) (oneof `msg`)                             |                                              |
| `command`        | 2      | [`Command`](#command) (oneof `msg`)                             |                                              |
| `cancel`         | 3      | [`Cancel`](#cancel) (oneof `msg`)                               |                                              |
| `desired`        | 4      | [`DesiredState`](#desiredstate) (oneof `msg`)                   |                                              |
| `ack`            | 5      | [`Ack`](#ack) (oneof `msg`)                                     |                                              |
| `open_tunnel`    | 6      | [`OpenTunnel`](#opentunnel) (oneof `msg`)                       |                                              |
| `drain`          | 7      | [`Drain`](#drain) (oneof `msg`)                                 |                                              |
| `revoked`        | 8      | [`RevocationNotice`](#revocationnotice) (oneof `msg`)           |                                              |
| `secret`         | 9      | [`SecretResponse`](#secretresponse) (oneof `msg`)               |                                              |
| `workload_token` | 10     | [`WorkloadTokenResponse`](#workloadtokenresponse) (oneof `msg`) | (v0.5) The answer to a WorkloadTokenRequest. |

### Hello [#hello]

The first frame of every session. Kept small for reconnect storms: inventory travels as hashes.

| Field                 | Number | Type                                         | Description                                                                                                                                                                                                                                                                                                                                      |
| --------------------- | ------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `node_id`             | 1      | `string`                                     | Must equal the node id in the certificate SAN.                                                                                                                                                                                                                                                                                                   |
| `boot_id`             | 2      | `string`                                     |                                                                                                                                                                                                                                                                                                                                                  |
| `journal_id`          | 3      | `string`                                     | The journal's id. A change (journal lost: ramdisk reboot, reinstall, corruption) makes the control plane fail in-flight commands with retry\_safe=false ("journal-lost") instead of replaying them.                                                                                                                                              |
| `role`                | 4      | [`AgentRole`](#agentrole)                    |                                                                                                                                                                                                                                                                                                                                                  |
| `daemon_version`      | 5      | `string`                                     |                                                                                                                                                                                                                                                                                                                                                  |
| `proto_min`           | 6      | `uint32`                                     | Supported protocol range (PROTOCOL\_VERSION in @better-compute/protocol). Below the server's minimum the session is restricted to daemon.upgrade (Welcome.upgrade\_required).                                                                                                                                                                    |
| `proto_max`           | 7      | `uint32`                                     |                                                                                                                                                                                                                                                                                                                                                  |
| `capabilities`        | 8      | repeated `string`                            |                                                                                                                                                                                                                                                                                                                                                  |
| `drivers`             | 9      | repeated [`DriverManifest`](#drivermanifest) |                                                                                                                                                                                                                                                                                                                                                  |
| `last_epoch`          | 10     | `uint64`                                     | The last session epoch this daemon saw (0 if none); diagnostics only.                                                                                                                                                                                                                                                                            |
| `applied_generation`  | 11     | `uint64`                                     | The newest NodeSpec generation the daemon holds (converged or not).                                                                                                                                                                                                                                                                              |
| `unacked_results`     | 12     | repeated [`CommandUpdate`](#commandupdate)   | Terminal results not yet acked; the control plane dedupes by command\_id.                                                                                                                                                                                                                                                                        |
| `running_command_ids` | 13     | repeated `string`                            | Commands the journal holds as running; the control plane does not re-send them.                                                                                                                                                                                                                                                                  |
| `last_event_seq`      | 14     | `uint64`                                     | The highest event seq written in this journal.                                                                                                                                                                                                                                                                                                   |
| `inventory_hashes`    | 15     | map of `string` to `bytes`                   | Inventory section name → SHA-256 of its InventorySection.content. The control plane asks for sections whose hash differs (Welcome.request\_sections).                                                                                                                                                                                            |
| `applied_spec_hash`   | 16     | `bytes`                                      | DesiredStateBody.spec\_hash of the applied generation, echoed so the control plane can detect a divergent spec and resend a full one.                                                                                                                                                                                                            |
| `trust_version`       | 17     | `uint64`                                     | TrustBundle.version the daemon holds and verifies with (0: none). The gateway records it (node\_sessions.trust\_version) so a command-root switch waits for every connected node (§13).                                                                                                                                                          |
| `approver_set_epoch`  | 18     | `uint64`                                     | (v1.0 §13) The approver set the daemon HOLDS: a bundle's version can be taken while its quorum part is refused (unsigned change, stale epoch), so trust\_version alone would read "up to date" for a node still enforcing the old approvers. approver\_set\_sha256 is SHA-256 of the held set's approverSetInput (empty: no approver keys held). |
| `approver_set_sha256` | 19     | `bytes`                                      |                                                                                                                                                                                                                                                                                                                                                  |

### Welcome [#welcome]

| Field                   | Number | Type                                      | Description                                                                                                                         |
| ----------------------- | ------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `epoch`                 | 1      | `uint64`                                  | node\_sessions.session\_epoch after this Hello's bump.                                                                              |
| `session_id`            | 2      | `string`                                  |                                                                                                                                     |
| `gateway_id`            | 3      | `string`                                  |                                                                                                                                     |
| `protocol_version`      | 4      | `uint32`                                  | The negotiated version: the highest version inside both ranges.                                                                     |
| `heartbeat_interval_ms` | 5      | `uint32`                                  | Heartbeat period (10 000 ms by default). Three missed heartbeats close the session.                                                 |
| `lease_ttl_ms`          | 6      | `uint32`                                  | Liveness lease TTL (40 000 ms by default).                                                                                          |
| `credits`               | 7      | repeated [`LaneCredits`](#lanecredits)    | Per-lane in-flight limits the gateway keeps outstanding (control 8, provision 4, bulk 2).                                           |
| `acked_event_seq`       | 8      | `uint64`                                  | Cumulative ack for Hello.journal\_id; the daemon resends events with a higher seq.                                                  |
| `request_sections`      | 9      | repeated `string`                         | Inventory sections to send now (hash mismatch or never seen).                                                                       |
| `server_time`           | 10     | `google.protobuf.Timestamp`               | Server time (compute\_now()). The daemon keeps the offset to its own clock and evaluates not\_after and not\_before in server time. |
| `trust`                 | 11     | [`SignedTrustBundle`](#signedtrustbundle) | Newer signing trust, if any. Only accepted when signed by the pinned JoinService key and newer than the bundle the daemon holds.    |
| `upgrade_required`      | 12     | `bool`                                    | The daemon is below the server's proto\_min: only daemon.upgrade is accepted (v0.4).                                                |
| `capabilities`          | 13     | repeated `string`                         |                                                                                                                                     |

### LaneCredits [#lanecredits]

| Field     | Number | Type            | Description |
| --------- | ------ | --------------- | ----------- |
| `lane`    | 1      | [`Lane`](#lane) |             |
| `credits` | 2      | `uint32`        |             |

### Heartbeat [#heartbeat]

| Field                | Number | Type                                   | Description                                                                         |
| -------------------- | ------ | -------------------------------------- | ----------------------------------------------------------------------------------- |
| `credits`            | 1      | repeated [`LaneCredits`](#lanecredits) | Free credits per lane right now; the gateway resynchronises its counters from them. |
| `load`               | 2      | [`NodeLoad`](#nodeload)                |                                                                                     |
| `inventory_hashes`   | 3      | map of `string` to `bytes`             | Every 10th heartbeat: the same map as Hello.inventory\_hashes; empty otherwise.     |
| `applied_generation` | 4      | `uint64`                               |                                                                                     |

### NodeLoad [#nodeload]

Coarse load for admission and dashboards. Telemetry proper never uses the session (P8).

| Field                    | Number | Type     | Description                              |
| ------------------------ | ------ | -------- | ---------------------------------------- |
| `load_average_1m`        | 1      | `double` | 1-minute load average.                   |
| `memory_available_bytes` | 2      | `uint64` |                                          |
| `running_commands`       | 3      | `uint32` | Commands currently running across lanes. |
| `pending_events`         | 4      | `uint32` | Unacked events in the journal.           |

### Command [#command]

A one-shot action. Delivery is at-least-once; the daemon's journal makes it at-most-once in
effect. The control plane stores `body` and `signature` verbatim and resends the same bytes.

| Field            | Number | Type                              | Description                                                                                                                                                                                                                    |
| ---------------- | ------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `body`           | 1      | `bytes`                           | Serialized CommandBody.                                                                                                                                                                                                        |
| `signature`      | 2      | [`Signature`](#signature)         | Worker signing key. Purpose "command", parts \[body]. Optional in v0.1; required for destructive kinds from v0.2 and for all kinds from v0.4 (the daemon's own kind table decides, never a flag an unsigned body could carry). |
| `attempt`        | 3      | `uint32`                          | Delivery attempt (1-based). Outside the signature: informational only.                                                                                                                                                         |
| `batch_approval` | 4      | [`BatchApproval`](#batchapproval) | (v1.0, §13 quorum) For a body with batch\_id: the operators' approval of the batch and this command's inclusion proof. Outside the worker signature: the approval carries its own.                                             |
| `batch_proof`    | 5      | [`BatchProof`](#batchproof)       |                                                                                                                                                                                                                                |

### BatchApproval [#batchapproval]

k-of-n operator approval of a batch of destructive commands (PLAN.md §13). The scope is a Merkle
tree (RFC 6962 style, SHA-256) with one leaf per (subject, kind, payload), leaves ordered by their
hash (bytewise ascending, duplicates removed):
leaf = H(0x00 ‖ lp(batch\_id) ‖ lp(subject) ‖ lp(kind) ‖ lp(payload\_sha256)),
node = H(0x01 ‖ left ‖ right),
with lp(x) = uint32\_be(len(x)) ‖ x, and an odd node promoted unchanged to the next level.
The subject is what the command acts on, computed by the executing agent from the signed body:

* kinds a node runs on itself (disk.wipe, deploy.image, node.reboot, gpu.reset, boot.kexec…):
  the command's node\_id;
* site-relayed BMC kinds (bmc.\*): "bmc:" + \<the executing site agent's node\_id> + "/" +
  BmcOperationRequest.address, the BMC acted on. BMC addresses are site-local (10.0.0.5 exists
  at every site), so the site is part of the subject: an approval for site A never admits a
  command at site B;
* site.httpProxy: "node:" + its target\_node\_id (global ids).
  payload\_sha256 is SHA-256 of the command's payload (CommandBody.payload.value), binding the leaf
  to exactly that operation (image, firmware, power action), or EMPTY for "any payload of this kind"
  where the payload is built later (approvers see unbound leaves flagged). The executing agent
  accepts a proof that walks to the root from either the bound leaf (its payload's digest) or the
  unbound one. Daemons run each leaf at most once per batch.
  Signing purpose "batch-approval", parts \[batch\_id, scope\_root, uint32\_be(scope\_size),
  int64\_be(not\_after.seconds) ‖ int32\_be(not\_after.nanos), trust\_domain]. The trust domain (the
  deployment's, JoinResponseBody.trust\_domain) is supplied by each verifier from its own config, so
  an approval for staging never admits a command in production, even with shared approver keys.

| Field        | Number | Type                               | Description                                                                           |
| ------------ | ------ | ---------------------------------- | ------------------------------------------------------------------------------------- |
| `batch_id`   | 1      | `string`                           |                                                                                       |
| `scope_root` | 2      | `bytes`                            |                                                                                       |
| `scope_size` | 3      | `uint32`                           | Leaves in the tree: what the operators saw and approved ("reimage 412 nodes").        |
| `not_after`  | 4      | `google.protobuf.Timestamp`        | Server time after which the approval no longer admits a command.                      |
| `signatures` | 5      | repeated [`Signature`](#signature) | Approver key signatures (Signature.kid names an approver key); duplicates count once. |

### BatchProof [#batchproof]

The path from this command's leaf to BatchApproval.scope\_root.

| Field      | Number | Type             | Description                                                                              |
| ---------- | ------ | ---------------- | ---------------------------------------------------------------------------------------- |
| `index`    | 1      | `uint32`         | The leaf's index in the sorted leaves.                                                   |
| `siblings` | 2      | repeated `bytes` | Sibling hashes from the leaf level up; a promoted odd node has no sibling at that level. |

### CommandBody [#commandbody]

| Field              | Number | Type                        | Description                                                                                                                                                                                                                                                           |
| ------------------ | ------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`               | 1      | `string`                    | ULID; the journal's dedupe key.                                                                                                                                                                                                                                       |
| `node_id`          | 2      | `string`                    | Must equal the session's node id; anything else is rejected UNSUPPORTED.                                                                                                                                                                                              |
| `seq`              | 3      | `uint64`                    | Per-node monotonic sequence (nodes.next\_cmd\_seq); resume replays in seq order.                                                                                                                                                                                      |
| `kind`             | 4      | `string`                    | Registered kind, e.g. "diag.nvml". Unknown or role-disallowed kinds → REJECTED/UNSUPPORTED.                                                                                                                                                                           |
| `payload`          | 5      | `google.protobuf.Any`       | The kind's request message (commands.proto), e.g. bettercompute.agent.v1.DiagNvmlRequest.                                                                                                                                                                             |
| `lane`             | 6      | [`Lane`](#lane)             |                                                                                                                                                                                                                                                                       |
| `not_after`        | 7      | `google.protobuf.Timestamp` | Server time after which the command must not start. Checked before accepting (REJECTED/ EXPIRED); the CommandSweeper expires undelivered commands at the same instant.                                                                                                |
| `min_generation`   | 8      | `uint64`                    | The NodeSpec generation the command presupposes. If the daemon's applied generation is lower it sends SpecRequest and defers; still lower at not\_after → REJECTED/STALE\_GENERATION. 0 = none.                                                                       |
| `retry_safe`       | 9      | `bool`                      | Safe to execute again when the journal shows `accepted` but not `running` after a crash. Otherwise that case reports FAILED\{"interrupted"} for the controller to decide.                                                                                             |
| `idempotency_key`  | 10     | `string`                    | UNIQUE(node\_id, idempotency\_key); workflow steps use "wf:\<id>:\<stepId>".                                                                                                                                                                                          |
| `traceparent`      | 11     | `string`                    | W3C trace context, propagated api → worker → gateway → daemon → driver.                                                                                                                                                                                               |
| `self_terminating` | 12     | `bool`                      | The side effect ends this daemon process (node.reboot, boot.kexec, daemon.upgrade). The journal fsyncs running\{boot\_id} right before it; after restart with another boot\_id (or on rejoin with another role) it completes as SUCCEEDED\{rebooted}. Never replayed. |
| `attested`         | 13     | `bool`                      | The result is a "this node says" statement: CommandUpdate.node\_sig is required (v0.2) and the worker verifies it against node\_certs before any state transition.                                                                                                    |
| `workflow_id`      | 14     | `string`                    | (v0.2) The workflow step this command executes; SecretRequest binds secrets to it.                                                                                                                                                                                    |
| `step_id`          | 15     | `string`                    |                                                                                                                                                                                                                                                                       |
| `batch_id`         | 16     | `string`                    | (v1.0, §13 quorum) The batch this destructive command belongs to. Inside the signed body, so a compromised gateway can neither add nor strip it; the daemon then requires Command.batch\_approval + batch\_proof.                                                     |

### CommandUpdate [#commandupdate]

Progress and outcome of one command. Terminal updates are journaled until acked and replayed in
Hello.unacked\_results after a reconnect.

| Field           | Number | Type                                          | Description                                                                                                             |
| --------------- | ------ | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `command_id`    | 1      | `string`                                      |                                                                                                                         |
| `state`         | 2      | [`CommandState`](#commandstate)               | One of ACKED, RUNNING, SUCCEEDED, FAILED, REJECTED, CANCELLED.                                                          |
| `result`        | 3      | `bytes`                                       | The kind's serialized result message (commands.proto), when there is one.                                               |
| `boot_id`       | 4      | `string`                                      | The boot the daemon was in when it produced this update.                                                                |
| `node_sig`      | 5      | [`Signature`](#signature)                     | Node key, for attested kinds (v0.2). Purpose "command-update", parts \[command\_id, uint32\_be(state), sha256(result)]. |
| `reject_reason` | 6      | [`CommandRejectReason`](#commandrejectreason) | Set when state is REJECTED.                                                                                             |
| `error`         | 7      | [`CommandError`](#commanderror)               | Set when state is FAILED.                                                                                               |
| `rebooted`      | 8      | `bool`                                        | A self-terminating command completed by the reboot/exec it caused.                                                      |

### CommandError [#commanderror]

| Field     | Number | Type     | Description                                                                                                                                           |
| --------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `code`    | 1      | `string` | Stable lowercase code: "interrupted" (accepted, not retry-safe, daemon restarted), "invalid-payload", "internal", or a kind- or driver-specific code. |
| `message` | 2      | `string` |                                                                                                                                                       |

### Cancel [#cancel]

Stop a command between actions. Never hides a completed side effect: the daemon reports the real
outcome, or CANCELLED if nothing ran.

| Field        | Number | Type     | Description |
| ------------ | ------ | -------- | ----------- |
| `command_id` | 1      | `string` |             |
| `reason`     | 2      | `string` |             |

### DesiredState [#desiredstate]

Generation-numbered desired state. Signed like commands; the daemon stores the last full spec,
its hash and signature in the journal and enforces it while offline.

| Field       | Number | Type                      | Description                                                                                                                                        |
| ----------- | ------ | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `body`      | 1      | `bytes`                   | Serialized DesiredStateBody.                                                                                                                       |
| `signature` | 2      | [`Signature`](#signature) | Worker signing key. Purpose "desired-state", parts \[body]. Optional in v0.1; required when the spec changes `access` (v0.2) and always from v0.4. |

### DesiredStateBody [#desiredstatebody]

| Field             | Number | Type                                   | Description                                                                                                                                                            |
| ----------------- | ------ | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node_id`         | 1      | `string`                               |                                                                                                                                                                        |
| `generation`      | 2      | `uint64`                               | Monotonic. A generation not greater than the applied one is ignored.                                                                                                   |
| `base_generation` | 3      | `uint64`                               | 0 for a full spec. For a patch: the generation it applies to; a daemon at any other generation discards the patch and sends SpecRequest\{PATCH\_BASE\_MISMATCH}.       |
| `spec_hash`       | 4      | `bytes`                                | Opaque id of the full spec at `generation` (SHA-256 over the control plane's canonical encoding). The daemon never recomputes it; it echoes it as applied\_spec\_hash. |
| `full`            | 5      | [`NodeSpec`](#nodespec) (oneof `spec`) |                                                                                                                                                                        |
| `patch`           | 6      | [`NodeSpec`](#nodespec) (oneof `spec`) | Section-level patch: every section that is SET replaces that section wholesale (an empty section message clears it); unset sections keep their current value.          |
| `traceparent`     | 7      | `string`                               |                                                                                                                                                                        |

### NodeSpec [#nodespec]

What the node should look like. The converger runs one idempotent plan(desired, observed) +
apply per section; observed\_generation advances only when every section converged or failed
terminally. A section a daemon does not know arrives as an unknown field: the daemon reports
condition "SectionUnsupported" (reason = the field number) and converges the rest.

| Field       | Number | Type                                    | Description                                                                                 |
| ----------- | ------ | --------------------------------------- | ------------------------------------------------------------------------------------------- |
| `host`      | 1      | [`HostSection`](#hostsection)           |                                                                                             |
| `devices`   | 2      | [`DevicesSection`](#devicessection)     |                                                                                             |
| `instances` | 3      | [`InstancesSection`](#instancessection) | (v0.3)                                                                                      |
| `access`    | 4      | [`AccessSection`](#accesssection)       | (v0.3) Empty in v0.1.                                                                       |
| `telemetry` | 5      | [`TelemetrySection`](#telemetrysection) |                                                                                             |
| `daemon`    | 6      | [`DaemonSection`](#daemonsection)       | (v0.4)                                                                                      |
| `site`      | 7      | [`SiteSpec`](#sitespec)                 | (v0.2, site agents only) DRAFT: its content is defined in site/v1, which is not yet frozen. |

### HostSection [#hostsection]

Host configuration units (profile hostConfig, v0.2): nvidia-driver, container-toolkit,
mig-layout, fabric-manager, sysctl, … Converged continuously so drift is corrected.

| Field   | Number | Type                                 | Description |
| ------- | ------ | ------------------------------------ | ----------- |
| `units` | 1      | repeated [`ConfigUnit`](#configunit) |             |

### ConfigUnit [#configunit]

| Field         | Number | Type     | Description                                                                           |
| ------------- | ------ | -------- | ------------------------------------------------------------------------------------- |
| `unit`        | 1      | `string` | The config driver id, e.g. "nvidia-driver".                                           |
| `config_json` | 2      | `string` | Canonical JSON (RFC 8785) validated against the unit's manifest by the control plane. |

### DevicesSection [#devicessection]

Device assignments (§3.4 step 9). The daemon rejects overlapping assignments, writes the CDI spec
/etc/cdi/bcd-\<allocation\_id>.json, an env file and device ownership, then reports a node-signed
AllocationBound; removing an assignment releases it (AllocationReleased). The daemon never drops
an assignment on its own, not even while disconnected (P6).

| Field         | Number | Type                                             | Description |
| ------------- | ------ | ------------------------------------------------ | ----------- |
| `assignments` | 1      | repeated [`DeviceAssignment`](#deviceassignment) |             |

### DeviceAssignment [#deviceassignment]

| Field            | Number | Type                                         | Description                                                                                                                                                                                                                                                                                      |
| ---------------- | ------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `allocation_id`  | 1      | `string`                                     |                                                                                                                                                                                                                                                                                                  |
| `reservation_id` | 2      | `string`                                     |                                                                                                                                                                                                                                                                                                  |
| `binding`        | 3      | [`Binding`](#binding)                        |                                                                                                                                                                                                                                                                                                  |
| `devices`        | 4      | repeated [`AssignedDevice`](#assigneddevice) |                                                                                                                                                                                                                                                                                                  |
| `instance_id`    | 5      | `string`                                     | (v0.3) Set when binding is BINDING\_INSTANCE.                                                                                                                                                                                                                                                    |
| `env`            | 6      | repeated `string`                            | Extra "KEY=VALUE" lines for the allocation's env file.                                                                                                                                                                                                                                           |
| `limits`         | 7      | [`ResourceLimits`](#resourcelimits)          | (v1.1 §3.8) The CPU, memory and hugepages booked with the devices. For HOLD, bcd creates the cgroup-v2 slice bcd-alloc-\<allocation\_id>.slice with these limits and names it in the CDI spec; for KUBERNETES they are what the kubelet may give pods using these devices. Absent: devices only. |

### ResourceLimits [#resourcelimits]

Host resources an allocation may use (cgroup v2 semantics). Zero or empty = not limited.

| Field           | Number | Type     | Description                                                                    |
| --------------- | ------ | -------- | ------------------------------------------------------------------------------ |
| `cpus`          | 1      | `string` | CPUs the allocation may run on (cpuset.cpus, cpulist form, e.g. "8-15,72-79"). |
| `mems`          | 2      | `string` | NUMA nodes its memory comes from (cpuset.mems, e.g. "1").                      |
| `cpu_millis`    | 3      | `uint32` | CPU time, in thousandths of a CPU (cpu.max; 4000 = 4 CPUs).                    |
| `memory_bytes`  | 4      | `uint64` | memory.max; bcd sets memory.high at 90% of it.                                 |
| `hugepages_2mi` | 5      | `uint64` | hugetlb.2MB.max / hugetlb.1GB.max, in pages.                                   |
| `hugepages_1gi` | 6      | `uint64` |                                                                                |

### AssignedDevice [#assigneddevice]

| Field          | Number | Type     | Description                                                                                  |
| -------------- | ------ | -------- | -------------------------------------------------------------------------------------------- |
| `id`           | 1      | `string` | Atom id: GPU UUID, "\<gpuUuid>/gi\<g>/ci\<c>" (v0.5), "\<gpuUuid>/s\<k>" (v0.5), disk WWN, … |
| `device_class` | 2      | `string` | Device class: "gpu", "gpu.mig", "gpu.slot", "nic.vf", "disk".                                |

### InstancesSection [#instancessection]

(v0.3) VPS instances on this host; the daemon converges each toward desired\_state.

| Field       | Number | Type                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ----------- | ------ | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `instances` | 1      | repeated [`InstanceSpec`](#instancespec) |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `volumes`   | 2      | repeated [`HostVolume`](#hostvolume)     | (v1.1, agent memory) Persistent volumes kept on this host across instances. bcd creates each missing one (a sparse raw file under its data directory; growing is a truncate, seen by the guest at its next start; never shrunk) and destroys (hole-punches, then unlinks) one the section no longer lists once no instance attaches it: an absent volume is a deleted volume, so a spec is only ever missing one the control plane deleted. Only a bcd with the `instance.volumes` capability keeps them. |

### HostVolume [#hostvolume]

| Field        | Number | Type     | Description                                                                                                                                          |
| ------------ | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `volume_id`  | 1      | `string` | \[A-Za-z0-9\_-]\{1,80}.                                                                                                                              |
| `size_bytes` | 2      | `uint64` | A positive multiple of 512.                                                                                                                          |
| `serial`     | 3      | `string` | The virtio-blk serial (at most 20 bytes, \[a-z0-9-]); the guest finds the disk at /dev/disk/by-id/virtio-\<serial> and mounts it from its user data. |

### InstanceVolume [#instancevolume]

(v1.1) A HostVolume attached to an instance as an extra virtio disk, in list order after the
root disk. bcd refuses (InstanceStatus error) an instance naming a volume the section lacks.

| Field       | Number | Type     | Description |
| ----------- | ------ | -------- | ----------- |
| `volume_id` | 1      | `string` |             |
| `read_only` | 2      | `bool`   |             |

### InstanceSpec [#instancespec]

| Field                 | Number | Type                                              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| --------------------- | ------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `instance_id`         | 1      | `string`                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `desired_state`       | 2      | [`InstanceDesiredState`](#instancedesiredstate)   |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `runtime`             | 3      | `string`                                          | Runtime driver id, e.g. "libvirt-kvm", "cloud-hypervisor".                                                                                                                                                                                                                                                                                                                                                                                                    |
| `allocation_id`       | 4      | `string`                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `vcpu`                | 5      | `uint32`                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `memory_bytes`        | 6      | `uint64`                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `disk_bytes`          | 7      | `uint64`                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `image`               | 8      | [`ImageRef`](#imageref)                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `user_data_ref`       | 9      | `string`                                          | Secret ref of the NoCloud user-data, fetched with SecretRequest\{allocation\_id}.                                                                                                                                                                                                                                                                                                                                                                             |
| `gpu_device_ids`      | 10     | repeated `string`                                 | GPU atoms passed through with VFIO (never overcommitted).                                                                                                                                                                                                                                                                                                                                                                                                     |
| `nics`                | 11     | repeated [`InstanceNic`](#instancenic)            |                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `restore_from`        | 12     | [`ImageRef`](#imageref)                           | (v0.5, sandboxes) Boot from this snapshot (an instance.snapshot upload) instead of `image`: disk and, when captured, memory state. The daemon verifies sha256 before restoring and fails the instance on a runtime or CPU model mismatch.                                                                                                                                                                                                                     |
| `network_policy`      | 13     | [`InstanceNetworkPolicy`](#instancenetworkpolicy) | (v0.5, sandboxes) What the instance's NICs may reach, enforced by bcd as host nftables rules on its taps, whatever the runtime. bcd's own tunnel dial from the host always works.                                                                                                                                                                                                                                                                             |
| `identity_rebind_ref` | 14     | `string`                                          | (v0.5, review 13 H2) After a restore, the guest's per-instance identity (SSH host keys, machine id, workload credentials) must not stay the snapshot's. A secret ref (fetched with SecretRequest) of the new identity material; bcd delivers it to the running guest (a hotplugged raw disk read by a vps-installed guest unit) and reports the ref back in InstanceStatus.identity\_rebound\_ref once the guest acked it. Changing it asks for a new rebind. |
| `ssh_from_host_only`  | 15     | `bool`                                            | (review 13 L5) Port 22 of the guest is reachable only from the host (bcd's tunnel dial), never from other guests or the network, whatever network\_policy allows.                                                                                                                                                                                                                                                                                             |
| `placement`           | 16     | [`InstancePlacement`](#instanceplacement)         | (v1.1 §3.8) Where the guest's vCPUs and memory live on the host. Absent: the runtime's default (floating vCPUs, host memory).                                                                                                                                                                                                                                                                                                                                 |
| `volumes`             | 17     | repeated [`InstanceVolume`](#instancevolume)      | (v1.1, agent memory) Persistent volumes to attach (InstancesSection.volumes).                                                                                                                                                                                                                                                                                                                                                                                 |
| `aws_role`            | 18     | `string`                                          | (AWS A7) The IAM role ARN the guest's metadata endpoint serves AWS credentials for. bcd exchanges the instance's workload token (audience sts.amazonaws.com) with STS AssumeRoleWithWebIdentity (RoleSessionName bc-\<instance\_id>) and serves the result on the AWS-compatible /latest/meta-data/iam/security-credentials/ paths. Empty: no AWS credentials are served.                                                                                     |

### InstancePlacement [#instanceplacement]

| Field                 | Number | Type     | Description                                                                                                                                     |
| --------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpus`                | 1      | `string` | Host CPUs for the vCPUs (cpulist). With exactly `vcpu` CPUs, vCPU i is pinned to the i-th one; otherwise the vCPUs float within the set.        |
| `mems`                | 2      | `string` | NUMA nodes the guest memory is bound to (e.g. "1").                                                                                             |
| `hugepage_size_bytes` | 3      | `uint64` | Back guest memory with hugepages of this size (2097152 or 1073741824); 0 = normal pages. The pages come from the host's pools on `mems`.        |
| `balloon`             | 4      | `bool`   | Give the guest a virtio balloon (libvirt memballoon, cloud-hypervisor balloon) so the host can reclaim free guest memory; never with hugepages. |

### ImageRef [#imageref]

A content-addressed image the daemon may fetch and cache.

| Field         | Number | Type     | Description                                            |
| ------------- | ------ | -------- | ------------------------------------------------------ |
| `ref`         | 1      | `string` | Catalog reference, e.g. "ubuntu-24.04-cloud\@2026.09". |
| `url`         | 2      | `string` |                                                        |
| `sha256`      | 3      | `bytes`  |                                                        |
| `format`      | 4      | `string` | "raw" or "qcow2".                                      |
| `compression` | 5      | `string` | "" or "zstd".                                          |

### InstanceNic [#instancenic]

| Field         | Number | Type              | Description                                                                                                                                                                                                    |
| ------------- | ------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `network`     | 1      | `string`          | Bridge or network name on the host.                                                                                                                                                                            |
| `mac`         | 2      | `string`          |                                                                                                                                                                                                                |
| `addresses`   | 3      | repeated `string` | Addresses assigned by IPAM, CIDR form.                                                                                                                                                                         |
| `gateway`     | 4      | `string`          | (v1.1) The IPAM range's default gateway; empty: no default route through this NIC. bcd renders it, with `addresses`, into the guest's NoCloud network-config (the guest runs no DHCP client against the host). |
| `nameservers` | 5      | repeated `string` | (v1.1) DNS servers for the guest; empty: the guest keeps its image's resolver configuration.                                                                                                                   |

### AccessSection [#accesssection]

(v0.3) Interactive access material. Empty in v0.1.

| Field                   | Number | Type              | Description                                                                                                                                                                                |
| ----------------------- | ------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `exec_ticket_jwks_json` | 1      | `string`          | JWKS (JSON) the daemon verifies exec tickets against (§7.6).                                                                                                                               |
| `ssh_ca_public_keys`    | 2      | repeated `string` | OpenSSH-format public keys of the compute SSH CA trusted for break-glass certificates.                                                                                                     |
| `ssh_principals`        | 3      | repeated `string` | Principals such certificates may use.                                                                                                                                                      |
| `ssh_revoked_serials`   | 4      | repeated `uint64` | (v0.3) Serials of break-glass certificates revoked before they expired. The daemon writes them into an OpenSSH KRL (RevokedKeys), so a revoked certificate stops working at the next spec. |

### TelemetrySection [#telemetrysection]

Where bcd sends its own telemetry (OTLP). Never the session or Postgres (P8).

| Field                 | Number | Type              | Description                                                            |
| --------------------- | ------ | ----------------- | ---------------------------------------------------------------------- |
| `otlp_endpoint`       | 1      | `string`          | Empty disables export.                                                 |
| `export_interval_ms`  | 2      | `uint32`          |                                                                        |
| `resource_attributes` | 3      | repeated `string` | Extra "key=value" resource attributes.                                 |
| `headers_ref`         | 4      | `string`          | Secret ref of the exporter's auth headers, fetched with SecretRequest. |

### DaemonSection [#daemonsection]

(v0.4) Self-update target; the Rollout controller sets it within the disruption budget.

| Field       | Number | Type     | Description                                   |
| ----------- | ------ | -------- | --------------------------------------------- |
| `version`   | 1      | `string` |                                               |
| `url`       | 2      | `string` |                                               |
| `sha256`    | 3      | `bytes`  |                                               |
| `signature` | 4      | `bytes`  | Detached release signature over the artifact. |

### SpecRequest [#specrequest]

Ask for a full DesiredState.

| Field             | Number | Type                                      | Description |
| ----------------- | ------ | ----------------------------------------- | ----------- |
| `have_generation` | 1      | `uint64`                                  |             |
| `reason`          | 2      | [`SpecRequestReason`](#specrequestreason) |             |

### StatusReport [#statusreport]

Convergence state, coalesced latest-wins.

| Field                 | Number | Type                                               | Description                                                                                                                                                                                                                                                                                                                                                   |
| --------------------- | ------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `observed_generation` | 1      | `uint64`                                           | Every section converged or failed terminally at this generation. Never regresses on the control plane (GREATEST).                                                                                                                                                                                                                                             |
| `applied_generation`  | 2      | `uint64`                                           | The newest generation held (converging or converged).                                                                                                                                                                                                                                                                                                         |
| `applied_spec_hash`   | 3      | `bytes`                                            |                                                                                                                                                                                                                                                                                                                                                               |
| `sections`            | 4      | repeated [`SectionStatus`](#sectionstatus)         |                                                                                                                                                                                                                                                                                                                                                               |
| `conditions`          | 5      | repeated [`Condition`](#condition)                 | The node's full current condition set as seen by the daemon (health, drift, section errors).                                                                                                                                                                                                                                                                  |
| `instances`           | 6      | repeated [`InstanceStatus`](#instancestatus)       | (v0.3) What the host runtimes hold: every VM or container they know, managed or not, so the control plane can move instances by what actually runs and find leaks (an instance no live row places here). Latest-wins with the rest of the report.                                                                                                             |
| `instances_complete`  | 7      | `bool`                                             | (v0.3) `instances` is the runtimes' whole inventory: a placed instance missing from it is absent.                                                                                                                                                                                                                                                             |
| `pressure`            | 8      | [`HostPressure`](#hostpressure)                    | (v0.3) Host resource pressure for the Disk/Memory/CpuPressure conditions (§10.1). Absent when the kernel exposes no PSI.                                                                                                                                                                                                                                      |
| `trust_version`       | 9      | `uint64`                                           | TrustBundle.version the daemon holds, as in Hello.trust\_version, reported again once it applies a newer bundle mid-session (Welcome.trust), so the ack does not wait for a reconnect. 0: unchanged.                                                                                                                                                          |
| `approver_set_epoch`  | 10     | `uint64`                                           | The approver set held after that bundle, as in Hello (set whenever trust\_version is).                                                                                                                                                                                                                                                                        |
| `approver_set_sha256` | 11     | `bytes`                                            |                                                                                                                                                                                                                                                                                                                                                               |
| `discovered`          | 12     | repeated [`DiscoveredMachine`](#discoveredmachine) | (v1.1, site agents only) Machines seen PXE/HTTP-booting (DHCP PXEClient/HTTPClient) from a MAC with no live boot entry: never answered, never enrolled; a list for metal to match (by product\_uuid, the SMBIOS UUID a BMC also reports) or an operator to act on. The whole list, capped at 256 (oldest last\_seen evicted, forgotten 24 h after last seen). |
| `volume_storage`      | 13     | [`VolumeStorage`](#volumestorage)                  | (v1.1, hosts with a runtime) The volume store's filesystem, so capacity books persistent volumes against what can really be written (volumes are sparse). bcd refuses a volume create or growth beyond available − reserved − outstanding; capacity applies the same rule.                                                                                    |

### VolumeStorage [#volumestorage]

| Field               | Number | Type     | Description                                                                     |
| ------------------- | ------ | -------- | ------------------------------------------------------------------------------- |
| `total_bytes`       | 1      | `uint64` | The data directory's filesystem (statfs).                                       |
| `available_bytes`   | 2      | `uint64` | statfs Bavail.                                                                  |
| `reserved_bytes`    | 3      | `uint64` | Kept from volumes: overlays, journal, images (5% of total, at least 2 GiB).     |
| `committed_bytes`   | 4      | `uint64` | The sum of every volume's size.                                                 |
| `outstanding_bytes` | 5      | `uint64` | The sum of every volume's unwritten (sparse) space: promised but not yet taken. |

### DiscoveredMachine [#discoveredmachine]

| Field          | Number | Type                        | Description                                                                                   |
| -------------- | ------ | --------------------------- | --------------------------------------------------------------------------------------------- |
| `mac`          | 1      | `string`                    | Lowercase colon form.                                                                         |
| `product_uuid` | 2      | `string`                    | DHCP option 97 (the SMBIOS system UUID); empty when the client sent none.                     |
| `arch`         | 3      | `string`                    | DHCP option 93, as the site agent names it: x86\_64-efi, bios, arm64-efi, x86\_64-efi-http, … |
| `first_seen`   | 4      | `google.protobuf.Timestamp` |                                                                                               |
| `last_seen`    | 5      | `google.protobuf.Timestamp` |                                                                                               |
| `count`        | 6      | `uint32`                    |                                                                                               |
| `relay`        | 7      | `string`                    | giaddr when the request was relayed (which L2 segment it came from); empty when direct.       |

### HostPressure [#hostpressure]

Linux pressure stall information (/proc/pressure/\*, "some" avg10, percent of wall time) and CPU
steal (percent over the last report interval).

| Field               | Number | Type    | Description |
| ------------------- | ------ | ------- | ----------- |
| `cpu_some_avg10`    | 1      | `float` |             |
| `memory_some_avg10` | 2      | `float` |             |
| `io_some_avg10`     | 3      | `float` |             |
| `steal_pct`         | 4      | `float` |             |

### InstanceStatus [#instancestatus]

(v0.3) One instance as a host runtime reports it.

| Field                  | Number | Type                                              | Description                                                                                                                                                                                                                                                                                                                                                                     |
| ---------------------- | ------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `instance_id`          | 1      | `string`                                          | The InstanceSpec.instance\_id, or the runtime's own name for an object bcd did not create.                                                                                                                                                                                                                                                                                      |
| `state`                | 2      | [`InstanceObservedState`](#instanceobservedstate) |                                                                                                                                                                                                                                                                                                                                                                                 |
| `addresses`            | 3      | repeated `string`                                 | Addresses the guest holds, CIDR form (guest agent or DHCP leases).                                                                                                                                                                                                                                                                                                              |
| `message`              | 4      | `string`                                          |                                                                                                                                                                                                                                                                                                                                                                                 |
| `runtime`              | 5      | `string`                                          | Runtime driver id, e.g. "libvirt-kvm".                                                                                                                                                                                                                                                                                                                                          |
| `identity_rebound_ref` | 6      | `string`                                          | (review 13 H2) The InstanceSpec.identity\_rebind\_ref the guest acknowledged (its nonce written back); empty until then. Until it matches the spec's, the control plane refuses exec into a restored instance (it would still carry the snapshot's identity); readiness still means "runs", since guests restored from snapshots taken before the guest half existed never ack. |

### SectionStatus [#sectionstatus]

| Field        | Number | Type                            | Description                                                                                     |
| ------------ | ------ | ------------------------------- | ----------------------------------------------------------------------------------------------- |
| `section`    | 1      | `string`                        | NodeSpec section name: "host", "devices", "instances", "access", "telemetry", "daemon", "site". |
| `state`      | 2      | [`SectionState`](#sectionstate) |                                                                                                 |
| `generation` | 3      | `uint64`                        | The generation this status refers to.                                                           |
| `reason`     | 4      | `string`                        |                                                                                                 |
| `message`    | 5      | `string`                        |                                                                                                 |

### Condition [#condition]

A Kubernetes-style condition (PLAN.md §10.1).

| Field       | Number | Type                                  | Description                                                                                                                                                              |
| ----------- | ------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`      | 1      | `string`                              | e.g. "SectionUnsupported", "DriverMismatch", "Xid79" (v0.2).                                                                                                             |
| `status`    | 2      | [`ConditionStatus`](#conditionstatus) |                                                                                                                                                                          |
| `reason`    | 3      | `string`                              |                                                                                                                                                                          |
| `message`   | 4      | `string`                              |                                                                                                                                                                          |
| `device_id` | 5      | `string`                              | When set, the condition concerns one device (atom id).                                                                                                                   |
| `age_ms`    | 6      | `uint64`                              | How long ago the daemon saw the last transition, on its monotonic clock. The control plane derives `since` from compute\_now() (P9: agents report ages, not timestamps). |

### InventorySection [#inventorysection]

One inventory section, sent when Welcome.request\_sections asks for it, when inventory.refresh
runs, or when a re-collection changes its hash.

| Field            | Number | Type                        | Description                                                                                                                                              |
| ---------------- | ------ | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `section`        | 1      | `string`                    | Section name (INVENTORY\_SECTIONS): "system", "cpu", "memory", "disks", "nics", "gpus", "bmc", "os". It selects the message type of `content`.           |
| `sha256`         | 2      | `bytes`                     | SHA-256 of `content`. The control plane verifies it and compares it with the hash stored with the last accepted section; it never re-encodes to compare. |
| `content`        | 3      | `bytes`                     | Binary encoding of the canonicalized section message from inventory.proto (e.g. GpusSection): arrays sorted by their stable key, no volatile values.     |
| `collected_at`   | 4      | `google.protobuf.Timestamp` | Daemon clock, corrected by the Welcome offset; advisory.                                                                                                 |
| `schema_version` | 5      | `uint32`                    | NodeInventory schema version of `content` (1).                                                                                                           |

### EventBatch [#eventbatch]

Journaled events, sent in seq order. Results and transitions are never dropped; metric-like
events are dropped oldest-first when the journal is full.

| Field        | Number | Type                       | Description                                         |
| ------------ | ------ | -------------------------- | --------------------------------------------------- |
| `journal_id` | 1      | `string`                   | The journal the seqs belong to (Hello.journal\_id). |
| `events`     | 2      | repeated [`Event`](#event) |                                                     |

### Event [#event]

| Field               | Number | Type                                                   | Description                                                                                                                     |
| ------------------- | ------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| `seq`               | 1      | `uint64`                                               | Per journal\_id, strictly increasing from 1. Cumulatively acked (Ack.event\_seq); deduplicated on (node\_id, journal\_id, seq). |
| `observed_at`       | 2      | `google.protobuf.Timestamp`                            | Daemon clock, corrected by the Welcome offset; advisory.                                                                        |
| `attested`          | 10     | [`AttestedReport`](#attestedreport) (oneof `body`)     |                                                                                                                                 |
| `allocation_failed` | 11     | [`AllocationFailed`](#allocationfailed) (oneof `body`) |                                                                                                                                 |

### AttestedReport [#attestedreport]

A node-signed statement about allocations (§6 "Attested kinds").

| Field      | Number | Type                      | Description                                                                                                                                                       |
| ---------- | ------ | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `body`     | 1      | `bytes`                   | Serialized AttestedReportBody.                                                                                                                                    |
| `node_sig` | 2      | [`Signature`](#signature) | Node key. Purpose "attested-report", parts \[body]. Optional in v0.1; required from v0.2, when workers verify it against node\_certs before any state transition. |

### AttestedReportBody [#attestedreportbody]

| Field                 | Number | Type                                                         | Description                                            |
| --------------------- | ------ | ------------------------------------------------------------ | ------------------------------------------------------ |
| `node_id`             | 1      | `string`                                                     | The reporting node (must equal the session's node id). |
| `boot_id`             | 2      | `string`                                                     |                                                        |
| `allocation_bound`    | 10     | [`AllocationBound`](#allocationbound) (oneof `report`)       |                                                        |
| `allocation_released` | 11     | [`AllocationReleased`](#allocationreleased) (oneof `report`) |                                                        |

### AllocationBound [#allocationbound]

The assignment is materialized: CDI spec, env file and device ownership are in place.

| Field           | Number | Type              | Description                                                                                                                                                                                                                                         |
| --------------- | ------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allocation_id` | 1      | `string`          |                                                                                                                                                                                                                                                     |
| `generation`    | 2      | `uint64`          | The NodeSpec generation that carried the assignment.                                                                                                                                                                                                |
| `device_ids`    | 3      | repeated `string` |                                                                                                                                                                                                                                                     |
| `cdi_devices`   | 4      | repeated `string` | Fully qualified CDI device names, e.g. "bcd.dev/gpu=GPU-…".                                                                                                                                                                                         |
| `cdi_spec_path` | 5      | `string`          | e.g. "/etc/cdi/bcd-\<allocation\_id>.json".                                                                                                                                                                                                         |
| `env_file`      | 6      | `string`          |                                                                                                                                                                                                                                                     |
| `cgroup_parent` | 7      | `string`          | (v1.1 §3.8) The cgroup-v2 slice a HOLD's workload must run in, e.g. "bcd-alloc-\<allocation\_id>.slice" (also in the CDI spec and env file); empty when the assignment carried no limits or its binding is KUBERNETES (the kubelet enforces those). |

### AllocationReleased [#allocationreleased]

The assignment is gone: CDI spec and env file removed, ownership restored.

| Field           | Number | Type              | Description |
| --------------- | ------ | ----------------- | ----------- |
| `allocation_id` | 1      | `string`          |             |
| `generation`    | 2      | `uint64`          |             |
| `device_ids`    | 3      | repeated `string` |             |

### AllocationFailed [#allocationfailed]

The daemon refused or failed to materialize an assignment (e.g. overlap with a live one).

| Field           | Number | Type     | Description                                                                  |
| --------------- | ------ | -------- | ---------------------------------------------------------------------------- |
| `allocation_id` | 1      | `string` |                                                                              |
| `generation`    | 2      | `uint64` |                                                                              |
| `reason`        | 3      | `string` | Stable lowercase code, e.g. "overlap", "device-missing", "cdi-write-failed". |
| `message`       | 4      | `string` |                                                                              |

### Ack [#ack]

Cumulative acknowledgement of durably stored events and results.

| Field         | Number | Type              | Description                                                          |
| ------------- | ------ | ----------------- | -------------------------------------------------------------------- |
| `journal_id`  | 1      | `string`          |                                                                      |
| `event_seq`   | 2      | `uint64`          | Every event with seq \<= event\_seq in journal\_id is stored.        |
| `command_ids` | 3      | repeated `string` | Terminal CommandUpdates stored; the journal may stop replaying them. |

### SecretRequest [#secretrequest]

Fetch one secret for current work. Forwarded to the api secret broker, which checks node\_sig and
that the ref is bound to the agent's current step or allocation.

| Field           | Number | Type                      | Description                                                                                                 |
| --------------- | ------ | ------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `ref`           | 1      | `string`                  | "secret://…", "vault://…", "env://…".                                                                       |
| `workflow_id`   | 2      | `string`                  |                                                                                                             |
| `step_id`       | 3      | `string`                  |                                                                                                             |
| `allocation_id` | 4      | `string`                  |                                                                                                             |
| `epk`           | 5      | `bytes`                   | Fresh ephemeral P-256 public key, uncompressed SEC1 point (65 bytes). Used once.                            |
| `node_sig`      | 6      | [`Signature`](#signature) | Node key. Purpose "secret-request", parts \[request\_id, ref, workflow\_id, step\_id, allocation\_id, epk]. |
| `request_id`    | 7      | `string`                  | Correlates the SecretResponse.                                                                              |

### SecretResponse [#secretresponse]

| Field        | Number | Type     | Description                                                                                |
| ------------ | ------ | -------- | ------------------------------------------------------------------------------------------ |
| `ref`        | 1      | `string` |                                                                                            |
| `jwe`        | 2      | `string` | Compact JWE (ECDH-ES+A256KW / A256GCM) to SecretRequest.epk. Gateways only see ciphertext. |
| `request_id` | 3      | `string` |                                                                                            |
| `error`      | 4      | `string` | Set instead of jwe on refusal: "not-found", "not-bound", "access-denied", "unavailable".   |

### WorkloadTokenRequest [#workloadtokenrequest]

(v0.5) Workload identity (§12.2, §13): a VPS host's IMDS, bound to one instance, asks for that
instance's token. The control plane mints it only for a live instance placed on this node, in a
pool with workload identity on, for a configured audience.

| Field         | Number | Type                      | Description                                                                                                                                                      |
| ------------- | ------ | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `request_id`  | 1      | `string`                  | Correlates the WorkloadTokenResponse.                                                                                                                            |
| `instance_id` | 2      | `string`                  |                                                                                                                                                                  |
| `audience`    | 3      | `string`                  |                                                                                                                                                                  |
| `epk`         | 4      | `bytes`                   | Fresh ephemeral P-256 public key, uncompressed SEC1 point (65 bytes). Used once: the token is sealed to it, so a gateway relaying it never holds a usable token. |
| `node_sig`    | 5      | [`Signature`](#signature) | Node key. Purpose "workload-token", parts \[request\_id, instance\_id, audience, epk].                                                                           |

### WorkloadTokenResponse [#workloadtokenresponse]

| Field        | Number | Type                        | Description                                                                                                         |
| ------------ | ------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `request_id` | 1      | `string`                    |                                                                                                                     |
| `jwe`        | 2      | `string`                    | Compact JWE (ECDH-ES+A256KW / A256GCM) of the workload JWT to WorkloadTokenRequest.epk.                             |
| `expires_at` | 3      | `google.protobuf.Timestamp` |                                                                                                                     |
| `error`      | 4      | `string`                    | Set instead of jwe on refusal: "not-bound", "disabled", "audience", "rate-limited", "access-denied", "unavailable". |

### Drain [#drain]

Reconnect elsewhere (rolling deploys, rebalancing). The daemon closes the session and redials
after the delay, keeping its journal.

| Field                | Number | Type     | Description               |
| -------------------- | ------ | -------- | ------------------------- |
| `reconnect_after_ms` | 1      | `uint32` | rand(0, 60 s) by default. |
| `reason`             | 2      | `string` |                           |

### RevocationNotice [#revocationnotice]

This agent's certificate was revoked; the gateway closes the session right after sending it.
The daemon stops using the certificate and does not redial with it.

| Field     | Number | Type             | Description                                 |
| --------- | ------ | ---------------- | ------------------------------------------- |
| `reason`  | 1      | `string`         |                                             |
| `serials` | 2      | repeated `bytes` | Serial numbers of the revoked certificates. |

### OpenTunnel [#opentunnel]

Ask the daemon to open a Tunnel stream. The daemon re-verifies the ticket against the
AccessSection JWKS and its jti set before opening it.

| Field       | Number | Type     | Description                                                              |
| ----------- | ------ | -------- | ------------------------------------------------------------------------ |
| `tunnel_id` | 1      | `string` |                                                                          |
| `ticket`    | 2      | `string` | Exec ticket: ES256 JWT, 60 s, single-use jti, aud "node:\<id>", sid, op. |

### TunnelFrame [#tunnelframe]

| Field       | Number | Type                                           | Description                                      |
| ----------- | ------ | ---------------------------------------------- | ------------------------------------------------ |
| `tunnel_id` | 1      | `string`                                       |                                                  |
| `channel`   | 2      | `uint32`                                       | 0 = stdin/stdout or the byte stream, 1 = stderr. |
| `opened`    | 10     | [`TunnelOpened`](#tunnelopened) (oneof `body`) | The daemon's first frame on the stream.          |
| `data`      | 11     | `bytes` (oneof `body`)                         |                                                  |
| `resize`    | 12     | [`TerminalSize`](#terminalsize) (oneof `body`) |                                                  |
| `closed`    | 13     | [`TunnelClosed`](#tunnelclosed) (oneof `body`) |                                                  |

### TunnelOpened [#tunnelopened]

No fields.

### TerminalSize [#terminalsize]

| Field  | Number | Type     | Description |
| ------ | ------ | -------- | ----------- |
| `rows` | 1      | `uint32` |             |
| `cols` | 2      | `uint32` |             |

### TunnelClosed [#tunnelclosed]

| Field       | Number | Type     | Description |
| ----------- | ------ | -------- | ----------- |
| `exit_code` | 1      | `int32`  |             |
| `reason`    | 2      | `string` |             |

### Binding [#binding]

How an allocation is consumed (§10.5).

| Value                 | Number | Description                                                                                                                                                                                        |
| --------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BINDING_UNSPECIFIED` | 0      |                                                                                                                                                                                                    |
| `BINDING_HOLD`        | 1      | CDI hold, consumed by an operator-run workload manager on the node (v0.1).                                                                                                                         |
| `BINDING_INSTANCE`    | 2      | Passed to a VPS instance (v0.3).                                                                                                                                                                   |
| `BINDING_NODE`        | 3      | The whole node, for a tenant deploy (v0.5).                                                                                                                                                        |
| `BINDING_KUBERNETES`  | 4      | (v1.1 §3.8) Given to the node's kubelet (the node joined a Kubernetes cluster through its pool): bcd's device plugin advertises exactly the devices bound this way, and withdraws them on release. |

### InstanceNetworkPolicy [#instancenetworkpolicy]

| Value                                 | Number | Description                                                             |
| ------------------------------------- | ------ | ----------------------------------------------------------------------- |
| `INSTANCE_NETWORK_POLICY_UNSPECIFIED` | 0      | EGRESS: existing specs do not change.                                   |
| `INSTANCE_NETWORK_POLICY_EGRESS`      | 1      | Anything the pool's network allows (as before this field).              |
| `INSTANCE_NETWORK_POLICY_PRIVATE`     | 2      | Only the pool's IPAM subnet and the host.                               |
| `INSTANCE_NETWORK_POLICY_NONE`        | 3      | Only the host: bcd's tunnel dial, DHCP and ARP; nothing else in or out. |

### InstanceDesiredState [#instancedesiredstate]

| Value                                | Number | Description                                                                                                                                                               |
| ------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INSTANCE_DESIRED_STATE_UNSPECIFIED` | 0      |                                                                                                                                                                           |
| `INSTANCE_DESIRED_STATE_RUNNING`     | 1      |                                                                                                                                                                           |
| `INSTANCE_DESIRED_STATE_STOPPED`     | 2      |                                                                                                                                                                           |
| `INSTANCE_DESIRED_STATE_DELETED`     | 3      |                                                                                                                                                                           |
| `INSTANCE_DESIRED_STATE_PAUSED`      | 4      | (v0.5, sandboxes warm pools) Booted, then paused with its memory resident; resuming is desired RUNNING. A runtime that cannot pause keeps it STOPPED and reports STOPPED. |

### SpecRequestReason [#specrequestreason]

| Value                                     | Number | Description                                                                         |
| ----------------------------------------- | ------ | ----------------------------------------------------------------------------------- |
| `SPEC_REQUEST_REASON_UNSPECIFIED`         | 0      |                                                                                     |
| `SPEC_REQUEST_REASON_PATCH_BASE_MISMATCH` | 1      | A patch's base\_generation is not the applied generation.                           |
| `SPEC_REQUEST_REASON_MIN_GENERATION`      | 2      | A command's min\_generation is above the applied generation.                        |
| `SPEC_REQUEST_REASON_NO_SPEC`             | 3      | The journal holds no usable spec (new journal, corruption, failed signature check). |

### InstanceObservedState [#instanceobservedstate]

| Value                                 | Number | Description                                                    |
| ------------------------------------- | ------ | -------------------------------------------------------------- |
| `INSTANCE_OBSERVED_STATE_UNSPECIFIED` | 0      |                                                                |
| `INSTANCE_OBSERVED_STATE_RUNNING`     | 1      |                                                                |
| `INSTANCE_OBSERVED_STATE_STOPPED`     | 2      |                                                                |
| `INSTANCE_OBSERVED_STATE_ERROR`       | 3      |                                                                |
| `INSTANCE_OBSERVED_STATE_PAUSED`      | 4      | (v0.5) Paused by the runtime (vCPUs stopped, memory resident). |

### SectionState [#sectionstate]

| Value                       | Number | Description                                                            |
| --------------------------- | ------ | ---------------------------------------------------------------------- |
| `SECTION_STATE_UNSPECIFIED` | 0      |                                                                        |
| `SECTION_STATE_CONVERGED`   | 1      |                                                                        |
| `SECTION_STATE_PROGRESSING` | 2      |                                                                        |
| `SECTION_STATE_FAILED`      | 3      | Failed terminally at this generation (e.g. an overlapping assignment). |
| `SECTION_STATE_UNSUPPORTED` | 4      |                                                                        |

### ConditionStatus [#conditionstatus]

| Value                          | Number | Description |
| ------------------------------ | ------ | ----------- |
| `CONDITION_STATUS_UNSPECIFIED` | 0      |             |
| `CONDITION_STATUS_TRUE`        | 1      |             |
| `CONDITION_STATUS_FALSE`       | 2      |             |
| `CONDITION_STATUS_UNKNOWN`     | 3      |             |

## common.proto [#commonproto]

Package `bettercompute.common.v1`, `proto/bettercompute/common/v1/common.proto`.

Shared types for every better-compute wire contract (PLAN.md §7).

### Signature [#signature]

An ES256 signature (ECDSA P-256 with SHA-256).

| Field   | Number | Type             | Description                                                                                                                                                                                                                                          |
| ------- | ------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kid`   | 1      | `string`         | Identifies the signing key. For node keys: base64url (no padding) of SHA-256 over the key's SubjectPublicKeyInfo DER. For control-plane keys: the key's certificate subject key id as published in the trust bundle.                                 |
| `sig`   | 2      | `bytes`          | The raw 64-byte r ‖ s form (IEEE P1363 / JWS), not ASN.1 DER.                                                                                                                                                                                        |
| `chain` | 3      | repeated `bytes` | Optional DER X.509 certificates, leaf first, chaining the key to a trust root the verifier already holds (worker signing keys chain to a TrustBundle root). Empty when the verifier knows the key by `kid` (node keys are looked up in node\_certs). |

### TrustBundle [#trustbundle]

The command/spec signing trust the daemon accepts (PLAN.md §13). Delivered inside the signed
JoinResponseBody, and refreshed through Welcome as a SignedTrustBundle. Rotation: new roots are
added first; the old ones are dropped only after every daemon acknowledged the new version.

| Field                            | Number | Type                                   | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| -------------------------------- | ------ | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version`                        | 1      | `uint64`                               | Monotonic. A daemon ignores a bundle whose version is not greater than the one it holds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `command_signing_roots`          | 2      | repeated `bytes`                       | DER X.509 roots that certify the short-lived (24 h) worker keys signing Command and DesiredState bodies (v0.2). Empty in v0.1: signatures are optional there.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `approver_keys`                  | 3      | repeated [`ApproverKey`](#approverkey) | (v1.0, §13 quorum) Operator approval keys (P-256), held off the control plane (laptops, HSMs). With any set, a destructive command carrying CommandBody.batch\_id runs only with a BatchApproval signed by `approver_threshold` distinct keys of this set.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `approver_threshold`             | 4      | `uint32`                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `unbatched_destructive_per_hour` | 5      | `uint32`                               | With approver keys set: destructive commands WITHOUT a batch\_id this daemon runs per rolling hour (break-glass only; the default and recommended value is 0). A daemon cannot count fleet-wide, so any unbatched allowance lets a compromised worker touch every node once: fleet-wide limits come from the approvers (see quorum\_tiers), never from this cap.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `quorum_tiers`                   | 6      | repeated [`QuorumTier`](#quorumtier)   | Thresholds by batch size, ascending max\_scope\_size with non-decreasing thresholds: a batch with scope\_size ≤ max\_scope\_size needs `threshold` distinct approver signatures (the first tier that covers it); above every tier, approver\_threshold. Small batches can then be approved by a rate-limited auto-approver (one approver key held by a service outside api/worker, which is the fleet-wide counter), while large ones need k people.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `approver_set_signatures`        | 7      | repeated [`Signature`](#signature)     | (v1.0) The quorum fields (3-6) are signed by the approvers themselves, not only by the join key the control plane holds: k-of-the-CURRENT-set signatures, purpose "approver-set", parts \[uint64\_be(approver\_set\_epoch), sha256(approver set input), trust\_domain] (see protocol's approverSetInput; the verifier's own trust domain). Published on every version while the set is current. One rule (review 10 M3): a daemon holding set H at epoch e adopts (set N, epoch e') only when e' > e AND k-of-H (H's threshold) signatures over (e', N, domain) verify; otherwise it keeps (H, e). That holds for N == H too: adopting a higher epoch for an unchanged set needs the set's own signatures. The list therefore carries both the previous set's signatures (which admit the change) and the new set's own signatures (which move daemons already holding it, or holding it again after a revert, to the new epoch, so a replay of an intermediate epoch fails). When the quorum part is refused, the rest of the bundle (roots, version) still applies. A daemon without keys takes the first set as-is unless pinned (approver\_set\_sha256 in its config). Residual risk: a daemon offline across two changes can be served an intermediate epoch by a compromised api; pin long-offline nodes again when they return. |
| `approver_set_epoch`             | 8      | `uint64`                               | (v1.0) The epoch of the quorum fields (3-6): monotonic, separate from `version` (roots rotate without touching it), raised with every change of the set.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

### QuorumTier [#quorumtier]

| Field            | Number | Type     | Description |
| ---------------- | ------ | -------- | ----------- |
| `max_scope_size` | 1      | `uint32` |             |
| `threshold`      | 2      | `uint32` |             |

### ApproverKey [#approverkey]

One operator approval key.

| Field      | Number | Type     | Description                                                             |
| ---------- | ------ | -------- | ----------------------------------------------------------------------- |
| `kid`      | 1      | `string` | base64url (no padding) SHA-256 of spki\_der, as Signature.kid names it. |
| `spki_der` | 2      | `bytes`  | SubjectPublicKeyInfo DER of the P-256 key.                              |

### SignedTrustBundle [#signedtrustbundle]

A TrustBundle signed by the pinned JoinService signing key, so a gateway cannot inject roots.
Signing purpose "trust-bundle", parts \[body].

| Field       | Number | Type                      | Description             |
| ----------- | ------ | ------------------------- | ----------------------- |
| `body`      | 1      | `bytes`                   | Serialized TrustBundle. |
| `signature` | 2      | [`Signature`](#signature) |                         |

### DriverManifest [#drivermanifest]

What a daemon-side driver (built into bcd or served over bcx) offers (PLAN.md §5.4, §5.5).
Advertised in Hello.drivers and returned by bcx DriverPlugin.Describe; the control plane merges
it into its registry as a remote driver scoped to the advertising agents.

| Field                   | Number | Type                | Description                                                                          |
| ----------------------- | ------ | ------------------- | ------------------------------------------------------------------------------------ |
| `id`                    | 1      | `string`            | Driver id, e.g. "redfish", "ipmi", "libvirt-kvm", "nvidia".                          |
| `version`               | 2      | `string`            | Semantic version of the driver implementation.                                       |
| `kinds`                 | 3      | repeated `string`   | Driver kinds it implements, e.g. "power", "management", "runtime", "config".         |
| `runs_on`               | 4      | [`RunsOn`](#runson) |                                                                                      |
| `config_schema_json`    | 5      | `string`            | JSON Schema 2020-12 of the driver's config (manifest().jsonSchema() on the TS side). |
| `capabilities`          | 6      | repeated `string`   | Free-form capability flags, e.g. "virtual-media", "uefi-http".                       |
| `retry_safe_operations` | 7      | repeated `string`   | Operations that are safe to repeat after an interruption, as "\<kind>.\<operation>". |

### RunsOn [#runson]

Where a step or driver runs (PLAN.md §5.1 `RunsOn`).

| Value                   | Number | Description |
| ----------------------- | ------ | ----------- |
| `RUNS_ON_UNSPECIFIED`   | 0      |             |
| `RUNS_ON_CONTROL_PLANE` | 1      |             |
| `RUNS_ON_SITE`          | 2      |             |
| `RUNS_ON_NODE`          | 3      |             |
| `RUNS_ON_RAMDISK`       | 4      |             |

### AgentRole [#agentrole]

The role carried by an agent's certificate (role OID) and declared in Hello (PLAN.md §6). Not
the node's scheduling role (`nodes.role`: unassigned, host, dedicated, site).

| Value                    | Number | Description                                                                     |
| ------------------------ | ------ | ------------------------------------------------------------------------------- |
| `AGENT_ROLE_UNSPECIFIED` | 0      |                                                                                 |
| `AGENT_ROLE_NODE`        | 1      | `bcd node`: every server; 24 h certificate.                                     |
| `AGENT_ROLE_SITE`        | 2      | `bcd site`: DHCP proxy, iPXE, image cache, BMC access (v0.2); 24 h certificate. |
| `AGENT_ROLE_RAMDISK`     | 3      | `bcd ramdisk`: PXE commissioning (v0.2); restricted certificate, at most 2 h.   |

### Lane [#lane]

Command lanes (PLAN.md §6, §7.5). Each lane has its own daemon worker pool and credit budget.

| Value              | Number | Description                                                                                |
| ------------------ | ------ | ------------------------------------------------------------------------------------------ |
| `LANE_UNSPECIFIED` | 0      |                                                                                            |
| `LANE_CONTROL`     | 1      | Short, latency-sensitive work: diagnostics, inventory, reboots, resets. Default credits 8. |
| `LANE_PROVISION`   | 2      | Provisioning steps: wipe, deploy, kexec (v0.2). Default credits 4.                         |
| `LANE_BULK`        | 3      | Large transfers and slow jobs: log collection, image pre-warm. Default credits 2.          |

### CommandState [#commandstate]

The command lifecycle (PLAN.md §7.4). The control plane stores every state; a daemon reports only
ACKED, RUNNING, SUCCEEDED, FAILED, REJECTED and CANCELLED in CommandUpdate.

pending → dispatched(epoch) → acked → running → succeeded | failed | rejected
pending → expired (CommandSweeper, past not\_after); any non-terminal → cancelled

| Value                       | Number | Description                                                                                 |
| --------------------------- | ------ | ------------------------------------------------------------------------------------------- |
| `COMMAND_STATE_UNSPECIFIED` | 0      |                                                                                             |
| `COMMAND_STATE_PENDING`     | 1      | Stored, not yet sent. A BUSY rejection returns a command here with a backoff.               |
| `COMMAND_STATE_DISPATCHED`  | 2      | Sent on the session with the given epoch.                                                   |
| `COMMAND_STATE_ACKED`       | 3      | The daemon journaled it as `accepted` (fsynced before any side effect).                     |
| `COMMAND_STATE_RUNNING`     | 4      | The side effect has started.                                                                |
| `COMMAND_STATE_SUCCEEDED`   | 5      |                                                                                             |
| `COMMAND_STATE_FAILED`      | 6      | Terminal failure; CommandUpdate.error says why (e.g. "interrupted").                        |
| `COMMAND_STATE_REJECTED`    | 7      | Refused before any side effect; CommandUpdate.reject\_reason says why.                      |
| `COMMAND_STATE_EXPIRED`     | 8      | Never delivered before not\_after (control plane only).                                     |
| `COMMAND_STATE_CANCELLED`   | 9      | Stopped by Cancel between actions. A completed side effect is reported as its real outcome. |

### CommandRejectReason [#commandrejectreason]

Why a daemon refused a command before any side effect (PLAN.md §6 "Checks before accepting").

| Value                                    | Number | Description                                                                                                                                                                                  |
| ---------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COMMAND_REJECT_REASON_UNSPECIFIED`      | 0      |                                                                                                                                                                                              |
| `COMMAND_REJECT_REASON_UNSUPPORTED`      | 1      | Unknown kind, or a kind this agent role may not run.                                                                                                                                         |
| `COMMAND_REJECT_REASON_BAD_SIGNATURE`    | 2      | Missing, unknown-kid or invalid signature on a kind that requires one (destructive, v0.2).                                                                                                   |
| `COMMAND_REJECT_REASON_EXPIRED`          | 3      | Past not\_after (server time, via the Welcome offset).                                                                                                                                       |
| `COMMAND_REJECT_REASON_STALE_GENERATION` | 4      | min\_generation still not applied at not\_after.                                                                                                                                             |
| `COMMAND_REJECT_REASON_BUSY`             | 5      | No free credit on the command's lane. NOT terminal: the control plane puts the command back to pending with not\_before = compute\_now() + backoff (1 s → 30 s).                             |
| `COMMAND_REJECT_REASON_BATCH_UNAPPROVED` | 6      | (v1.0) A destructive command with a batch\_id but no valid BatchApproval covering it (too few distinct approver signatures, expired, or an inclusion proof that does not reach scope\_root). |
| `COMMAND_REJECT_REASON_UNBATCHED_LIMIT`  | 7      | (v1.0) Over TrustBundle.unbatched\_destructive\_per\_hour.                                                                                                                                   |

## driver.proto [#driverproto]

Package `bettercompute.driver.v1`, `proto/bettercompute/driver/v1/driver.proto`.

bcx: the out-of-process driver protocol (PLAN.md §5.5), served by plugins to both bcd and the control plane (externalDriver, remoteDriver).

### DriverPlugin [#driverplugin]

| RPC        | Request                                  | Response                                   | Description                                                                  |
| ---------- | ---------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------- |
| `Describe` | [`DescribeRequest`](#describerequest)    | [`DescribeResponse`](#describeresponse)    | The drivers this plugin serves.                                              |
| `Validate` | [`ValidateRequest`](#validaterequest)    | [`ValidateResponse`](#validateresponse)    | Validate a driver config against the driver's manifest and live checks.      |
| `Invoke`   | [`InvokeRequest`](#invokerequest)        | stream [`InvokeResponse`](#invokeresponse) | Run one operation; streams progress, then exactly one result or error.       |
| `Stream`   | stream [`StreamRequest`](#streamrequest) | stream [`StreamResponse`](#streamresponse) | Duplex byte stream for consoles and similar (first request frame is `open`). |
| `Health`   | [`HealthRequest`](#healthrequest)        | [`HealthResponse`](#healthresponse)        |                                                                              |

### DescribeRequest [#describerequest]

No fields.

### DescribeResponse [#describeresponse]

| Field              | Number | Type                                         | Description               |
| ------------------ | ------ | -------------------------------------------- | ------------------------- |
| `drivers`          | 1      | repeated [`DriverManifest`](#drivermanifest) |                           |
| `plugin_version`   | 2      | `string`                                     |                           |
| `protocol_version` | 3      | `uint32`                                     | bcx protocol version (1). |

### ValidateRequest [#validaterequest]

| Field         | Number | Type                                   | Description                    |
| ------------- | ------ | -------------------------------------- | ------------------------------ |
| `driver_id`   | 1      | `string`                               |                                |
| `kind`        | 2      | `string`                               |                                |
| `config_json` | 3      | `string`                               | Driver config, canonical JSON. |
| `secrets`     | 4      | repeated [`SecretValue`](#secretvalue) | Declared secret fields only.   |

### ValidateResponse [#validateresponse]

| Field    | Number | Type                                           | Description |
| -------- | ------ | ---------------------------------------------- | ----------- |
| `valid`  | 1      | `bool`                                         |             |
| `issues` | 2      | repeated [`ValidationIssue`](#validationissue) |             |

### ValidationIssue [#validationissue]

| Field     | Number | Type     | Description                   |
| --------- | ------ | -------- | ----------------------------- |
| `path`    | 1      | `string` | JSON pointer into the config. |
| `message` | 2      | `string` |                               |

### SecretValue [#secretvalue]

One secret field value, delivered only to the plugin that declared it.

| Field   | Number | Type     | Description                                         |
| ------- | ------ | -------- | --------------------------------------------------- |
| `field` | 1      | `string` | The config field name declared with field.secret(). |
| `value` | 2      | `bytes`  |                                                     |

### InvokeRequest [#invokerequest]

| Field             | Number | Type                                   | Description                          |
| ----------------- | ------ | -------------------------------------- | ------------------------------------ |
| `driver_id`       | 1      | `string`                               |                                      |
| `kind`            | 2      | `string`                               |                                      |
| `operation`       | 3      | `string`                               |                                      |
| `config_json`     | 4      | `string`                               | Driver config, canonical JSON.       |
| `args_json`       | 5      | `string`                               | Operation arguments, canonical JSON. |
| `secrets`         | 6      | repeated [`SecretValue`](#secretvalue) |                                      |
| `idempotency_key` | 7      | `string`                               |                                      |
| `deadline`        | 8      | `google.protobuf.Timestamp`            |                                      |
| `traceparent`     | 9      | `string`                               | W3C trace context.                   |

### InvokeResponse [#invokeresponse]

| Field         | Number | Type                                        | Description                                       |
| ------------- | ------ | ------------------------------------------- | ------------------------------------------------- |
| `progress`    | 1      | [`Progress`](#progress) (oneof `msg`)       |                                                   |
| `result_json` | 2      | `string` (oneof `msg`)                      | The operation's result, canonical JSON. Terminal. |
| `error`       | 3      | [`DriverError`](#drivererror) (oneof `msg`) |                                                   |

### Progress [#progress]

| Field     | Number | Type     | Description               |
| --------- | ------ | -------- | ------------------------- |
| `percent` | 1      | `uint32` | 0-100, or 0 when unknown. |
| `phase`   | 2      | `string` |                           |
| `message` | 3      | `string` |                           |

### DriverError [#drivererror]

| Field       | Number | Type     | Description                                                                            |
| ----------- | ------ | -------- | -------------------------------------------------------------------------------------- |
| `code`      | 1      | `string` | Stable lowercase code, e.g. "unsupported", "invalid-config", "timeout", "unreachable". |
| `message`   | 2      | `string` |                                                                                        |
| `retryable` | 3      | `bool`   | The host may retry the operation (only for retry-safe operations).                     |

### StreamRequest [#streamrequest]

| Field   | Number | Type                                            | Description                                                   |
| ------- | ------ | ----------------------------------------------- | ------------------------------------------------------------- |
| `open`  | 1      | [`InvokeRequest`](#invokerequest) (oneof `msg`) | First frame: what to open, with the same fields as an Invoke. |
| `data`  | 2      | `bytes` (oneof `msg`)                           |                                                               |
| `close` | 3      | [`StreamClose`](#streamclose) (oneof `msg`)     |                                                               |

### StreamResponse [#streamresponse]

| Field   | Number | Type                                        | Description |
| ------- | ------ | ------------------------------------------- | ----------- |
| `data`  | 1      | `bytes` (oneof `msg`)                       |             |
| `error` | 2      | [`DriverError`](#drivererror) (oneof `msg`) |             |
| `close` | 3      | [`StreamClose`](#streamclose) (oneof `msg`) |             |

### StreamClose [#streamclose]

| Field    | Number | Type     | Description |
| -------- | ------ | -------- | ----------- |
| `reason` | 1      | `string` |             |

### HealthRequest [#healthrequest]

No fields.

### HealthResponse [#healthresponse]

| Field     | Number | Type                            | Description |
| --------- | ------ | ------------------------------- | ----------- |
| `status`  | 1      | [`HealthStatus`](#healthstatus) |             |
| `message` | 2      | `string`                        |             |

### HealthStatus [#healthstatus]

| Value                       | Number | Description |
| --------------------------- | ------ | ----------- |
| `HEALTH_STATUS_UNSPECIFIED` | 0      |             |
| `HEALTH_STATUS_SERVING`     | 1      |             |
| `HEALTH_STATUS_NOT_SERVING` | 2      |             |

## site.proto [#siteproto]

Package `bettercompute.site.v1`, `proto/bettercompute/site/v1/site.proto`.

Site agent contracts (`bcd site`, PLAN.md §3.5, §8.5): boot lease state, proxy-DHCP entries, iPXE scripts, BMC operations and discovery assertions.

### SiteSpec [#sitespec]

The `site` section of a site agent's NodeSpec.

| Field           | Number | Type                                   | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| --------------- | ------ | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `boot`          | 1      | [`BootLeaseState`](#bootleasestate)    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `cache_images`  | 2      | repeated [`CacheImage`](#cacheimage)   | Images the site cache pre-warms (§11.5), from its pools' policy. The agent pulls each one and re-verifies its digest before serving it.                                                                                                                                                                                                                                                                                                                                                    |
| `proxy_targets` | 3      | repeated [`ProxyTarget`](#proxytarget) | The machines site.httpProxy may reach through this agent, each with its addresses: the source of "an address of target\_node\_id known to the site" (HttpProxyRequest). The agent refuses a request whose URL host is not one of target\_node\_id's addresses (IP literals compared as IPs, names compared by name and then by every resolved IP), so a request "about" one node can never reach another machine or a BMC. Contributed for nodes whose profile is agent-less (e.g. Talos). |

### ProxyTarget [#proxytarget]

| Field       | Number | Type              | Description                                                                  |
| ----------- | ------ | ----------------- | ---------------------------------------------------------------------------- |
| `node_id`   | 1      | `string`          |                                                                              |
| `addresses` | 2      | repeated `string` | IP literals or DNS names of the node's API endpoint(s), never a BMC address. |

### CacheImage [#cacheimage]

A verified catalog image the site cache holds.

| Field    | Number | Type     | Description                 |
| -------- | ------ | -------- | --------------------------- |
| `url`    | 1      | `string` |                             |
| `sha256` | 2      | `bytes`  | SHA-256 of the image bytes. |
| `format` | 3      | `string` | "qcow2", "raw", "iso", …    |

### BootLeaseState [#bootleasestate]

The site:\<id>:boot lease as published by the SiteBoot controller. Only the active holder answers
proxyDHCP, serves iPXE scripts and vouches for discovery. An agent that loses its session
self-fences after ttl\_ms: it stops all three.

| Field    | Number | Type     | Description                                                                                          |
| -------- | ------ | -------- | ---------------------------------------------------------------------------------------------------- |
| `active` | 1      | `bool`   |                                                                                                      |
| `epoch`  | 2      | `uint64` | Lease epoch; every assertion the agent signs carries it, and the control plane rejects older epochs. |
| `ttl_ms` | 3      | `uint32` |                                                                                                      |

### ProxyDhcpEntry [#proxydhcpentry]

One machine the active boot server answers for (boot.prepareRamdisk / boot.cleanUp).

| Field                 | Number | Type                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------------- | ------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mac`                 | 1      | `string`                    | PXE MAC, lowercase colon form.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `product_uuid`        | 2      | `string`                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `arch`                | 3      | [`BootArch`](#bootarch)     |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `boot_file`           | 4      | `string`                    | First-stage boot file served over TFTP or HTTP (e.g. an iPXE build).                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `ipxe_script_url`     | 5      | `string`                    | HTTPS URL of this machine's iPXE script, served by the site agent.                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `workflow_id`         | 6      | `string`                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `not_after`           | 7      | `google.protobuf.Timestamp` | Server time after which the entry is dropped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `discovery_token_ref` | 8      | `string`                    | Secret ref of this machine's MAC-bound discovery join token. The site agent fetches it with SecretRequest\{workflow\_id} and puts it only into the ramdisk's boot parameters it serves, so the token never travels in plaintext in commands.body.                                                                                                                                                                                                                                                                                 |
| `sanboot_sha256`      | 9      | `bytes`                     | (v1.1) Boot this ISO instead of the ramdisk, for BMC drivers without virtual media (ipmi): the sha256 of a SiteSpec.cache\_images entry with format "iso". The machine's iPXE script is then `sanboot --no-describe <site agent>/cache/<hex>`, the vendor or OS ISO as is (no discovery token, no bc.\* parameters). A digest the cache does not hold verified is never answered (the site section fails) rather than falling back to the ramdisk. `boot_file` still chains iPXE; `ipxe_script_url` is ignored while this is set. |

### HttpProxyRequest [#httpproxyrequest]

site.httpProxy: one HTTPS request made by the site agent on the control plane's behalf, for
agent-less profiles (e.g. Talos, PLAN.md §5.6 walkthrough 3) whose API is reachable only inside the
site. P5 holds: the control plane never dials into a site. The agent enforces, whatever the command
says: the host resolves to an address of `target_node_id` known to the site (its inventory or BMC
network), https only unless the address is on the provisioning VLAN, no redirects, a response cap
(default 1 MiB) and the timeout. Anything else fails with "forbidden-target".

| Field             | Number | Type                                           | Description                                                                                  |
| ----------------- | ------ | ---------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `target_node_id`  | 1      | `string`                                       | The node the request is about; the target host must be one of its addresses.                 |
| `url`             | 2      | `string`                                       |                                                                                              |
| `method`          | 3      | `string`                                       | "GET", "POST", "PUT", "DELETE".                                                              |
| `headers`         | 4      | repeated [`HttpProxyHeader`](#httpproxyheader) |                                                                                              |
| `body`            | 5      | `bytes`                                        |                                                                                              |
| `tls_spki_sha256` | 6      | `bytes`                                        | Pinned SubjectPublicKeyInfo SHA-256 of the target's TLS certificate (Talos uses its own CA). |
| `timeout_ms`      | 7      | `uint32`                                       |                                                                                              |
| `secret_headers`  | 8      | repeated [`HttpProxyHeader`](#httpproxyheader) | Headers whose values are secret refs, fetched with SecretRequest and never logged.           |

### HttpProxyHeader [#httpproxyheader]

| Field   | Number | Type     | Description |
| ------- | ------ | -------- | ----------- |
| `name`  | 1      | `string` |             |
| `value` | 2      | `string` |             |

### HttpProxyResponse [#httpproxyresponse]

| Field       | Number | Type                                           | Description                            |
| ----------- | ------ | ---------------------------------------------- | -------------------------------------- |
| `status`    | 1      | `uint32`                                       |                                        |
| `headers`   | 2      | repeated [`HttpProxyHeader`](#httpproxyheader) |                                        |
| `body`      | 3      | `bytes`                                        |                                        |
| `truncated` | 4      | `bool`                                         | The body exceeded the cap and was cut. |

### BootCleanUpRequest [#bootcleanuprequest]

boot.cleanUp: drops the machine's proxy-DHCP entries and iPXE script (workflow cleanup).

| Field         | Number | Type              | Description                                                          |
| ------------- | ------ | ----------------- | -------------------------------------------------------------------- |
| `node_id`     | 1      | `string`          |                                                                      |
| `macs`        | 2      | repeated `string` | PXE MACs whose entries to drop; empty drops every entry of the node. |
| `workflow_id` | 3      | `string`          |                                                                      |

### IpxeScriptRequest [#ipxescriptrequest]

The parameters iPXE sends when chainloading ($\{mac}, $\{uuid}, $\{buildarch}, $\{platform}, …).
The site agent matches them against its entries; an unknown machine becomes a discovery event.

| Field       | Number | Type     | Description                                     |
| ----------- | ------ | -------- | ----------------------------------------------- |
| `mac`       | 1      | `string` |                                                 |
| `uuid`      | 2      | `string` |                                                 |
| `ip`        | 3      | `string` |                                                 |
| `buildarch` | 4      | `string` | iPXE $\{buildarch}: "x86\_64", "arm64", "i386". |
| `platform`  | 5      | `string` | iPXE $\{platform}: "efi", "pcbios".             |
| `serial`    | 6      | `string` |                                                 |
| `asset`     | 7      | `string` |                                                 |

### IpxeScript [#ipxescript]

| Field       | Number | Type                        | Description                                                                               |
| ----------- | ------ | --------------------------- | ----------------------------------------------------------------------------------------- |
| `script`    | 1      | `string`                    | The rendered script, starting with "#!ipxe". Carries the single-use discovery credential. |
| `not_after` | 2      | `google.protobuf.Timestamp` |                                                                                           |

### BmcOperationRequest [#bmcoperationrequest]

A BMC operation run by a site agent through bmclib (steps with runsOn: site).

| Field                 | Number | Type     | Description                                                                                                                                                                                                                                                                   |
| --------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `driver`              | 1      | `string` | "redfish" or "ipmi" (or a vendor driver).                                                                                                                                                                                                                                     |
| `address`             | 2      | `string` | Host or URL of the BMC.                                                                                                                                                                                                                                                       |
| `credentials_ref`     | 3      | `string` | Secret ref of the BMC credentials, fetched with SecretRequest; never a value.                                                                                                                                                                                                 |
| `operation`           | 4      | `string` | "\<kind>.\<operation>", e.g. "power.setPower", "management.setBootDevice", "virtual-media.insert", "inspect.inspect".                                                                                                                                                         |
| `args_json`           | 5      | `string` | Operation arguments, canonical JSON.                                                                                                                                                                                                                                          |
| `tls_spki_sha256`     | 6      | `bytes`  | SHA-256 of the BMC certificate's SubjectPublicKeyInfo, when pinned.                                                                                                                                                                                                           |
| `driver_options_json` | 7      | `string` | Per-BMC driver options set at enrollment, canonical JSON (e.g. bcx-ipmi \{"cipherSuite": 3} for old BMCs). The site agent hands them to the driver as they are; the driver validates them and refuses unknown keys. Never credentials: those travel only as credentials\_ref. |

### BmcOperationResult [#bmcoperationresult]

| Field         | Number | Type     | Description                                              |
| ------------- | ------ | -------- | -------------------------------------------------------- |
| `ok`          | 1      | `bool`   |                                                          |
| `result_json` | 2      | `string` | Operation result, canonical JSON (e.g. \{"power":"on"}). |
| `error_code`  | 3      | `string` |                                                          |
| `message`     | 4      | `string` |                                                          |

### SignedDiscoveryAssertion [#signeddiscoveryassertion]

A site agent's statement that a PXE-booted machine asked for a boot from this MAC/IP. It vouches
for the ramdisk's `discovery` join (v0.2). Signing purpose "discovery-assertion", parts \[body],
by the site agent's node key.

| Field  | Number | Type                      | Description                    |
| ------ | ------ | ------------------------- | ------------------------------ |
| `body` | 1      | `bytes`                   | Serialized DiscoveryAssertion. |
| `sig`  | 2      | [`Signature`](#signature) |                                |

### DiscoveryAssertion [#discoveryassertion]

| Field                | Number | Type                        | Description                               |
| -------------------- | ------ | --------------------------- | ----------------------------------------- |
| `site_id`            | 1      | `string`                    |                                           |
| `site_agent_node_id` | 2      | `string`                    | The vouching site agent.                  |
| `boot_lease_epoch`   | 3      | `uint64`                    | BootLeaseState.epoch held while vouching. |
| `mac`                | 4      | `string`                    |                                           |
| `ip`                 | 5      | `string`                    |                                           |
| `product_uuid`       | 6      | `string`                    |                                           |
| `workflow_id`        | 7      | `string`                    |                                           |
| `issued_at`          | 8      | `google.protobuf.Timestamp` |                                           |
| `not_after`          | 9      | `google.protobuf.Timestamp` |                                           |

### BootArch [#bootarch]

| Value                       | Number | Description |
| --------------------------- | ------ | ----------- |
| `BOOT_ARCH_UNSPECIFIED`     | 0      |             |
| `BOOT_ARCH_BIOS`            | 1      |             |
| `BOOT_ARCH_X86_64_EFI`      | 2      |             |
| `BOOT_ARCH_ARM64_EFI`       | 3      |             |
| `BOOT_ARCH_X86_64_EFI_HTTP` | 4      |             |


# Roadmap (/docs/reference/roadmap)

> The version plan from P0 to v1.0 with each phase's deliverables and exit gates, and the honest state of every item in the repository today.



Better Compute is built in phases, each defined in PLAN.md §16 by its deliverables, its acceptance criteria and its
exit tests. A phase closes only when its exit tests are green in CI; the benchmark gates are measured on the
reference hardware in `bench/REFERENCE.md` (a gateway host with 8 vCPU and 32 GiB, PostgreSQL 17 on 16 vCPU and
64 GiB NVMe).

The phases were planned in order, but the work did not stay in order: the TypeScript control plane for every phase
up to v0.5 exists in the repository, and the Go daemon covers the v0.1 node agent, while the daemon's later modes and
the pieces that need a PostgreSQL server or real hardware do not exist yet. So no phase after P0 is formally closed
yet. This page tracks each item as:

* **Done**: in the code, with tests, and listed in the changelog.
* **In progress**: partly there. The note says which part is missing.
* **Not started**: nothing in the repository yet.

> **Three things hold every phase open.** 
  - **The Go daemon is not released, and only its node agent runs end to end.** `bcd join`, `bcd node` and
    `bcd status` work and are tested end to end against a real gateway (`tests/e2e/bcd.test.ts`, the fake-hardware
    build in WSL), including a node console tunnel, and the node agent also converges break-glass SSH (`access`).
    `bcd site` and `bcd ramdisk` are written and covered by Go unit tests only. The `instances` converger, instance
    tunnels and the IMDS run end to end only on fakehw builds (`tests/e2e/bcd-vps.test.ts`); the `libvirt-kvm` and
    `cloud-hypervisor` runtimes of host builds (362317c, 2598f0a) are tested against fake tools only. The release
    pipeline exists but no release has been cut. GPUs are read through `nvidia-smi` (cb987e5) with no go-nvml backend.
    Self-update, the WebSocket fallback and `bcd helper` (self-update included) are done and exercised through the
    TypeScript `FakeDaemon`. `bcd sim` runs many real node agents in one process (fakehw builds only).
    One real host has run a host build of bcd (round 6, 2026-10-04): a WSL 2 workstation with a GeForce RTX 5090.
    `tests/e2e/bcd-host-real.test.ts` (`BCD_E2E_REAL=1`) joins it to a dev stack, adopts it, reads its real inventory
    and GPU, exports the GPU's live metrics over OTLP and answers `bcd status` and `bcd doctor`. That bcd ran as an
    ordinary user without `[helper]`, so it binds no GPUs: no real `/etc/cdi` spec, hold slice, passthrough or MIG has
    run yet. See [running bcd on real hardware](/docs/daemon/real-hardware) and
    [what bcd does today](/docs/daemon#what-bcd-does-today).
  - **No PostgreSQL server has run the suites.** Local runs use PGlite (PostgreSQL 18 in WebAssembly). Suites that need
    real concurrency skip unless `BETTER_COMPUTE_POSTGRES_URL` is set, and every benchmark number below is from PGlite
    on a development machine, which the gates do not count.
  - **The hardware gates (PLAN.md D15) have not run.** They need one NVIDIA box, a self-hosted KVM runner and the
    reference benchmark machines, and rented A100 or H100 capacity before the v0.5 exit.

## At a glance [#at-a-glance]

| Phase                                  | Theme                      | State                                                                                                                                                                                                                                                                                                        |
| -------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [P0](#p0-scaffold)                     | Scaffold                   | Done                                                                                                                                                                                                                                                                                                         |
| [v0.1](#v01-adopt-and-reserve)         | Adopt and reserve          | Control plane done; bcd's node agent done and tested end to end, not released (one real GPU host has joined, been adopted and reported its GPU; a real GPU hold, a data-center GPU server and the install-script run are open; GPUs come from `nvidia-smi`, cb987e5); PostgreSQL and the reference gate runs |
| [v0.2](#v02-metal)                     | Bare metal                 | Control plane done and tested in simulation; `bcd site` and `bcd ramdisk` written (Go unit tests only, not released), never run against the control plane or on hardware                                                                                                                                     |
| [v0.3](#v03-vps-and-access)            | VPS and access             | Control plane done; tunnels and the `access` section done in bcd (not released); `libvirt-kvm` and `cloud-hypervisor` written in bcd and tested against fake tools only                                                                                                                                      |
| [v0.4](#v04-scale-and-ha)              | Scale and HA               | Mostly done in simulation; `bcd helper` done, self-update through it included; the numbers need the reference PostgreSQL                                                                                                                                                                                     |
| [v0.5](#v05-gpu-depth-and-ecosystem)   | GPU depth and ecosystem    | Control plane done ahead of schedule; sandboxes served in full on the control plane; hardware exits not run                                                                                                                                                                                                  |
| [v1.0](#v10-freeze)                    | Freeze                     | Started: the API surface freeze check, bcd's SLSA3 release pipeline, the N-2 skew matrix and the upgrade-in-place scenario exist (FakeDaemons), and the destructive-batch quorum is built end to end, its producers included; the security review and npm provenance do not                                  |
| [v1.1](#cpu-memory-and-kubernetes-v11) | CPU, memory and Kubernetes | Mostly in the code (host configuration, host resources, slices, MIG and slot holds, Kubernetes workers, bcd as a DaemonSet), tested against fakes only                                                                                                                                                       |

Every package is on npm at version 0.2.0 on the `latest` dist-tag (0.1.0 was the first release, on 2026-10-07; 0.1.1
followed the same day) and
accepts Better IAM `>=0.2.0 <0.4.0` from npm. Provenance attestations come once CI publishes from the GitHub repository. bcd has no released
binary yet.

## P0: Scaffold [#p0-scaffold]

P0 was committed with `pnpm check` green, and its contracts are frozen: `proto/bettercompute/agent/v1`, the
`@better-compute/core` contracts, the v0.1 schema and the v0.1 rows of PLAN.md §9.2 now change only by additive,
reviewed edits.

| Deliverable                                                                                                                                                                                        | State | Notes                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| git and `git init`                                                                                                                                                                                 | Done  | With GitHub Desktop's bundled git; git is not on the machine's `PATH`                                                                                                                                                                                                                                                                                                                                |
| A pnpm workspace mirroring Better IAM's                                                                                                                                                            | Done  |                                                                                                                                                                                                                                                                                                                                                                                                      |
| Scripts `check-deps`, `check-sql-now`, `umbrella`, `sync-better-iam`, `proto-gen`, `authz-matrix`                                                                                                  | Done  | Under `scripts/`; `pnpm lint:repo` runs the repository rules                                                                                                                                                                                                                                                                                                                                         |
| Better IAM vendored                                                                                                                                                                                | Done  | 0.1.0 as tarballs in `vendor/better-iam`, pinned through `overrides`, until Better IAM was on npm; the releases depend on Better IAM from npm (`>=0.2.0 <0.4.0` since 0.1.1)                                                                                                                                                                                                                         |
| Packages `core`, `protocol`, `postgres` (with `./pglite` and `compute_now()`), shells of `server`, `client`, `cli`, `testing`, stubs of `inventory`, `provision`, `metal`, `vps`, and the umbrella | Done  | The stubs have since been implemented                                                                                                                                                                                                                                                                                                                                                                |
| `protocol` with TypeScript and Go generated code committed                                                                                                                                         | Done  | TypeScript by `scripts/proto-gen.mjs`; Go in `daemon/gen/` by `scripts/daemon-proto-gen.mjs` (protoc-gen-go v1.36.12, protoc-gen-connect-go v1.21.0), with `--check`                                                                                                                                                                                                                                 |
| `daemon/` skeleton and goreleaser configuration                                                                                                                                                    | Done  | The module builds and its tests pass (`node scripts/daemon-test.mjs`; cross-compiled and run in WSL on Windows). goreleaser is configured and checked in CI, but no release has been cut                                                                                                                                                                                                             |
| CI: Ubuntu and Windows on Node 22 and 24, a PostgreSQL job, a Go job; the IAM contract test; `bench/REFERENCE.md`                                                                                  | Done  | `.github/workflows/ci.yml` and `daemon.yml` are defined, and since then `nightly.yml` (the simulation matrix with the 10k cold start, the IAM-guarded suite in both IAM modes, the whole suite on Postgres, a green-night streak) and `release.yml` (bcd with SBOMs, cosign and SLSA3 provenance); see [CI](/docs/operations/testing#ci-and-the-repository-checks). This page does not track CI runs |

| Exit                                                                             | State                                          |
| -------------------------------------------------------------------------------- | ---------------------------------------------- |
| `pnpm check` green, including `umbrella --check`                                 | Done locally                                   |
| `buf breaking` against `proto/baseline.binpb`                                    | Done: the baseline is committed and CI runs it |
| Packed smoke importing `better-compute/{inventory,provision,metal,vps}`          | Done: `test:packed`                            |
| Freeze of `agent/v1`, the core contracts, the v0.1 schema and the v0.1 §9.2 rows | Done                                           |

Of the day-one verifications, PGlite's `btree_gist` is verified. The go-nvml mock (go-nvml is not a dependency yet),
the KVM runner and CDI on target hosts are not, for want of hardware.

## v0.1: Adopt and reserve [#v01-adopt-and-reserve]

v0.1 adopts servers you already run (a brownfield join with a token) and reserves their GPUs. It was split into five
workstreams.

| Workstream                                                                                                                                                                                                                                       | State       | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **W1, Go daemon**: join, session, journal, Renew with proof of possession, NVML inventory, `devices` and `access` convergence, CDI holds, `diag.nvml`, `node.reboot`, release artifacts, install script                                          | In progress | Done: `bcd join` (sealed token, verified `JoinResponse`), `bcd node` (session over pinned mTLS, bbolt journal, Renew with proof of possession at 60% of the lifetime, the `devices` converger with CDI holds and node-signed `AllocationBound`, the `access` converger for break-glass SSH (CA, node-bound principals, a native KRL), `inventory.refresh`, attested `diag.nvml`, `logs.collect`, `gpu.reset`, self-terminating `node.reboot`, signed-intent checks, the server time mark, the trust version in `Hello` and `StatusReport`), and beyond W1: tunnels with node-verified exec tickets (instance tickets never reach the host), `SecretRequest` and `WorkloadTokenRequest`, XIDs from the kernel log (kept across a bcd restart in the same boot) and host pressure, and the v0.2 `bcd site` and `bcd ramdisk` and the v0.3 `instances` converger (see below). Also `bcd status` over the unix socket, `bcd doctor`'s host checks, procfs/sysfs inventory, and the install script (`better-compute daemon install-script`); `tests/e2e/bcd.test.ts` runs it against a real gateway. The GPU inventory is read through `nvidia-smi` without cgo (cb987e5: MIG, NVLink, topology, ECC and row remapping, tested against captured output; no go-nvml backend and no NVML events). Open: release artifacts (the release workflow, with SBOMs, cosign signatures and SLSA provenance, exists, but no tag has been released), and a run on a real host |
| **W2, gateway and PKI**: mTLS sessions, the serving CA, epoch fencing, liveness batching, dispatch by join, resume rules, `BUSY` re-queue, local CAs, the `join-token` attestor, JoinService, the downloads route                                | Done        | `@better-compute/gateway`, `@better-compute/pki`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| **W3, store, capacity and inventory**: schema, `compute_now()`, work queue, outbox, leases, CommandSweeper, `InlinePlacer`, atoms, rollups, `gpu-models.json`, conformance and torture tests                                                     | Done        | `@better-compute/postgres`, `capacity`, `inventory`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| **W4, IAM bridge**: the catalog, plugin and inbox, projection and resolver, `embedded()`, IAM jobs, flag cache, usage sink, IamReconcile, contract test, authz matrix                                                                            | Done        | `@better-compute/iam`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| **W5, server, client, CLI, simulator**: `operation()`, the method registry, the v0.1 groups, the typed client, the CLI (`dev`, `iam setup`, `daemon install-script`, `pki issue-gateway`), the deterministic simulator and the real-time harness | Done        | Since then the CLI has commands for every API group (PLAN.md §9.3; 44915e5, e2fe6f8, bb89083, d1e83ca, 573b0b0, 171d9be), and a test ties every route they call to the method registry (fb6304f). See [command groups](/docs/guides/clients/cli#command-groups)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

| Acceptance                                                                                                                      | State            | Notes                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A real `bcd` installed by the install script, plus 3 FakeDaemons, join and match the golden inventory                           | In progress      | The FakeDaemon half is done, and Go's `fakehw` matches the TypeScript section hashes. A real `bcd` (fakehw, in WSL) joins and is adopted in `tests/e2e/bcd.test.ts`, but not installed by the script (there is no release binary) and not beside FakeDaemons                                                                                                                 |
| An org user reserves 4 GPUs from an offered pool; the CDI spec lists exactly the booked UUIDs, which `reservations.get` returns | In progress      | `tests/e2e/spine.test.ts` runs it with embedded Better IAM, the example policies, a real gateway and three FakeDaemons: the node's CDI spec, held devices and env file list exactly the booked UUIDs. `tests/e2e/bcd.test.ts` binds a 2-GPU reservation on the real bcd and checks its CDI spec file under the fake root. A CDI spec on a real GPU host has not been checked |
| `ReserveBoundedH100` allows 8 GPUs and denies 9; `BoundedExtend` denies 72 h + 1 h                                              | Done             | IAM bridge and contract tests                                                                                                                                                                                                                                                                                                                                                |
| `gpu.h100.hours` reaches Better IAM billing without duplicates across worker restarts, even with a deleted identity in a batch  | Done             | Idempotency keys on every usage window; the sink re-attributes deleted identities                                                                                                                                                                                                                                                                                            |
| Place-to-active p99 under 2 s with 1k FakeDaemons in the real-time harness                                                      | Done, indicative | 1.02 s at 5 placements/s on PGlite; the inline placer saturates one pool between 5 and 10/s                                                                                                                                                                                                                                                                                  |
| No upstream Better IAM change                                                                                                   | Done             | The supported topology of PLAN.md §12.1                                                                                                                                                                                                                                                                                                                                      |

| Exit test                                                                                                                                                                                                                                                            | State                                                                                                                                                                                                                                                       |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The spine end to end (`tests/e2e/spine.test.ts`)                                                                                                                                                                                                                     | Done with FakeDaemons: join with sealed tokens, `diag.nvml`, adopt, reserve 4 GPUs as a policy-bound org user, bind, release; 9 GPUs, 73 h, non-pool-users and other organizations denied. `tests/server-spine-v01.test.ts` covers the same path in-process |
| `tests/capacity-concurrency.test.ts`: 1,000 concurrent operations never double-book, and the torture test                                                                                                                                                            | Done on PGlite (interleaved); the truly concurrent PostgreSQL half skips without a server                                                                                                                                                                   |
| Faults: gateway kill mid-command runs nothing twice, journal loss fails instead of replaying, a reboot completes on rejoin, invariants I1, I2, I5, I8                                                                                                                | Done in `tests/sim/fleet-gateway.test.ts` and `fleet-stub.test.ts`                                                                                                                                                                                          |
| Security negatives: an IamPort spy records zero IAM calls on fleet paths over 10 simulated minutes                                                                                                                                                                   | Done (`tests/sim/iam-negative.test.ts`)                                                                                                                                                                                                                     |
| Security negatives: authz matrix at 100%, resolver timeout denies, a failed `authorizeMany` fails the list, join tokens honour `maxUses` and TTL, node A cannot get node B's spec, Renew without proof of possession is rejected, cross-tenant requests leak nothing | Done                                                                                                                                                                                                                                                        |

The v0.1 gates are where the phase is really held open:

| Gate                                                                                                                                  | State                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Gateway**: one Node gateway holds 5,000 mTLS sessions at 10 s heartbeats for 30 minutes, dispatch p99 under 500 ms, RSS under 2 GiB | Sessions, dispatch and RSS criteria met on a workstation; the reference run remains. On a development workstation (PGlite, 60 s, the gateway and its store in their own process): 5,000 mTLS sessions, one TLS 1.3 connection each, dispatch p99 26 ms, gateway process RSS 1,240 MiB, 0 drops, and a reconnect storm with 5,000 of 5,000 back in 39 s and every command settled once. A 30-minute run of the same setup on PGlite met every criterion (dispatch p99 24 ms over 90,000 commands, 0 drops), RSS only just: it grew to 2,016 MiB with the JS heap flat at 235 MiB, which a 5-minute attributed run traced to PGlite's WebAssembly memory in the gateway process. The 30-minute run with PostgreSQL 17 on the reference host, which is to confirm the RSS criterion, has not happened, so D20 (a Go gateway) stays formally undecided. See [scaling](/docs/operations/scaling#gateway-sessions) |
| **IAM lane**: embedded Better IAM on adapter-postgres sustains 50 admissions/s with lock wait under 50%, and the ceiling is recorded  | Not started: the suite exists (`tests/bench/iam-lane.bench.ts`) but needs PostgreSQL, so D18 (proposing R1 upstream) stays open                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| Placements/s recorded                                                                                                                 | Done on PGlite: 35.7/s for the inline placer at 1k nodes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

## v0.2: Metal [#v02-metal]

v0.2 takes servers from a BMC and network boot to a deployed OS and back, with verified wipes.

| Deliverable                                                                    | State              | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `bcd site` and `bcd ramdisk`                                                   | In progress        | Written, with Go unit tests. `bcd site`: Redfish BMC operations, BIOS settings, RAID layouts and firmware updates from the verified image cache included (944221c), each only under its own `bmc.*` kind, other BMC drivers as hosted bcx plugins with `bcx-ipmi` for IPMI power, boot device and inspection (51806ab, 7818f19; tested against a fake `ipmitool`; no vendor driver), `site.httpProxy`, boot entries, and the boot server (proxy DHCP, TFTP, iPXE scripts with signed discovery assertions under the boot lease, the image cache). `bcd ramdisk`: the `discovery` join from its kernel command line, `disk.wipe`, `deploy.image`, `boot.kexec`, `diag.dcgm`. Open: a run against the control plane or on hardware, a release, and a ramdisk image and iPXE builds. The FakeDaemon plays both roles in the tests |
| `provision` and `metal()`                                                      | Done               | The workflow engine, profiles, the `metal` group and reconciler                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `siteTokens` and the SiteBoot lease; `joinRequests`                            | Done               | A standby site agent is promoted within 30 s of the active one's loss                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `discovery`, `provisioned` and `x509pop` attestors                             | Done               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Signed destructive commands and the trust bundle; node-signed attested results | Done               | Every command is worker-signed when `commandSigning` is configured; a gateway-forged wipe result is refused                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `SecretRequest` sealed secrets, the `secrets` API and BMC rotation             | Done               | Two-phase BMC password rotation. bcd sends them for BMC credentials, `site.httpProxy` secret headers, discovery tokens and ramdisk user data, and signs `bmc.rotate` results                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| The image catalog and site caches; FleetConfig plan and apply                  | Done               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| XID and health classification, and `diag.dcgm` acceptance                      | In progress        | Classification, quarantines and remediation are done, and bcd reports XIDs from the kernel log and runs `gpu.reset` and `diag.dcgm` (`dcgmi diag`, in the ramdisk and on nodes); no DCGM run on real GPUs yet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| The revocation-push SLA                                                        | Done               | A revoked node's session closes within 5 s (about 1 s in tests)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Emulation CI                                                                   | In progress        | `examples/emulated-site` describes metal on QEMU/KVM with sushy-tools and `bcd site` on a Linux host with KVM; nobody has run it (it also needs iPXE builds and a ramdisk image; BMC addresses may name each VM's system on one emulator since 4e4c1d7), and there is no KVM runner (D15)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Walkthrough 1, Ubuntu autoinstall                                              | Done in simulation | `tests/extensions/ubuntu-autoinstall.test.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

| Acceptance and exit                                                                                                          | State                                                                                                                                                                                        |
| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 20 unattended enroll, ready, deployed, released, ready loops on UEFI and BIOS VMs                                            | Done in simulation; not run on VMs                                                                                                                                                           |
| Killing the site agent or the worker mid-deploy resumes cleanly (`metal-chaos`)                                              | Done (`tests/metal-chaos.test.ts`)                                                                                                                                                           |
| Killing the active site agent mid-PXE: the standby takes over boot within 30 s                                               | Done in simulation (`tests/metal-site-failover.test.ts`): the standby is promoted when the dead agent's lease runs out, the machine boots from it, and the boot step's BMC commands ran once |
| A gateway-forged `AllocationBound` is rejected                                                                               | Done (`tests/e2e/attested-bindings.test.ts`, through a real gateway with signing FakeDaemons); see [attested bindings](/docs/guides/capacity/reservations#attested-bindings)                 |
| Every `ready` follows a verified wipe; `metal-lifecycle`                                                                     | Done                                                                                                                                                                                         |
| Site agents refuse other sites' steps                                                                                        | Done                                                                                                                                                                                         |
| A `SecretResponse` cannot be opened with another key or after its TTL                                                        | Done                                                                                                                                                                                         |
| Unsigned images are rejected; a ramdisk certificate cannot open a `node` session; unsigned destructive commands are rejected | Done                                                                                                                                                                                         |

## v0.3: VPS and access [#v03-vps-and-access]

| Deliverable                                                                      | State         | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `vps()`: instances, flavors, provider accounts, overcommit, IPAM                 | Done          | The control plane and the instance reconciler                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Host runtimes `libvirt-kvm` and `cloud-hypervisor`                               | In progress   | They run inside bcd on each host, under its `instances` converger. `libvirt-kvm` is written (362317c): a host build with `virsh` runs instances through `virsh` and `qemu-img` (qcow2 overlays on sha256-verified base images, NoCloud seeds, VFIO GPUs, host-passthrough CPU, snapshots and restores), but it has been tested only against a fake `virsh` and `qemu-img`, never on a real KVM host. `cloud-hypervisor` is written (2598f0a): a host build with the `cloud-hypervisor` binary runs one VMM per instance as a confined `systemd-run` unit over its REST API, with the same overlays and seeds, VFIO GPUs, memory snapshots and restores; it has been tested only against a fake VMM API and fake tools, never on a real KVM host. Security review 12's two high-severity findings are fixed (933c51b): taps are persistent and bridged before the VMM starts, with an `inet bcd_taps` backstop for a tap off its bridge, and both runtimes refuse a tenant image or restored snapshot that names a backing file or an external data file; the VMM's unit sees only its own instance and base image and is bounded by its flavor. fakehw builds run a file-backed runtime, and the FakeDaemon's `FakeHypervisor` stands in for the control-plane tests; see [delivery](/docs/daemon/delivery#the-runtimes) |
| `provider-hetzner` (Cloud, and Robot with `bootstrap`)                           | Done          | Outside the umbrella package                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Single-tenant containers                                                         | In progress   | Pools declare `policy.runtime` and must set `policy.singleTenant` for container runtimes, and vps refuses a container on a pool that serves other tenants; the host runtime needs bcd                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Tunnels with daemon-verified tickets, `nodes.issueSshCert`, the `access` section | Done          | Including node-bound break-glass certificates and an SSH KRL for revocation. bcd (not released) re-verifies tickets against the pinned JWKS and its journaled `jti` set, runs node consoles on a real pseudo-terminal, dials instance `ssh` and `tcp` tunnels to the instance's own address, answers an instance `console` ticket with the instance's serial console (from its runtime: the pty `libvirt-kvm` gave the serial port, or `cloud-hypervisor`'s serial socket; f8273ef), which `tunnels.open` issues as op `console` with `instanceId`, decided as `compute:instances:exec` (798e710; `better-compute console --instance`, the console's `/instances/<id>/console` page, 191898b), advertises `tunnel.instance-dial` when it runs a runtime, and converges `access` into sshd; it refuses `exec` into a guest, which needs a guest agent. The FakeDaemon answers tunnels with an echo shell and has no `access` converger                                                                                                                                                                                                                                                                                                                                                                                    |
| Webhooks and SSE                                                                 | Done          | `@better-compute/events`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `mcp`                                                                            | Done          | `@better-compute/mcp`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Delegated confirmation                                                           | Done          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| The `cloud-iid` attestor                                                         | Done          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Walkthrough 2 (a VPS provider plugin) and walkthrough 3 (Talos)                  | Done in tests | `tests/extensions/acme-cloud.test.ts` (against an emulator, live and from a recording), `tests/extensions/talos.test.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

| Acceptance and exit                                                                 | State                                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 200-instance churn on 3 emulated hosts with no leaks                                | Done (`tests/vps-churn.test.ts`) against emulated hosts in the test process                                                                                                                                                                                                                                                                                                                     |
| Host loss recreates instances per policy                                            | Done                                                                                                                                                                                                                                                                                                                                                                                            |
| An agent reserves over MCP with a person's confirmation, retrying with the same key | Done (`tests/mcp-e2e-confirmation.test.ts`)                                                                                                                                                                                                                                                                                                                                                     |
| A replayed exec ticket is rejected by the gateway and the daemon                    | Done: end to end with the FakeDaemon as the daemon, and in bcd's own unit and interop tests, which also check that the `jti` is journaled                                                                                                                                                                                                                                                       |
| `delegation:revoke` closes the agent's stream within 5 s                            | Done (`tests/e2e/access.test.ts`)                                                                                                                                                                                                                                                                                                                                                               |
| Multi-tenant container offers are rejected; no tenant VM runs bcd                   | Done                                                                                                                                                                                                                                                                                                                                                                                            |
| `tests/e2e/vps.test.ts`                                                             | Done with FakeDaemon hosts on the dev stack (real server runtime and gateway, fake hypervisors): cloud-config through the secret broker, 200-instance churn with no leaks, host loss and recreation, container offers refused, and a tenant VM cannot join as a node with anything it holds. Real KVM hosts need bcd's `libvirt-kvm` or `cloud-hypervisor`, neither of which has run on one yet |

## v0.4: Scale and HA [#v04-scale-and-ha]

| Deliverable                                                                      | State              | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| -------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `PoolLeasePlacer`                                                                | Done               | Opt-in: `capacityModule({ placer: 'pool-lease' })` (D22)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Shard rebalancing                                                                | Done               | A silent worker's shards are taken at lease expiry                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Several gateways                                                                 | Done in simulation | Three real gateways behind the fleet simulator, with gateway kill and zombie fencing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| WebSocket fallback, signed frames only                                           | Done               | Gateway and FakeDaemon sides, and bcd (6cd48be): `[websocket] url`, `proxy_ca_file` and `audience`, the gateway's proof checked against the pinned serving CA, renewal without a client certificate over `POST /v1/renew` on the WebSocket listener (06cb334), no tunnels, command-signing roots required; `daemon install-script --websocket-url` writes the section (0ea4ab4). `tests/e2e/bcd.test.ts` runs bcd over the real WebSocket gateway, renewal included (4b2d5aa). See [the fallback](/docs/daemon/protocol#websocket-fallback-v04)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| A/B self-update and Rollout                                                      | Done, not released | Signed releases and wave rollouts, and bcd's node half (de9f380): signed manifests under `[update] release_pins`, a release floor, A/B slots, `upgrade.json` with a 120 s deadline, confirmation on the first `Welcome`, a swap back without one, a crash-loop rollback unit, and `daemon.upgrade`; `tests/e2e/bcd.test.ts` updates a real bcd through a real exec (fc0ca6a). A confirmed release is never undone: bcd refuses every older binary, the other slot's included (c139b9c), `rollouts.abort` with `rollback` returns only unconfirmed upgrades (8347a3d), and a bad release is fixed by rolling forward to one with a higher sequence. No release has been cut. See [self-update](/docs/daemon/install#self-update)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `bcd helper`                                                                     | Done               | Privilege separation (28dc2ae): with `[helper] socket` and `user`, `bcd node` runs unprivileged and `bcd helper`, as root, serves it typed operations over a unix socket checked with `SO_PEERCRED` and the caller's executable: the files bcd owns by kind and id (CDI specs, allocation env files, the sshd drop-in, CA, KRL and principals), `firewall.apply` and `firewall.check` on `bridge bcd_instances` and `inet bcd_taps`, `gpu.reset` of the host's own GPUs, `reboot` and `ssh.reload`, each logged. `tests/e2e/bcd.test.ts` runs an unprivileged node beside a real helper through a GPU hold's CDI spec. `daemon install-script --unprivileged` sets it up (169c459): the `bcd` account, `bcd-helper.service`, a `User=bcd` drop-in, the socket in `/run/bcd-helper`. Self-update through the helper (2d67b25): the `update.*` operations, root's `[update] slots_dir`, the helper re-verifying the manifest and hashing the staged binary as it copies it, its own deadline watchdog; the e2e delivers a signed build to an unprivileged node. Security review 11 (c139b9c): no downgrade through the other slot, the staged file opened beneath its directory without symlinks or FIFOs, the node's claimed version never trusted, at most 32 requests at once, and root-only directories on the way to the slots and socket; its follow-ups (9e71b1f) keep the entrypoint out of `data_dir`, adopt only a root-owned binary, and answer `busy` over the cap. The instance runtimes go through it too: `libvirt-kvm`'s checked `libvirt.*` operations (ed3c7d2) and, since 7fe74a1, `cloud-hypervisor`'s `ch.*` (taps, VFIO groups and VMM units root builds from the instance record it checks; tested against fakes only). Not helper operations: PLAN.md's `runtime.*` and `mig.apply`; bcx plugins need a root node. See [privilege separation](/docs/daemon#privilege-separation) |
| Signing of all commands and specs                                                | Done               | With `pki.commandSigning`, every command and every NodeSpec generation is signed by a worker key; the gateway relays stored signed specs verbatim, and the FakeDaemon verifies them (`intent.signedSpecs`). bcd verifies them too once its trust bundle holds command-signing roots: every spec and every command, whatever its kind (0d42de8; the FakeDaemon since 12ccfa1)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Remote IAM (`remoteIam`, `computeIamHost`, `replicaProjection`, `postgresInbox`) | Done               | U0 is an upstream Better IAM item outside this repository. With `remoteIam({ refresh })`, an admission waits (at most 2 s) until the IAM host's replica has replayed compute's commit and re-reads the row, failing closed with `UNAVAILABLE`; `doctor` warns `iam.refresh` when it is missing                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Remote-IAM MCP                                                                   | Done               | `remoteMcpIam()` gives `@better-iam/mcp`'s gate an IAM reached over HTTP; see [MCP](/docs/guides/clients/mcp#remote-iam)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Remote-IAM event intake over webhooks                                            | Done               | `host.iam.webhook`: Better IAM's signed subtree webhooks into `iam_inbox`, for deployments that cannot read the IAM database. See [IAM events](/docs/guides/iam#iam-events-compute-reacts-to)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Scheduler worker role                                                            | Done               | `Reconciler.role` in core, the runtime's scheduler role with its own shard membership, and the pool-lease placer `capacity.placer` running in it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Partitioned history                                                              | Done               | `audit_events`, `inventory_snapshots` and `usage_windows` by UTC month (`core/0018_partitioned_history`); `retentionModule()` keeps months created ahead, prunes and drops emptied months; audit months are archived by hand. See [retention](/docs/operations/retention#history-partitions)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Merkle audit checkpoints                                                         | Done               | `@better-compute/audit`, with `better-compute audit verify`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `bcd sim`                                                                        | In progress        | Written (aa7501c): `bcd sim --nodes N --seed S` runs N real node agents on fakehw in one process against a gateway, each with its own identity, journal and session; `tests/e2e/bcd.test.ts` joins five with one five-use token. Open: the nightly `bcd sim --nodes 10000` against real gateways and PostgreSQL through Toxiproxy. `better-compute sim` remains the FakeDaemon fleet simulator; see [bcd sim](/docs/daemon#bcd-sim)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| R1, if the IAM-lane benchmark demands it                                         | Not started        | Waits for the v0.1 IAM-lane gate                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

The v0.4 acceptance is set at 10,000 nodes and 100,000 GPUs on the reference PostgreSQL. What exists is simulation:

| Criterion                                                           | State                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Gateway-kill recovery under 60 s                                    | Met in simulation: at 10k nodes, the 3,334 re-homed nodes are back at a p99 of 13.25 s simulated time, with no invariant violated                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 0 false unreachable marks                                           | Met in simulation (`tests/sim/v04-scenarios.test.ts`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| At least 200 placements/s                                           | Indicative: with the pool-lease placer on PGlite, 1,092/s end to end at 1k nodes; at 10k nodes 464/s in the latest recorded run (`bench/placements/results`), and about 1,058/s in the measurement taken when the pool index started loading devices per node on demand (not yet a recorded result); see [scaling](/docs/operations/scaling#placements). The benchmark's sustained mode (`BENCH_PLACE_SUSTAIN_SECONDS=1800` for the acceptance, gate `BENCH_PLACE_GATE`, default 200, judged on the slowest 10 s window, I1 checked) measured 699/s, slowest window 639/s, over 30 s at 1k nodes on PGlite |
| Cold start under 3 minutes                                          | Met in simulation: `runColdStart` in `@better-compute/testing` runs the scenario on the real control plane and 3 real gateways in simulated time. The recorded 10k run (`bench/cold-start`) had every node connected, adopted and schedulable at 80 s and back 40 s after a power cycle, with 0 unreachable marks and 0 invariant violations; see [scaling](/docs/operations/scaling#cold-start). The CI-sized run (24 nodes, `tests/sim/cold-start.test.ts`) checks the same in every build. The wall-clock run needs the reference PostgreSQL                                                            |
| Command p99 under 1.5 s with polling only, PostgreSQL CPU under 50% | Not measured: needs the reference PostgreSQL                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Suites green in both IAM modes                                      | Done: `pnpm check` runs `pnpm test:iam-remote`, which reruns the IAM-dependent suites against IAM over HTTP, the authz matrix's IAM side included (every §9.2 action and resource pair decided through `bridge.remote`)                                                                                                                                                                                                                                                                                                                                                                                    |
| Every §15 v0.4 scenario green on 3 consecutive nights               | In progress: the scenarios run in the simulator, and the `nightly` workflow runs `sim:nightly`, the IAM-guarded suite and the Postgres suite every night and counts consecutive green nights; this page records no nightly history                                                                                                                                                                                                                                                                                                                                                                         |

## v0.5: GPU depth and ecosystem [#v05-gpu-depth-and-ecosystem]

Most of v0.5 landed early, on the control plane. Sandboxes joined the phase on 2026-09-24 (D24): all 15 methods are
served, with templates, TTL expiry, warm pools, snapshots and exec, and the exit test runs on emulated hosts. bcd
enforces the template `network`, and a host build runs sandboxes on `libvirt-kvm` (362317c) or `cloud-hypervisor`
(2598f0a), both tested against fake tools only: no sandbox has booted on a real host yet.

| Deliverable                                                                                 | State                                              | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MIG, time-slice and MPS atoms                                                               | Done                                               | See [fractional GPUs](/docs/guides/capacity/fractional-gpus); applying a MIG layout on a GPU is bcd's                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Calendar reservations and BookingRepair                                                     | Done                                               | See [gang and calendar](/docs/guides/capacity/gang-and-calendar)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Gang scheduling                                                                             | Done                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Spot                                                                                        | Done                                               | See [spot and preemption](/docs/guides/capacity/spot-and-preemption)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `dedicated-root` metal and `binding: 'node'`                                                | Done                                               | See [dedicated root](/docs/guides/bare-metal/dedicated-root)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| TPM attestor                                                                                | Done in code, software TPM only                    | The verifier and a software TPM producer are done. `bcd join --attestor tpm` joins with the host's TPM (`/dev/tpmrm0`) and no token (8545688), and `daemon install-script --attestor tpm` writes a script for it (a006d9f). A fixture contract checks bcd's evidence and answer against `tpmAttestor` in both directions, and `tests/e2e/bcd-tpm.test.ts` joins a fakehw bcd with its software TPM through the real JoinService, reclaiming the same node after a reinstall (db9bd46); since 8a9af54 with an EK certificate from a test manufacturer root the attestor verifies (no `allowUncertifiedEk`) and the node key in that TPM. bcd keeps its node key in the host's TPM when it can open one (965c14d, 50e18f9, 0ba2a79). Pools can require it with `policy.joinAttestors` (1516728). Since cf24876 a `tpm` join also proves the node key: bcd makes it in the attestor's TPM, the AK certifies it over `sha256(csr)`, and `tpmAttestor` refuses a join without that proof by default (`nodeKey: 'certified'`), recording `nodeKey: 'tpm'` in `node.joined` and the join's audit row. Since d94f1f3 the proof holds through renewal: the certificate names the AK (`urn:bc:tpm-ak:<nameAlg>:<digest>` beside the SPIFFE ID), and every renewal must carry that AK's certification of the next key (`RenewRequest.key_attestation`), refused otherwise with no downgrade; bcd makes the next key in the TPM or fails the renewal loudly, and the fakehw e2e renews and keeps the statement. Open: a join, a renewal and a TPM node key on a hardware TPM with a real manufacturer's EK certificate. See [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm) |
| Optional workload identity                                                                  | Done on the control plane                          | Workload OIDC tokens and the IMDS token exchange in pki, `WorkloadTokenRequest` over the session, the workload broker, and the issuer's discovery document and JWKS in `serve`. bcd sends the request from its own metadata service (`[imds] listen`), which serves the instances of any runtime it runs; end to end it has run only with fakehw builds' file-backed runtime. See [workload identity](/docs/guides/vps/instances#workload-identity)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| bcx and `driver-sdk`                                                                        | In progress                                        | The TypeScript SDK runs drivers out of process with mutual TLS. bcd hosts sha256-pinned plugins from `[plugins] dir` and runs `driver.invoke` in them (`daemon/internal/bcx`, with `bcx.Serve` as the start of a Go plugin SDK), tested by Go unit tests only. Plugins run sandboxed on Linux (d132fef, 3872c6b): executed from the file that was hashed, each under its own uid (root only when named in `[plugins] privileged`), with `no_new_privs`, a seccomp filter on every thread and optional cgroup limits, and restarted when they exit (9006eee: back-off from 1 s to 1 min, re-hashed and sandboxed again). The first Go plugin in the repository is `bcx-ipmi`, the IPMI BMC driver a site agent hosts (51806ab), shipped as its own release archive (348ff56). Open: an end-to-end run of bcd with a plugin, health checks, and the control plane merging advertised drivers into its registry. See [drivers](/docs/extending/drivers#driver-plugins-in-bcd)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `provider-proxmox`                                                                          | Done                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| console-kit and `apps/console`                                                              | Done                                               | Including an instance's serial console page (191898b) beside the exec terminal                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Walkthrough 4, a Kata runtime in Go                                                         | In progress                                        | bcd serves runtimes from bcx plugins: a hosted driver of kind `runtime` becomes an instances runtime (`list`, `apply`, `remove` over `Invoke`; no snapshots), covered by Go unit tests (494de69). No Kata plugin is written. See [walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Sandboxes (PLAN.md §3.7): templates, TTL, warm pools, snapshots, exec over SSH certificates | Done on the control plane: 15 of 15 methods served | Added to v0.5 by D24. Served: `sandboxTemplates` create, list, get, update and delete (with `env` and `network`), `sandboxes` create (with `fromSnapshot`), list, get, extend, snapshot, exec and delete, and `sandboxSnapshots` list, get and delete, with migrations `vps/0004` to `vps/0012`, the `vps.sandboxes` reconciler (TTL expiry, `sandbox.*` events, warm pools, snapshot settling and purging), the `vps.sandbox.warm` meter, the instance guard and vps's tenant teardown. The CLI's `sandbox` and `sandbox-template` commands, eight MCP tools and the console's Sandboxes page use them. bcd enforces `network` (6db49c8) and runs sandboxes on `libvirt-kvm` (362317c) and `cloud-hypervisor` (2598f0a). Open: a run of either on a real host (both are tested against fake tools only). See [sandboxes](#sandboxes)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

| Exit                                                                                                                                                                                                                                                                                          | State                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| The MIG torture test and the real-hardware nightly                                                                                                                                                                                                                                            | The torture test runs on a simulated MIG pool; nothing has run on hardware (D15)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| A `dedicated-root` lease and release yield a new identity generation                                                                                                                                                                                                                          | Done                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| A third-party TypeScript driver runs on a node via bcx                                                                                                                                                                                                                                        | Done with a FakeDaemon as the node (`tests/e2e/bcx.test.ts`): a module profile's `power.*` steps go out as `driver.invoke` commands through the real engine, queue and gateway, and run in a pinned bcx plugin process. bcd now hosts bcx plugins and runs `driver.invoke`, but no test runs a TypeScript plugin under bcd yet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| IMDS cross-instance access fails                                                                                                                                                                                                                                                              | Done with FakeDaemon hosts (`tests/e2e/imds.test.ts`, `tests/pki-workload.test.ts`): one instance's session token fails for another and on another host, a host gets nothing for an instance placed elsewhere, forwarded requests and replays are refused                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `tests/e2e/sandboxes.test.ts`: 500 sandboxes expire on emulated hosts with the v0.3 zero-leak diff; a warm claim reaches `ready` without a placement; a snapshot round trip restores memory state; another tenant's snapshot is refused; instance methods on a sandbox's instance are refused | Done with FakeDaemon hosts (a8f3dfc): `tests/e2e/sandboxes.test.ts` runs the real server and gateway with three FakeDaemon VPS hosts (fake `cloud-hypervisor`, which pauses). It proves that a warm claim reaches `ready` without a placement (the paused instance resumes on the same node and allocation, booted once) and the pool refills to `warm.min`; that 500 sandboxes (`E2E_SANDBOXES`) with a 60 s TTL expire on their own, each publishing `sandbox.expired` then `sandbox.deleted` once, with the zero-leak diff clean (domains, overlays, taps, IP leases, bookings, scalar claims) and every readiness and expiry counted in the metrics; that guest memory captured on one sandbox comes back in one created `fromSnapshot`, and the stored object is deleted with its sandbox; and that deleting the warm template drains its pool until the hosts are empty. Since bfb1d45 it also asserts the last two bullets: `instances.stop`, `start` and `delete` on a sandbox's instance are refused `INVALID_STATE` naming the sandbox (`get` still works), and a user of another tenant, with its own template on the same pool, gets `NOT_FOUND` restoring from the snapshot and reading it (`sandboxSnapshots.get`); `tests/vps-sandboxes.test.ts` covers both against emulated host reports too. Nothing runs on a real host |

## Sandboxes [#sandboxes]

Sandboxes are part of the product beside GPU and CPU capacity. Decision D24 (PLAN.md section 17) was taken on
2026-09-24: build a dedicated sandbox API, a `sandboxes` group on `vps()`. PLAN.md drops "a sandbox product" from the
v1 non-goals, describes the flow in section 3.7, adds the `sandboxes` and `sandboxTemplates` rows to section 9.2,
and makes it a v0.5 deliverable with its own exit test ([v0.5](#v05-gpu-depth-and-ecosystem)). What exists and what
does not:

| Item                                                       | State                     | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ---------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The sandbox catalog                                        | Done                      | `compute-sandbox` (parent: the pool; attributes `templateId`, `vcpu`, `memoryGiB`, `gpus`, `ttlSeconds`, `fromSnapshot`, `warm`), `compute-sandbox-template` and `compute-sandbox-snapshot` (attributes `sandboxId`, `templateId`, `includeMemory`; owned by the sandbox's owner, whoever took it), the `compute:sandboxes:*` actions, and 15 method rows at phase v0.5, all registered. See [actions and resources](/docs/guides/iam/actions-and-resources)                                                                                                                                                                                                                                                                                                                                |
| Templates                                                  | Done                      | `sandboxTemplates` create, list, get, update and delete: a hosted pool, a flavor, an image and the pool's own runtime, checked once; user data; `defaultTtlSeconds` (1 h) and `maxTtlSeconds` (24 h); `warm: { min, max }`. Also in FleetConfig (`sandboxTemplates`). See [templates](/docs/guides/vps/sandboxes#templates)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Sandboxes with a lifetime                                  | Done                      | `sandboxes` create, list, get, extend and delete. One decision on `compute-sandbox/{draftId}`, the pool's budget check, then one hosted instance admitted, placed and charged like `instances.create`. The TTL starts at admission; `extend` stays within `maxTtlSeconds`; the `vps.sandboxes` reconciler expires sandboxes into an ordinary instance delete                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Warm pools                                                 | Done                      | `warm.min` instances per template, kept `paused` on runtimes that can pause (`libvirt-kvm`, `cloud-hypervisor`, `firecracker`) and `stopped` otherwise, at most 4 created per template per pass; claimed with `FOR UPDATE SKIP LOCKED` only after the sandbox's own allow, without a new placement, and only when made from the template's current shape; drained beyond `warm.max` and when stale; metered as `vps.sandbox.warm` until claimed                                                                                                                                                                                                                                                                                                                                             |
| Paused instances                                           | Done                      | `desiredState: 'paused'` in the lifecycle, `InstanceObservedState.PAUSED` in host reports, `RuntimeTraits.pause`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Instance guard and teardown                                | Done                      | `instances.start`, `stop`, `resize` and `delete` refuse a sandbox's or a warm instance with `INVALID_STATE`, and so does `tunnels.open` for every op (`reason: 'sandbox-managed'`); `instances.list` hides warm instances; `vps().tenantDeleted` deletes every instance and snapshot of a deleted tenant, sandboxes' and warm ones included                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Hosted runtimes on your own servers                        | In progress               | bcd has the `instances` converger (fab94d4), `instance.snapshot` and `restore_from` (be1e7d9), serves runtimes from bcx plugins (494de69), and runs `libvirt-kvm` on a host build with `virsh` (362317c: overlays on verified base images, NoCloud seeds, VFIO GPUs, snapshots with memory, restores checked for runtime and CPU model) and `cloud-hypervisor` on one with the `cloud-hypervisor` binary (2598f0a, the same features). Both are tested only against fake tools and no Kata plugin exists, so no sandbox has booted on a real host yet                                                                                                                                                                                                                                       |
| Snapshots and restore                                      | Done on the control plane | `sandboxes.snapshot` with `vps({ snapshots: SnapshotStore })`, the `instance.snapshot` command kind, the `sandbox_snapshots` table, `sandbox.snapshot.ready`, `.failed` and `.deleted`; `create({ fromSnapshot })` of the caller's own snapshot on the same runtime and, for memory, the same `cpu.model`; `sandboxSnapshots` list, get and delete, so a kept snapshot can be removed. bcd runs snapshots and restores on fakehw builds and, as a tar of the flattened disk, the seed and `virsh save`'s memory state, on `libvirt-kvm` (tested against a fake `virsh`), and with cloud-hypervisor's own memory snapshot on `cloud-hypervisor` (tested against a fake VMM API). See [snapshots](/docs/guides/vps/sandboxes#snapshots-and-restore)                                           |
| Exec                                                       | Done                      | `sandboxes.exec` (tunnels module): an `ssh` ticket and a certificate of at most 15 minutes, never past the sandbox's expiry, from the deployment's sandbox SSH CA (`pki add-sandbox-ca`, `tunnelsOptions`), which guests trust through vps's user data; the tunnel is re-decided on the sandbox every minute; with `certifyHostKey`, guests present host keys the same CA certified and the CLI pins them. Exec is the CLI's `better-compute sandbox exec`, which runs your OpenSSH client through the tunnel. No agent in the guest. Deferred: the SDK helpers PLAN.md section 3.7 describes (`sandbox.exec(cmd)`, `files.put`, `files.get`), because they need an SSH client in the SDK, a new dependency decision. See [running code inside](/docs/guides/vps/sandboxes#run-code-inside) |
| Template `env` and `network`; the `vps.sandbox.warm` meter | Done                      | `env` from the tenant's secrets, sealed into `/etc/sandbox/env`; `network` carried as `InstanceSpec.network_policy` and enforced by bcd with host nftables rules (6db49c8; `private` and `none` need the pool's IP range and place only on hosts labelled `bc.dev/cap.instance.network-policy`, cfdb918); `vps.sandbox.warm` from placement to claim (`iam setup --meters` creates the meter). Template secrets must be the tenant's own, written with `secrets.put` and not fleet-scoped, and naming them also decides `compute:secrets:manage`                                                                                                                                                                                                                                            |
| Clients                                                    | Done                      | The CLI's `sandbox`, `sandbox-template` and `sandbox-volume` commands, eight MCP tools with gate rules (none takes a snapshot or runs code), the console's Sandboxes page, and FleetConfig `sandboxTemplates`. See [MCP](/docs/guides/clients/mcp#the-tools)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

## v1.0: Freeze [#v10-freeze]

v1.0 freezes the API and the protocol, guarantees N-2 version skew, adds an external security review and SLSA level 3
builds, and publishes to npm with provenance from CI. Its acceptance: upgrading from v0.5
disturbs no running allocation, with no open critical or high findings.

| Item                                               | State              | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| -------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API freeze                                         | In progress        | `scripts/api-surface.mjs`, run by `pnpm lint:repo`, fails on a breaking change to any served method or error code against `generated/api-surface.json`; additions are recorded with `pnpm api:surface`. See [the API surface freeze](/docs/operations/testing#the-api-surface-freeze)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Protocol freeze                                    | In progress        | `agent/v1` and `common/v1` have been frozen since P0 (`buf breaking` in CI); the draft `site` and `driver` packages are not yet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| N-2 version skew                                   | In progress        | `createGateway({ protocol })` sets the range a gateway speaks. `tests/skew-matrix.test.ts` runs the matrix on real dev stacks: FakeDaemons built like the N, N-1 (v0.4) and N-2 (v0.3) lines (`daemonSkewProfiles` in `@better-compute/testing`) are welcomed, adopted and refuse the kinds they predate `UNSUPPORTED`, and a daemon below the gateway's minimum gets `upgrade_required`. No older real bcd build runs in it, and only protocol version 1 is defined. See [version skew](/docs/daemon/protocol#version-skew)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| SLSA level 3 builds                                | In progress        | `release.yml` builds bcd with SBOMs, keyless cosign signatures and SLSA3 provenance; no release has been cut                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| npm publishing with provenance                     | In progress        | 0.1.0, 0.1.1 and 0.2.0 are on npm (`latest`: 0.2.0), published from a verified clean checkout, with Better IAM `>=0.2.0 <0.4.0` from npm. Provenance attestations come once `ci.yml`'s publish job publishes from the GitHub repository                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| External security review                           | Not started        | A process item                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Destructive-batch quorum (PLAN.md §13)             | In progress        | Fleet-wide destructive batches need k-of-n operator approval, checked on the machine. Done: the protocol seams (`CommandBody.batch_id`, `BatchApproval`, `BatchProof`, the trust bundle's `approver_keys`, `approver_threshold`, `quorum_tiers` and `unbatched_destructive_per_hour`, `BATCH_UNAPPROVED` and `UNBATCHED_LIMIT`; 525c199, 4f11548, 081928a), the pki primitives (approver keys, the `(subject, kind)` scope tree, approvals, `verifyBatchCommand`; 36387c8, 2d2fe6f), `pki.approvers` in the server's signed trust bundle (b816600), the server's `quorum` module with `commandBatches` get, list, scope, approve and cancel (ce067e0), the command queue's `createBatch` and `enqueue({ batchId })`, which holds batched commands until approval (6067791), metal batch scopes and steps that declare what they send (add7e0e), `better-compute batch keygen` and `batch sign` (0210671), and enforcement in the FakeDaemon (4b48f08) and bcd (e7ddb38). Since then: quorum review #6 (payload-bound leaves, site-qualified BMC subjects, the trust domain in approvals, each leaf run once; 77cdf4f, d566cbe, 8b0f50c), the gateway attaching approvals and proofs to batched commands (40be696), `batch list`, `show`, `approve` and `cancel` with the approver-side checks (b3fd712), approver-set changes signed by the current approvers (`batch sign-approvers`, `pki.approvers.setSignatures`; 6a7f1b7, edbd63f), a trust bundle version that follows its content (9c40cd5), and enqueue refusing an unbatched destructive command up front (`batch-required`, 508728e). Then the review 7 fixes: every command signed once signed intent is in force and `site.httpProxy` destructive (0d42de8, 12ccfa1), approver-set signatures bound to the trust domain (4835f4f), and leaves that carry their payload for approvers (`quorum/0007_leaf_payload`, 2ca0ba4), then approver sets with their own epoch (c6214dc), a first set pinned out of band (`--approver-set-sha256`, 9b7a199), and leaves limited to the tenant's own machines with payloads of their kind's type (fc2df7d); and the producers: metal's per-workflow batches and `metal.bulk` (f66e236), `metal.power` (478025a), the health reconciler's remediations (e594a4f) and the MCP server's `propose_command_batch` (171ad2f). Open: the batches built-in workflows create leave their leaves' payloads unbound, since the bytes are only known later (a BMC credential rotated mid-workflow, user data rendered by an earlier step); PLAN.md §13 (a1bd0aa) names binding a leaf to the operation instead of its bytes, for example `payloadMatch { operation, args }` checked by the agent after decoding, as a post-v1.0 option. Since then, review 10: one epoch-adoption rule for every change, the held set reported in `Hello` and `StatusReport`, recorded by the gateway (fa39201, `core/0026`) and checked by doctor and `pki trust-version`, a node holding no keys counting as lagging (5fe7cbf, e171f0d, 6612496, 08eb62e), the pin enforced by bcd, operations grouped by payload digest in the approver CLI (b118616), and `batch create` and `batch members`. See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) |
| Upgrading from v0.5 disturbs no running allocation | Done in simulation | `upgradeInPlace(rig, { migrations, rollDaemons })` in `@better-compute/testing` runs expand migrations, workers and api, each gateway killed and revived, then the daemons, and reports every disturbed allocation. `tests/sim/upgrade.test.ts` (6 nodes, 3 gateways, 6 active reservations, a signed rollout to 0.6.0) finds none, bills the hour exactly, and keeps I1, I2, I5 and I8; it also runs two instances and a sandbox on two VPS hosts (`ControlPlaneSim` `extraPools`), and every VM stays running on its host with the same boot count while the sandbox keeps its expiry. A second scenario (80195df) starts from a v0.5-era database without the quorum tables, upgrades, then turns the quorum on: no allocation moves, setting approvers moves no spec generation, the trust bundle's version bumps exactly once, every daemon takes the set, a `node.reboot` queued before the switch is sent but rejected `UNBATCHED_LIMIT`, and a new unbatched one is refused `batch-required` (see [turning the quorum on](/docs/operations/pki#turning-the-quorum-on)); FakeDaemons only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

## CPU, memory and Kubernetes (v1.1) [#cpu-memory-and-kubernetes-v11]

PLAN.md §3.8 (a760805) designs the complete node daemon for v1.1: compute books a host's CPUs, memory and huge pages
with its GPUs, the node enforces exactly that, bcd applies the host's configuration, and a reservation can hand its
GPUs to a Kubernetes cluster. The principle stays P1: compute decides who gets what, and the node enforces exactly
that and reports it. Most of it is now in the code, on both sides. Every node-side piece is tested against fakes
only (a fake host, a fake root, a fake kubelet, fake VMMs); none has run on a real machine or cluster.

| Item                              | State                      | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| --------------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Host configuration applied by bcd | Done in code, mostly fakes | The host converger applies twelve units in order, `kernel-modules`, `sysctl`, `hugepages`, `cpu`, `iommu`, `nvidia-driver`, `fabric-manager`, `gpu-settings`, `mig-layout`, `gpu-sharing`, `container-toolkit` and `k8s-join` (6c2343f, 7579184, 1cad924, 3fc3628, 3fd1c41), a failing one never holding back the rest (c21ef84), drift repaired by a pass every 10 minutes (3c0fe02), and settings that wait for a reboot reported as the informational node condition `RebootRequired` (c21ef84, 0aea5c0); under `bcd helper` each unit is applied as root only from a signed spec (03c023d, 2d18327), and root writes only under root's directories, with its records in `/var/lib/bcd-hostconfig` (f8e13ef, 02485b3, 1bb78d6). Provision renders every unit from profiles and from pool policy (8ce9140, 1f0f683, 7db110a, 09669d3, e4b0c01), and the pools API refuses a host policy nodes cannot take (37dcd02); `[host] units = false` for hosts configured by something else (a193249), which `bcd helper` also honours from root's own root-only `config.toml` (3fa9021). `TestRealKernel` runs sysctl, THP and 2 MiB pages on a real kernel (60143bf); `tests/e2e/bcd-host.test.ts` runs a pool's units through the control plane (91f5e81). See [host configuration](/docs/daemon/host-config). Open: no driver install, MIG repartition, boot argument or cluster join has run on a real machine                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Host resources on every host      | Done                       | `capacity.hosts` writes `vcpu`, `memoryGiB`, `hugepages2Mi` and `hugepages1Gi` for every host, under `policy.hostReserve` and the overcommit ratio (8a7f8eb); vps's `memoryGiB` leaves out the huge page pools (9d55b36). Inventory reports per-NUMA huge pages and isolated CPUs, with the labels `cpu.count`, `cpu.sockets`, `cpu.isolated`, `numa.nodes`, `memory.totalGiB`, `hugepages.2Mi` and `hugepages.1Gi` (badb8c2). Inventory does not report available memory                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Reservations book CPU and memory  | Done                       | `reservations.create` takes `resources { vcpu, memoryGiB, hugepages1Gi }`, booked on the devices' node in the same statement; without it a request books `policy.perGpu` or the pool's smallest per-GPU share, for whole GPUs and, since 004579e, for MIG instances and slots as fractions of a GPU (a21c1c1). The share is the floor, and anything above it is charged to `vcpu` and `memoryGiB` quota (41826a2, 004579e). See [host resources](/docs/guides/capacity/reservations#host-resources)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Enforcement on the node           | Done in code, fakes only   | Device assignments carry the booked resources as limits, with a cpuset on the devices' NUMA nodes that leaves out the host's reserved CPUs, the lowest non-isolated, `policy.hostReserve.vcpu` (c5a419c, b6b804e); bcd admits a hold against the host's memory beside its VMs, failing it `resources-exhausted` for re-placement (31555bf), creates `bcd-alloc-<allocation>.slice` before the CDI spec names it, restores it after a reboot and stops it once empty, retrying every pass (1d03c28, 260e33c, e01d7ff), and reports it in `AllocationBound.cgroup_parent`, which `reservations.get` shows as `binding.cgroupParent` (6aa412b, c81f230, 19dbf60). VMs get `InstanceSpec.placement` on both runtimes (a184ec2), pinned guests take isolated CPUs first (b6b804e), guests of memory-overcommitted pools get a balloon (073324a), and bcd refuses an instance start the host cannot back (`ResourcesExhausted`, 0dd122a), which vps records as `HostResourcesExhausted` and answers by placing a not-yet-started instance on another host after a minute, at most three times (564f543). `bcd doctor` checks the cgroup controllers, huge page pools, `vfio-pci` and systemd (ab26e87). Reservations with host resources, and `kubernetes` bindings, place only on a bcd that advertises `devices.limits` or `binding.kubernetes` (409f783). Open: the workload manager must place workloads in the slice                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Holds of MIG instances and slots  | Done in code, fakes only   | bcd holds `gpu.mig` and `gpu.slot` atoms with parent-aware conflicts and no hold beside a VM's GPU (46a61db), and `gpu.reset` and `diag.dcgm` refuse GPUs those holds or VMs use (539c4ce)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Kubernetes                        | Done in code, fakes only   | Pools name the cluster in `policy.kubernetes`; the `k8s-join` unit installs, joins and keeps the Node's labels and taints, with the bootstrap token released by binding rule 3 (88ecb07, 7579184, e93bf9c); `binding: 'kubernetes'` (a21c1c1) binds like a hold (d02ba4f), and bcd's device plugin advertises exactly those devices as `bcd.dev/gpu`, `.mig` and `.slot` and deletes pods that keep released ones. See [Kubernetes workers](/docs/guides/capacity/kubernetes). The DRA driver `gpu.bcd.dev` offers the same devices through one ResourceSlice per node instead, chosen by `gpuApi` (7d24836, fa8611e, e4b0c01), and neither API offers a GPU that went bad (6f5a99f). bcd also runs as a DaemonSet on an existing cluster (8df8c70, 2c9f953), with hosted join tokens that land its nodes `tenant-controlled` unless a platform-owned pool is platform-operated (a27156a, 259d82c), in hosted mode with the device plugins or DRA and its `bcd.dev/*` labels but no join (1e5f96b), and `kubernetes manifest` renders it with operator-given pins (a2eb28f, d2cd813); see [bcd on Kubernetes](/docs/daemon/kubernetes). Kubernetes bindings get no slice: the kubelet enforces pods' limits, and their booked limits count in host admission (e01d7ff). The pools API checks `policy.kubernetes` (37dcd02). Only a pool's owner books `binding: 'kubernetes'` (3825ec7), and `ForeignGpuExposure`, `PodResourcesUnavailable` and `KubernetesReleaseStuck` are NoSchedule node conditions, the first also degrading the node's reservations (ebd9806, 4570dfe). Security review 2 (8bcbe92, f88bcdb): a released GPU stays in use until no pod, prepared DRA claim or pod process holds it, failing closed on any node that ever was a worker, and `ForeignGpuExposure` also follows the toolkit the container runtime runs, other GPU DRA drivers and, hosted, an unreadable runtime configuration. The DaemonSet mounts only the kubelet sockets and credentials bcd uses, never `/var/lib/kubelet` or `/etc/kubernetes` (27f2c1b, a5eee84), and the runtime and toolkit configuration read-only (d11810c, 3775f46). `daemon/Containerfile` bundles a pinned, checksummed `kubectl` v1.34.1, which bcd runs as the kubelet (9bdd43a, e7d0765); CI builds the image and server-side dry-runs the manifests on kind (ac1ca4b). Open: no real cluster has run it (the CI job is a dry run), and pod requests are not derived from the reservation |
| Sandboxes and VM hosts            | Done in code, fakes only   | Flavors take `placement { numaAffine, pinned, hugepages }`, which sandbox templates inherit; vps chooses the NUMA node and pinned CPUs and marks a host that cannot honour them `PlacementDegraded` (9d55b36). Sandboxes, placed flavors and volume-bearing instances land only on a bcd that advertises the enforcing capability, `instance.ssh-host-only`, `instance.placement` or `instance.volumes` (82f4f51). Memory snapshots of GPU sandboxes and of sandboxes with a read-write volume are refused up front (2a09202). See [flavors](/docs/guides/vps/flavors#placement)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Agent memory: sandbox volumes     | Done in code, fakes only   | `sandboxVolumes.create`, `list`, `get`, `resize` and `delete`: a persistent disk booked as `diskGiB` on one host (core/0028, vps/0020), attached by serial to the sandboxes of templates that name it, one writer or many readers, pinned to its host, metered as `vps.volume.gib.hours` (2a09202, 8cbece0); bcd keeps each as a sparse raw file, grows it live, keeps one writer through handoffs and destroys it when unlisted (550da91, b5415a2, 448dc5f, 39458f9, 408e002). bcd never promises more than its data directory's filesystem holds and reports the space in `StatusReport.volume_storage` (851016e, 37d6d6d, b8e7271), which vps books volumes against, never overcommitted (131f5ee). The CLI's `sandbox-volume` commands and `sandbox-template --volumes` (f181f43), four MCP tools (e6759df), FleetConfig templates (94b8b70) and the console's Volumes page (50add82) expose them. A template mounts a volume only with a decision on that volume (77bf9f5), lists are per caller by default, and create and resize are decided with the size, so a policy can bound it (6cb30e9). See [volumes](/docs/guides/vps/sandboxes#volumes). The console's Sandboxes page attaches and detaches a template's volumes (dd5555b). At scale (38405a0), `tests/e2e/volumes.test.ts` churns 240 sandboxes over 12 volumes on three FakeDaemon hosts through the real server and gateway, host reboots included, with no leak, no second writer, no overbooked store and no lost data; see [testing](/docs/operations/testing#volumes-at-scale)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |

The work's owners are in `.claude/coordination.md` ("Round 5").

## What unblocks what [#what-unblocks-what]

| Unblocked by                                                                                                                                                                                                                                                                                | Moves                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Writing more of bcd (Go is installed now; P0's code generation, W1's node agent, `bcd site`, `bcd ramdisk`, the `instances` converger, `libvirt-kvm`, `cloud-hypervisor`, self-update, the WebSocket fallback, `bcd helper`, bcx plugin supervision and the `bcx-ipmi` BMC driver are done) | Vendor BMC drivers, bcx plugins beside an unprivileged node, NVML events and a go-nvml backend; an end-to-end run of the site agent and ramdisk, and of bcd with a bcx plugin                                                      |
| A bcd release                                                                                                                                                                                                                                                                               | Installing real nodes with the install script                                                                                                                                                                                      |
| A PostgreSQL 17 server                                                                                                                                                                                                                                                                      | The concurrent capacity and torture suites, the IAM-lane gate, the 30-minute gateway gate, every v0.4 number                                                                                                                       |
| The D15 hardware                                                                                                                                                                                                                                                                            | CDI and a bound GPU hold on an NVIDIA host (real inventory and live GPU metrics have run, on one RTX 5090 under WSL 2), emulation CI on a KVM runner, the first `libvirt-kvm` run on real KVM, the MIG and real-hardware nightlies |

For what changed recently, see the [changelog](/docs/reference/changelog).


# Access (/docs/guides/access)

> How people reach nodes and instances through tunnels with daemon-verified exec tickets, break-glass SSH certificates and allowlisted commands.



Sooner or later someone needs a shell. A training job hangs on one GPU, a VM does not boot its service, a node
reports an XID and an engineer wants to look at `dmesg`. The usual answer, SSH keys copied onto every machine,
leaves standing access that nobody reviews and nobody removes.

Better Compute gives no one standing access. Every interactive session is decided by IAM at the moment it opens,
carries a credential that is bound to one machine and expires within minutes, and is checked again while it
lasts. The gateway that relays the bytes holds no key that could open a session on its
own.

## Three ways in [#three-ways-in]

| Path                                                                       | Target                | Decided as                                                | Credential                                            | Lifetime                         |
| -------------------------------------------------------------------------- | --------------------- | --------------------------------------------------------- | ----------------------------------------------------- | -------------------------------- |
| Tunnel, op `exec`, `ssh`, `tcp` or `console` (the serial console, 798e710) | An instance           | `compute:instances:exec` on the instance                  | An exec ticket from `tunnels.open`                    | Ticket 60 s; tunnel at most 12 h |
| Tunnel, op `console`                                                       | A node (a root shell) | `compute:nodes:console` on the node                       | An exec ticket from `tunnels.open`                    | Ticket 60 s; tunnel at most 12 h |
| Break-glass SSH                                                            | A node                | `compute:nodes:ssh-root` on the node, with an MFA session | An OpenSSH user certificate from `nodes.issueSshCert` | 15 min by default, 1 h at most   |
| Allowlisted commands                                                       | A node                | `compute:nodes:exec` on the node                          | None: the daemon runs a registered command kind       | The kind's timeout               |

* **[Tunnels](/docs/guides/access/tunnels)** are the everyday path. The caller gets a short
  exec ticket and opens a WebSocket with it; the API relays the stream to the gateway
  that holds the node's session, and the node opens a `Tunnel` stream back. The browser
  terminal, an instance's serial console page and a node's console in the console use exactly this. bcd refuses
  `exec` into an instance until guests have an agent, so on a real host an instance's serial console is the terminal
  that works.
* **[Break-glass SSH](/docs/guides/access/ssh)** is for platform operators when the agent itself is the problem. It
  signs the caller's own public key with the compute SSH CA; nothing goes through the gateway.
* **[Commands](/docs/guides/nodes/commands)** (`commands.run`) cover the repeatable jobs, such as a diagnostic or a
  log collection, without any shell at all. Each command kind is registered and
  audited.

## How a tunnel is trusted [#how-a-tunnel-is-trusted]

Three processes check every exec ticket before a byte flows, and each keeps its own set of ticket ids, so a ticket
opens one tunnel, once.

<Mermaid
  chart="flowchart LR
  C[&#x22;Caller&#x22;] -->|&#x22;tunnels.open&#x22;| A[&#x22;api: IAM decision, issues ticket&#x22;]
  C -->|&#x22;WebSocket with ticket&#x22;| A
  A -->|&#x22;verify, jti set&#x22;| A
  A -->|&#x22;internal mTLS&#x22;| G[&#x22;Gateway: verify, jti set&#x22;]
  G -->|&#x22;OpenTunnel on the session&#x22;| D[&#x22;bcd: verify against pinned JWKS, jti set&#x22;]
  D -->|&#x22;Tunnel stream&#x22;| G"
/>

* **The api** decides the caller, then signs an ES256 ticket for one node (`aud` = `node:<id>`), one tunnel
  (`sid`), one operation (`op`) and at most 60 seconds.
* **The gateway** verifies the signature against the issuer's JWKS, the audience, the lifetime and the `jti`, then
  asks the node over its session.
* **The daemon** verifies the ticket again against the JWKS pinned in its NodeSpec
  `access` section and its own `jti` set, and only then opens the stream.

The last check is the one that matters. PLAN.md's security model treats the gateway as relay-only: a compromised
gateway can delay, drop or read frames, but it holds no signing key, so it cannot mint a ticket the daemon would
accept. The gateway test suite covers exactly that case: a gateway handed an attacker's JWKS accepts a forged
ticket, and a bcd-like test daemon holding the real JWKS refuses it, so no tunnel opens. bcd's own tests refuse
replayed, expired, tampered and other-node tickets and a ticket whose `sid` is not the tunnel's.

## Authorization [#authorization]

The actions live on the `compute-node` and `compute-instance` resource types (see
[actions and resources](/docs/guides/iam/actions-and-resources)):

| Action                   | Resource                | Used by                             |
| ------------------------ | ----------------------- | ----------------------------------- |
| `compute:instances:exec` | `compute-instance/<id>` | `tunnels.open` with an `instanceId` |
| `compute:nodes:console`  | `compute-node/<id>`     | `tunnels.open` with a `nodeId`      |
| `compute:nodes:ssh-root` | `compute-node/<id>`     | `nodes.issueSshCert`                |
| `compute:nodes:exec`     | `compute-node/<id>`     | `commands.run`                      |

A node must also be visible to the calling tenant through the authorization projection; otherwise the answer is
`NOT_FOUND`, never a hint that the node exists.

PLAN.md §12.6 describes break-glass as an eligible binding: a policy that allows `compute:nodes:console` and
`compute:nodes:ssh-root` only with MFA and only on platform-trusted nodes, bound so that people activate it when
they need it. `better-compute iam setup` does not print this policy; write it yourself in the
[policy](/docs/guides/iam/policies) format:

```json title="break-glass (eligible binding)"
{
  "name": "break-glass",
  "document": {
    "version": 1,
    "statements": [
      {
        "sid": "BreakGlass",
        "effect": "allow",
        "actions": ["compute:nodes:console", "compute:nodes:ssh-root"],
        "resources": ["compute-node/*"],
        "conditions": {
          "Bool": { "principal.mfa": true },
          "StringEquals": { "resource.trust": "platform" }
        }
      }
    ]
  }
}
```

The `agent-guardrails` example policy that `iam setup` does print denies `compute:nodes:ssh-root` to agent keys and
to delegated sessions, so an AI agent never gets a root shell through a person's delegation.

### Re-checks while a tunnel is open [#re-checks-while-a-tunnel-is-open]

The api re-authorizes the caller every 60 seconds. When the WebSocket upgrade carried a credential (an
`Authorization` header or the IAM cookie), each re-check is an ordinary decision with it. Browsers cannot set
headers on a WebSocket, so without one the re-check goes by the identity and session recorded in the ticket: the
session must still be listed and unexpired, the identity active, and `policies.simulate` must allow the same action
on the same target. Sessions of agents, delegations and assumed roles cannot be simulated exactly, so they answer
false and the tunnel closes at its first re-check. A refusal closes the tunnel; IAM unreachable for 2 minutes closes
it too.

## Audit and events [#audit-and-events]

* `tunnels.open`, `nodes.issueSshCert` and `commands.run` are mutations, so each allowed call writes an
  `audit_events` row with its admission context: who, which action on which resource, and when. See
  [audit](/docs/guides/events/audit).
* `tunnels.open` publishes `tunnel.issued` (subject `instance/<id>` or `node/<id>`) with the tunnel id, node,
  op, identity and expiry. `nodes.issueSshCert` publishes `node.ssh-cert.issued` with the certificate's key id,
  serial, principals and expiry. Both reach [webhooks](/docs/guides/events/webhooks) and
  [watch](/docs/guides/events/watch) like any other event.
* The end of a tunnel is reported to the host's `onClose` hook with the tunnel id, node, identity and reason
  (`revoked`, `iam-unreachable`, `max-duration`, `caller-closed` and so on). No audit row is written for it by
  default.

## What ships today [#what-ships-today]

Interactive access is the v0.3 roadmap item "tunnels with daemon-verified tickets, `nodes.issueSshCert`, the
`access` section". Both halves are written and tested: the control plane, and the node half in the Go bcd (built,
not released yet) and in the TypeScript FakeDaemon.

| Piece                                                                                               | State                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tunnels.open`, `nodes.issueSshCert`                                                                | Implemented in `@better-compute/server`, registered when you configure `tunnels`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Exec tickets, the SSH CA, JWKS rotation                                                             | Implemented in `@better-compute/pki`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `GET /api/compute/tunnels/<ticket>` relay, re-authorization                                         | `handleTunnel`, mounted by `better-compute serve` and by the console's `server.mjs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Gateway relay (`OpenTunnel`, `Tunnel` stream, internal `/v1/tunnels`)                               | Implemented in `@better-compute/gateway`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Browser terminal, instance serial console and node console                                          | `apps/console` with `@better-compute/console-kit`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| NodeSpec `access` section (JWKS, SSH CA keys)                                                       | Sent to every node                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Daemon side: verifying tickets, opening `Tunnel` streams, running shells, applying `access` to sshd | In bcd (built and tested, not released): it re-verifies each ticket against the NodeSpec's JWKS and its journaled `jti` set, runs a node console on a real pseudo-terminal and pipes `ssh` and `tcp` to loopback ports (`tests/e2e/bcd.test.ts` opens a console end to end), and its `access` converger writes the sshd drop-in, CA, node-bound principals and KRL. `exec` into an instance waits for bcd's instances runtime. The TypeScript FakeDaemon, with its `tunnels` option (on in `dev --sim`), answers `OpenTunnel` with an echo shell, which the CLI's `console` and `exec` tests drive end to end; it still reports a non-empty `access` section as `SectionUnsupported` |

> **The generated reference says planned.** 
  The [API reference](/docs/reference/api/tunnels#open) lists `tunnels.open` and `nodes.issueSshCert` as planned for
  v0.3, because it is generated from a control plane built without `tunnels` options. Both methods exist in the
  code and are served as soon as you pass `tunnels` to `betterCompute`.

## In this section [#in-this-section]

  - [Tunnels](/docs/guides/access/tunnels): tunnels.open, exec tickets, the WebSocket relay, re-authorization, and the web terminal.

  - [Break-glass SSH](/docs/guides/access/ssh): Short OpenSSH user certificates from the compute SSH CA, for MFA sessions only.

  - [Commands](/docs/guides/nodes/commands): Allowlisted command kinds with commands.run, without a shell.

  - [Security model](/docs/operations/security): Keys, trust boundaries, and what a compromised gateway can and cannot do.


# Break-glass SSH (/docs/guides/access/ssh)

> Issue short-lived OpenSSH user certificates from the compute SSH CA with nodes.issueSshCert, only for MFA sessions that hold compute:nodes:ssh-root on the node.



[Tunnels](/docs/guides/access/tunnels) and [commands](/docs/guides/nodes/commands) both go through the node's agent.
When the agent itself is the problem (it crashed, it cannot reach its gateway, its session is stuck), a platform
operator needs a way in that does not depend on it. That is break-glass SSH.

Better Compute runs an SSH certificate authority. An operator sends the public key of their own key pair to
`nodes.issueSshCert`; the CA signs it into an OpenSSH user certificate for one Unix account that lasts minutes, and
the operator connects with plain `ssh`. The private key never leaves their machine, no `authorized_keys` file
changes, and every certificate is recorded.

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant O as Operator
  participant A as api
  participant I as Better IAM
  participant N as Node sshd
  O->>A: nodes.issueSshCert (public key, account, TTL)
  A->>I: require compute:nodes:ssh-root on the node
  A->>A: MFA session? node visible to the tenant?
  A-->>O: certificate, valid 15 min by default
  O->>N: ssh root@node with the certificate
  N->>N: check the CA signature, principal and validity"
/>

The certificate does not travel through a gateway: you reach the node's `sshd` over your own network path, such as
a management network or VPN.

## Set up the CA [#set-up-the-ca]

`better-compute pki init` writes an SSH CA into the PKI bundle, and `better-compute pki show` prints its public key
as `sshCaPublicKey`. Pass the bundle's CA to `betterCompute` beside the exec ticket issuer; the CA is part of the
`tunnels` options, so it needs an issuer too (see [tunnels](/docs/guides/access/tunnels)).

```ts title="compute.ts"
export const compute = betterCompute({
  // database, secret, iam, modules, pki ...
  tunnels: {
    issuer: execTicketIssuer(bundle, { issuer: 'https://compute.example.com' }),
    ssh: bundle.sshCa, // [!code highlight]
    sshRequiresMfa: true, // the default
    sshAccounts: ['root'], // the default: the Unix accounts certificates may name
    limits: { tunnelsPerMinute: 10, sshCertsPerHour: 20 }, // the defaults, per identity and api replica
  },
});
```

Beyond the limits a call is `RATE_LIMITED` with `retryAfterMs`. Each api replica counts its own calls in memory.

To manage the key yourself, build the CA with `createSshCertificateAuthority` from `better-compute/pki` and give
every api replica the same private key:

<TypeTable
  type="{
  algorithm: {
    type: &#x22;'ed25519' | 'ecdsa-p256'&#x22;,
    description: 'The CA key type when no privateKey is given. ECDSA P-256 suits FIPS-style deployments.',
    default: &#x22;'ed25519'&#x22;,
  },
  privateKey: {
    type: 'KeyObject | Uint8Array | string',
    description: 'The CA private key (PEM or DER). ca.export().privateKeyPem returns it.',
    default: 'a new key',
  },
  comment: {
    type: 'string',
    description: 'The comment on the CA public key line.',
    default: &#x22;'better-compute-ssh-ca'&#x22;,
  },
  maxTtlSeconds: {
    type: 'number',
    description: 'The longest certificate this CA signs. Values above 3600 are lowered to 3600.',
    default: '3600',
  },
  clock: { type: 'Clock', description: 'The clock validity is computed from.', default: 'systemClock' },
}"
/>

`ca.publicKey` is the OpenSSH public key line (`ssh-ed25519 AAAA... better-compute-ssh-ca`). The tunnels module puts
it into every node's NodeSpec as `access.sshCaPublicKeys`.

## Issue a certificate [#issue-a-certificate]

`nodes.issueSshCert` is decided as `compute:nodes:ssh-root` on the node (see
[the reference](/docs/reference/api/nodes#issuesshcert)).

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant acting.', required: true },
  nodeId: { type: 'string', description: 'The node (node_...).', required: true },
  publicKey: {
    type: 'string',
    description: 'Your OpenSSH public key line, Ed25519 or ECDSA P-256, at most 4096 characters.',
    required: true,
  },
  principal: {
    type: 'string',
    description: &#x22;The Unix account to log in as: one of the module's sshAccounts.&#x22;,
    default: &#x22;'root'&#x22;,
  },
  ttlSeconds: { type: 'number', description: &#x22;Lifetime, 60 to 3600, and never past the caller's session.&#x22;, default: '900' },
}"
/>

A certificate never outlives the session that asked for it: when the caller's session has an expiry,
`validBefore` is the earlier of now plus `ttlSeconds` and that expiry.

| Situation                                                                          | Error                                                                  |
| ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| No SSH CA is configured                                                            | `UNSUPPORTED`                                                          |
| The session has no MFA (and `sshRequiresMfa` is not `false`)                       | `ACCESS_DENIED` with `details.reason: 'mfa-required'`                  |
| The session ends within a minute                                                   | `ACCESS_DENIED` with `details.reason: 'session-ending'`: sign in again |
| IAM does not allow `compute:nodes:ssh-root` on the node                            | `ACCESS_DENIED`                                                        |
| The node is not visible to the tenant                                              | `NOT_FOUND`                                                            |
| More than `sshCertsPerHour` certificates for this identity                         | `RATE_LIMITED` with `retryAfterMs`                                     |
| An RSA or malformed key, an account not in `sshAccounts`, a TTL outside 60 to 3600 | `INVALID_INPUT`                                                        |

  **CLI:**

    ```bash
    export BETTER_COMPUTE_URL=https://compute.example.com
    export BETTER_COMPUTE_TOKEN=...   # a session verified with a second factor
    better-compute ssh-cert --tenant root --node node_01jzq3v0m6y8x7w5t4s3r2p1na
    ssh -i ~/.ssh/id_ed25519 root@10.20.0.14
    ```

    `ssh-cert` sends `--key` (default `~/.ssh/id_ed25519.pub`, else `~/.ssh/id_ecdsa.pub`) as `publicKey`, `--account`
    (default `root`) as `principal` and `--minutes` (default 15, 1 to 60) as `ttlSeconds`, and writes the certificate
    beside the key as `<key>-cert.pub`. It prints the certificate's path, serial, principals, account and
    `validBefore`. A `--key` that is not a `.pub` file, or whose content holds a private key, is refused before anything
    is sent. What the server answers is checked before it is used: the account must be a Unix account name (it reaches
    ssh's `%r` and `%u` expansions, such as a bastion's `ProxyCommand`), and the certificate must parse as an OpenSSH user
    certificate for your own key before it is written into `~/.ssh`; otherwise `INVALID_RESPONSE`.

    `--connect HOST` issues the certificate and connects in one step. After writing it, `ssh-cert` runs your OpenSSH
    client (`--ssh`, default `ssh`) to the node's address, and the command after `--` runs there (without one, a
    shell):

    ```bash
    better-compute ssh-cert --tenant root --node node_01jzq3v0m6y8x7w5t4s3r2p1na --connect 10.20.0.14 -- journalctl -u bcd -n 50
    # runs: ssh -i <key> -o CertificateFile="<key>-cert.pub" -o IdentitiesOnly=yes -l root -- 10.20.0.14 journalctl -u bcd -n 50
    ```

    `HOST` must be a host name or an IPv4 or IPv6 address. Anything ssh would parse as an option, such as
    `-oProxyCommand=…`, is a usage error before any certificate is issued, and a command without `--connect` is one too.
    The command exits with ssh's own status (`REMOTE_EXIT`). It keeps your ssh configuration on purpose, so a bastion or
    `ProxyJump` you configured for the node still applies (unlike `sandbox exec`, which disables it for an untrusted
    guest).

    The same call through `api`:

    ```bash
    better-compute api nodes.issueSshCert \
      tenantId=root nodeId=node_01jzq3v0m6y8x7w5t4s3r2p1na \
      publicKey=@$HOME/.ssh/id_ed25519.pub principal=root ttlSeconds:=900 \
      --query certificate > ~/.ssh/id_ed25519-cert.pub
    ssh -i ~/.ssh/id_ed25519 root@10.20.0.14
    ```

    `--query certificate` prints the certificate line alone. Saved as `<key>-cert.pub` next to the private key, it is
    picked up by `ssh` automatically.
  
  **TypeScript:**

    ```ts
    const issued = await client.$request<{
      nodeId: string;
      certificate: string;
      serial: string;
      keyId: string;
      principals: string[];
      validBefore: number;
    }>('nodes/issueSshCert', {
      tenantId: 'root',
      nodeId: 'node_01jzq3v0m6y8x7w5t4s3r2p1na',
      publicKey: 'ssh-ed25519 AAAA... alice@laptop',
      principal: 'ubuntu',
      ttlSeconds: 600,
    });
    ```

    The tunnels module adds `issueSshCert` to the `nodes` group at run time, so the typed client does not list it; call
    it by path.
  
## The certificate [#the-certificate]

Every certificate is an OpenSSH `cert-v01` user certificate, written without external tools:

* **Type.** It follows your key: `ssh-ed25519-cert-v01@openssh.com` or `ecdsa-sha2-nistp256-cert-v01@openssh.com`.
* **One node-bound principal.** `<account>@<nodeId>`, such as `root@node_01jzq3…`, never a bare account. Each node
  accepts only its own principals (NodeSpec `access.sshPrincipals`), so a certificate for one node opens no other
  node that trusts the same CA. An empty principal list, which OpenSSH reads as "any account", is never issued.
* **Validity.** From 1 minute before issue (for clock skew) until `validBefore`, which the server computes on its own
  clock as now plus `ttlSeconds`, capped at the session's expiry.
* **Permissions.** Only `permit-pty`: no agent, port or X11 forwarding, and no user rc file.
* **Key id.** `bc:<identityId>:<nodeId>:ssh_<id>`, which sshd writes to its log on every login, so a log line leads
  back to the person and the issue record.
* **Serial.** Microseconds since the epoch, bumped past the last serial the CA issued.

To read one back, run `ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub`, or call `parseSshCertificate(line)` from
`better-compute/pki`, which returns every field and whether the signature verifies (`signatureValid`). Check that
`signatureKey` is your CA separately.

## Audit [#audit]

Each issue writes an `audit_events` row with the admission context and publishes `node.ssh-cert.issued` (subject
`node/<id>`) with the key id, serial, principals, identity and `validBefore`. Route it to a
[webhook](/docs/guides/events/webhooks) if every break-glass login should page someone. The certificate itself is
recorded in `ssh_certificates` (serial, node, tenant, identity, issuing session, principal, key id, expiry) for
revocation, re-authorization and audit.

## Revoke a certificate [#revoke-a-certificate]

`nodes.revokeSshCert({ tenantId, nodeId, serial, reason? })` ends a certificate before `validBefore`. It is decided
as `compute:nodes:ssh-root` on the node, takes the decimal `serial` that `issueSshCert` returned and a `reason` of up
to 500 characters, and answers `{ nodeId, serial, principal, expiresAt, revokedAt }`. It is idempotent: the first
revocation stands. An unknown serial, or one issued for another node or tenant, is `NOT_FOUND`.

A revocation invalidates the node's desired state and publishes `node.ssh-cert.revoked`. From the next NodeSpec on,
`access.sshRevokedSerials` lists the node's revoked certificates that are still within their lifetime: the node's
KRL, for sshd's `RevokedKeys`. The hourly `tunnels.ssh-certificates-gc` reconciler drops records a day past their
expiry, in batches of 5,000.

Revocation also happens without a call, the same way (the serial joins the node's KRL through its next NodeSpec):

| Trigger                                                                                                               | Revokes                                                                                                                                                                                                         | Where                                                 |
| --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| A Better IAM role activation ends (`binding:deactivate`)                                                              | The holder's live certificates in every tenant (an inherited binding reaches child tenants). A revocation that does not name its holder in `metadata.identityId` revokes nothing and is left to the re-decision | `serve`'s default IAM `binding` handler               |
| The identity is offboarded, expires or is deleted (or the `IamReconcile` backstop finds it gone)                      | Every certificate it holds, in every tenant, before its streams close; an update such as a rename keeps them                                                                                                    | `serve`'s default `identity` handler                  |
| The identity's sessions are revoked (`auth:session:revoke`, `auth:session:revoke-others`, `identity:revoke-sessions`) | Every certificate it holds, since the event does not say which session ended; issue a new one from a live session                                                                                               | `serve`'s default `sessions` handler                  |
| A re-decision says no                                                                                                 | That certificate, with reason `access-withdrawn`                                                                                                                                                                | The `tunnels.ssh-certificates-reauthorize` reconciler |

On a departure, `serve`'s host runs these revocations first (b3f7a81, review 7 M3): the identity handler revokes the
SSH certificates and closes the tunnels, and the agent handler drops its cached authentications, before any module
reaction (the vps sandbox reactions) runs in the same inbox transaction. Each module reaction then runs under its own
savepoint (`host_iam_reaction`), so one that throws is undone alone and cannot roll the revocation back. The identity
handler reports such a failure through `host.iam.onError` and the `IamReconcile` sweep re-runs departed identities;
the agent handler rethrows after undoing the reaction, so the inbox retries the event without undoing the
revocations (a3be14c), since an agent stays active in IAM and the sweep would never re-run it.

The reconciler runs every minute and re-decides the 500 longest-unchecked live certificates: each one as
`compute:nodes:ssh-root` on its node for the session that issued it (`ssh_certificates.session_id`), through
`IamPort.authorizeSession`. An allow stamps `reauthorized_at`. This catches what Better IAM does not announce as an
event: a deleted binding, an expired role activation, a changed policy. A certificate recorded without a session
id, or an IAM port without `authorizeSession`, is not re-decided. Tenant-wide session revocations name no identity
and are left to this re-decision. Each revocation publishes `node.ssh-cert.revoked` in the certificate's own tenant.
See [IAM events](/docs/guides/iam#iam-events-compute-reacts-to).

`authorizeSession` answers `false` for everything about the holder (session gone, identity gone or inactive, the
decision says no), and that revokes. It counts a `NOT_FOUND` from Better IAM as "gone" only when it is Better IAM's
own `Identity not found`, `Tenant not found` or `Resource not found`; a bare 404 (a proxy's error page, or `Endpoint
not found` on a wrong base path) is IAM trouble and throws, so an IAM incident never revokes every certificate or
closes every tunnel. It throws only when compute's own read fails, so the reconciler treats only two
error codes as a denial: `NOT_FOUND` and `TENANT_SUSPENDED`. A thrown `ACCESS_DENIED` or `UNAUTHENTICATED` means
compute's own credential was refused, not the holder: like any other error, it keeps the certificate, stamps
`reauthorized_at` and logs `Re-authorizing an SSH certificate failed; it stays valid until it expires`. Otherwise an
expired service token would revoke every break-glass certificate within a minute, mid-outage. A certificate whose
tenant `compute_tenant_states` marks suspended or deleted is revoked without asking (reason `tenant-suspended` or
`tenant-deleted`).

> **IAM down keeps certificates valid.** 
  When the re-decision cannot reach Better IAM, or IAM refuses compute's own credential, the certificate stays valid
  until it expires, at most 1 hour after issue, and the reconciler logs a warning. This is deliberate: break-glass
  access exists for outages, and IAM may be part of one.

## Who has access [#who-has-access]

`nodes.listSshCerts` answers who holds, or held, break-glass access to a node: the certificates issued for it
through the tenant, newest first. It is decided as `compute:nodes:read` on the node, not `ssh-root`, because it is
audit information that the node's `node.ssh-cert.issued` and `node.ssh-cert.revoked` events already show to the same
readers (see [the reference](/docs/reference/api/nodes#listsshcerts)).

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant that issued the certificates.', required: true },
  nodeId: { type: 'string', description: 'The node (node_...).', required: true },
  includeExpired: {
    type: 'boolean',
    description: 'Also certificates past their expiry. Records are kept a day after expiry, then dropped.',
    default: 'false',
  },
  limit: { type: 'number', description: 'Page size, 1 to 500.', default: '100' },
  cursor: { type: 'string', description: 'The nextCursor of the previous page.' },
}"
/>

It answers `{ items, nextCursor? }`. Each item carries `serial`, `nodeId`, `identityId` (who asked for it),
`principal`, `keyId`, `issuedAt`, `expiresAt` and `state`: `active`, `expired` or `revoked`. A revoked item adds
`revokedAt`, `revokedBy` (an identity, or `iam` for a revocation driven by IAM) and `revokeReason`. The list never
returns the certificate or any key. Without `includeExpired`, revoked certificates that have not yet expired are
still listed.

Pages are keyed on `(issuedAt, serial)`, so a page boundary never drops or repeats a certificate. A node the tenant
cannot see is `NOT_FOUND`; a malformed cursor is `INVALID_INPUT`. Like `issueSshCert`, the method is added to the
`nodes` group at run time, so call it by path:

```bash
better-compute api nodes.listSshCerts tenantId=root nodeId=node_01jzq3v0m6y8x7w5t4s3r2p1na includeExpired:=true
```

## What the node needs [#what-the-node-needs]

The node must trust the CA for user certificates, accept only its own principals and honour the revocation list.
The control plane sends all three in NodeSpec `access` (`sshCaPublicKeys`, `sshPrincipals`, `sshRevokedSerials`),
and bcd's `access` converger (`daemon/internal/agent/access.go`) turns them into sshd configuration:

| File                                            | Content                                                                     |
| ----------------------------------------------- | --------------------------------------------------------------------------- |
| `/etc/ssh/sshd_config.d/10-better-compute.conf` | The drop-in below                                                           |
| `/etc/ssh/bc_ca.pub`                            | The CA public keys, one per line                                            |
| `/etc/ssh/bc_principals/<account>`              | This node's principals for that account, `<account>@<nodeId>`, one per line |
| `/etc/ssh/bc_krl`                               | An OpenSSH KRL revoking `sshRevokedSerials` under every CA key              |

```text title="/etc/ssh/sshd_config.d/10-better-compute.conf"
# Written by bcd from NodeSpec access (break-glass SSH certificates). Do not edit.
TrustedUserCAKeys /etc/ssh/bc_ca.pub
AuthorizedPrincipalsFile /etc/ssh/bc_principals/%u
RevokedKeys /etc/ssh/bc_krl
```

* **Only this node's principals.** A principal for another node, or one whose account is empty or contains `/`,
  `\`, `.` or whitespace, is never written, so a certificate for one node cannot open another that trusts the same CA.
  Without an `AuthorizedPrincipalsFile`, sshd would accept only a principal equal to the account name, which these
  certificates never carry.
* **A native KRL.** bcd encodes the KRL itself (OpenSSH's `PROTOCOL.krl`, a certificate-serial section per CA key,
  `krl_version` set to the spec generation). With no revoked serials it is a valid empty KRL. The Go tests check the
  file with `ssh-keygen -Q` when it is installed.
* **Reloads only when needed.** sshd reads the CA, principals and KRL files at each login, so a new revocation takes
  effect without a reload. bcd runs `systemctl reload-or-restart` on `ssh`, then `sshd`, only when the drop-in
  itself is written or removed.
* **Empty removes everything.** An empty or absent `access` section removes every file bcd wrote, so the node
  stops trusting the CA when the control plane stops sending it. Stale principal files are removed on every pass.

The FakeDaemon has no such converger and still reports a non-empty `access` section as `SectionUnsupported`. On a
node without bcd, configure sshd the same way by hand with the `sshCaPublicKey` that `better-compute pki show`
prints, and build the KRL with `ssh-keygen -k`. Logging in as `root` also needs `PermitRootLogin` to allow key
logins; its default, `prohibit-password`, does.

## In the console [#in-the-console]

A platform administrator sees a **Break-glass SSH** panel on each node's page. It takes your public key, the
account (default `root`) and a lifetime of 5, 15, 30 or 60 minutes, and returns the certificate with a copy button
and the commands to save and use it. A pasted private key is refused in the browser before anything is sent. When
the session lacks MFA, it asks you to sign in again with a second factor instead of showing the error.

Under **Who holds access** the panel lists the node's certificates from `nodes.listSshCerts` (holder, principal,
issued, until, state and revocation reason, with **Include expired**), and every active one has a **Revoke** button
that calls `nodes.revokeSshCert` after a confirmation. See [the console](/docs/guides/clients/console#break-glass-ssh).

## Sandbox certificates are separate [#sandbox-certificates-are-separate]

[Sandboxes](/docs/guides/vps/sandboxes#run-code-inside) use SSH certificates too, from a second CA: `tunnels.sandboxSsh`
(the PKI bundle's sandbox SSH CA), never `tunnels.ssh`. The differences:

|                   | Break-glass                                                              | Sandbox exec                                                                                       |
| ----------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| Method and action | `nodes.issueSshCert`, `compute:nodes:ssh-root` on the node               | `sandboxes.exec`, `compute:sandboxes:exec` on the sandbox                                          |
| Principal         | `<account>@<nodeId>`                                                     | `<account>@<instanceId>`, account default `sandbox` (a memory restore keeps its source instance's) |
| Lifetime          | 60 s to 1 h, default 15 min                                              | 60 s to 15 min, default 10 min, never past the sandbox's expiry                                    |
| Session           | MFA by default                                                           | a signed-in session, not an API key; no MFA requirement                                            |
| Who trusts the CA | nodes, through NodeSpec `access` and bcd                                 | sandbox guests, through the user data vps renders                                                  |
| Revocation        | recorded in `ssh_certificates`, KRL on the node, re-decided every minute | none in the guest: the short lifetime, plus tunnel re-authorization                                |
| Path              | your own network to the node's sshd                                      | always through a tunnel                                                                            |

Because the two CAs differ, a sandbox certificate never opens a node and a break-glass certificate never opens a
sandbox. The tunnels module refuses a CA public key that is not one printable line, since it is written into sshd
configuration, and refuses a `tunnels.sandboxSsh` that is the same key as `tunnels.ssh` (`INVALID_CONFIG`), so the
split cannot be configured away.

## Shipped and planned [#shipped-and-planned]

| Behavior                                                                                                                                   | State                                                                                                |
| ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| `nodes.issueSshCert` with MFA, TTL at most 1 h and never past the session, node-bound principal, `permit-pty` only, recorded, rate-limited | Shipped when `tunnels.ssh` is set                                                                    |
| The CA key, principals and revoked serials in NodeSpec `access`                                                                            | Shipped                                                                                              |
| Nodes applying them automatically                                                                                                          | In bcd's `access` converger (built and tested, bcd not released yet); not in the FakeDaemon          |
| `nodes.revokeSshCert` and the node's KRL (`access.sshRevokedSerials`)                                                                      | Shipped in the control plane; bcd writes the KRL that sshd enforces                                  |
| Revoking certificates on `binding:deactivate`                                                                                              | Shipped as `serve`'s default IAM `binding` handler                                                   |
| Revoking certificates on offboarding, expiry, deletion and session revocation                                                              | Shipped as `serve`'s default `identity` and `sessions` handlers                                      |
| Re-deciding live certificates every minute (`tunnels.ssh-certificates-reauthorize`)                                                        | Shipped; IAM unreachable, or refusing compute's own credential, keeps a certificate until it expires |
| `ssh-cert --connect HOST [-- command]`: issue and connect in one step                                                                      | Shipped in the CLI                                                                                   |
| `nodes.listSshCerts`: who holds or held a certificate for a node, with its state                                                           | Shipped, and in the console's node page with a revoke button per certificate                         |
| Rotating the SSH CA                                                                                                                        | No rotation flow; `access.sshCaPublicKeys` lists only the current key                                |


# Tunnels (/docs/guides/access/tunnels)

> Open exec, console, SSH and TCP tunnels with tunnels.open, single-use 60-second exec tickets and a WebSocket relayed through the node's gateway.



A tunnel is a two-way byte stream between a caller and something on a node: a shell in an
instance, an instance's serial console, the console of a node, an instance's SSH server, or one TCP port. It rides the connection the node already
holds to its gateway, so the node needs no inbound port, no VPN and no bastion host. The
API decides who may open one, and every hop checks a short-lived ticket before any byte flows.

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant C as Caller
  participant A as api
  participant I as Better IAM
  participant G as Gateway
  participant D as bcd
  C->>A: POST tunnels/open
  A->>I: require compute:instances:exec
  A-->>C: ticket (60 s, single use) and path
  C->>A: GET tunnels/ticket as a WebSocket
  A->>A: verify the ticket, record its jti
  A->>G: POST /v1/tunnels over internal mTLS
  G->>G: verify the ticket, record its jti
  G->>D: OpenTunnel on the node's Session
  D->>D: verify against the pinned JWKS and its jti set
  D->>G: Tunnel stream, first frame opened
  G-->>C: frames both ways, relayed by the api
  loop every 60 s
    A->>I: re-authorize the ticket's identity
  end"
/>

> **Node consoles work in bcd; instance tunnels need a VM runtime.** 
  Every half is implemented and tested, including the daemon's (steps 8 and 9) in the Go `bcd`, which is built but
  not released. `tests/e2e/bcd.test.ts` opens a node console through the api's relay, the gateway and a real bcd, and
  the shell's exit status comes back. bcd dials instance `ssh` and `tcp` tunnels to the instance's own address, and
  only a host that runs an instance runtime advertises `tunnel.instance-dial`: a host build with `virsh` (the
  `libvirt-kvm` runtime, 362317c) or with the `cloud-hypervisor` binary (2598f0a), both so far tested against fake
  tools only, never on a real KVM host, a fakehw
  build, or a host with a bcx runtime plugin (none is written). On a host without one, `tunnels.open` refuses
  instance `ssh`, `tcp` and `console` tunnels outright. `exec` into an instance closes at once with exit status 127
  on every host until guests have an agent, so an instance's serial console (op `console`, 798e710) is the terminal
  that works on a real host. See
  [the node half](#the-node-half). The TypeScript FakeDaemon answers every op with an echo shell, so tunnels also
  open end to end in the tests and in `better-compute dev --sim N`.

## Turn tunnels on [#turn-tunnels-on]

Tunnels are off until you give `betterCompute` an exec ticket issuer. With one, the server registers
`tunnels.open` (and `nodes.issueSshCert`, see [break-glass SSH](/docs/guides/access/ssh)) and puts the issuer's JWKS
into every node's NodeSpec as `access.execTicketJwks`.

```ts title="compute.ts"
import { loadPkiBundle } from 'better-compute/cli';
import { execTicketIssuer } from 'better-compute/pki';
import { betterCompute } from 'better-compute/server';

// `better-compute pki init` puts an exec ticket key and the SSH CA into the bundle.
const bundle = await loadPkiBundle('.better-compute/pki.json', process.env);

export const compute = betterCompute({
  // database, secret, iam, modules, pki ...
  baseURL: 'https://compute.example.com',
  tunnels: {
    // `iss` must be the deployment's public origin: `serve` and its gateways check it.
    issuer: execTicketIssuer(bundle, { issuer: 'https://compute.example.com' }),
    ssh: bundle.sshCa,
  },
});
```

<TypeTable
  type="{
  issuer: {
    type: 'ExecTicketIssuer',
    description: 'From execTicketIssuer(bundle, { issuer }) or createExecTicketIssuer({ issuer, privateKey }). A tunnels object without one throws INVALID_CONFIG.',
    required: true,
  },
  ssh: {
    type: 'SshCertificateAuthority',
    description: 'The compute SSH CA for nodes.issueSshCert. Without it that method answers UNSUPPORTED.',
  },
  sshRequiresMfa: {
    type: 'boolean',
    description: 'Refuse nodes.issueSshCert for sessions without MFA.',
    default: 'true',
  },
}"
/>

Every api replica and every gateway must use the same ticket key. A key's `kid` is its RFC 7638 thumbprint, so
processes holding the same key name it the same way and a node's pinned JWKS matches any replica. Without
`privateKey`, `createExecTicketIssuer` generates a key per process, which only works for a single process.

<TypeTable
  type="{
  issuer: { type: 'string', description: 'The iss claim. Use the origin of baseURL.', required: true },
  privateKey: {
    type: 'KeyObject | Uint8Array | string',
    description: 'A P-256 private key (PEM or DER). Shared by every replica.',
    default: 'a new key',
  },
  keepPrevious: {
    type: 'number',
    description: 'How many earlier keys rotate() keeps in the JWKS.',
    default: '1',
  },
  clock: { type: 'Clock', description: 'The clock iat and exp come from.', default: 'systemClock' },
}"
/>

### With better-compute serve [#with-better-compute-serve]

`better-compute serve` wires both roles from `tunnels.issuer`:

* The **api role** mounts `GET {basePath}/tunnels/{ticket}`, verifies tickets against the issuer's JWKS with
  `iss` equal to the public origin, relays through `gatewayTunnelRelay`, and re-authorizes callers with
  `tunnelAuthorizer(iam)` from `@better-compute/iam`.
* The **gateway role** gets `tunnels: { jwks, issuer }` with the same JWKS and origin.

The optional `host.tunnels` block of the configuration's `host` export tunes the api side:

<TypeTable
  type="{
  authorize: {
    type: '(input) => Promise<boolean>',
    description: 'The re-check. Default tunnelAuthorizer(iam), which needs an IamPort with authorizeSession (the embedded bridge with sessionReviewer, or remote authorizeSessions).',
  },
  control: {
    type: '{ tls, timeoutMs }',
    description: &#x22;Client TLS from the internal CA for the gateways' internal control APIs.&#x22;,
  },
  reauthorizeMs: { type: 'number', description: 'How often the caller is re-checked.', default: '60000' },
  maxDurationMs: { type: 'number', description: 'Hard cap on a tunnel.', default: '43200000 (12 h)' },
  onClose: {
    type: '(event) => void',
    description: 'Called with { tunnelId, nodeId, subject, reason } when a tunnel ends, for your audit trail.',
  },
}"
/>

`better-compute doctor` reports `host.tunnels` when no re-authorizer can be built, and `host.gateway.tunnels` when
tunnels are configured but the gateway has no internal control API (`host.gateway.internal`) to open them on. See
[deployment](/docs/operations/deployment) and [PKI](/docs/operations/pki).

### With your own HTTP server [#with-your-own-http-server]

`handleTunnel` answers an `upgrade` request. It returns `false` for paths that are not tunnel paths, answers every
failure on the socket itself, and never rejects. The console's `server.mjs` mounts it this way:

```ts title="server.ts"
import { createServer } from 'node:http';
import { gatewayTunnelRelay } from 'better-compute/gateway';
import { tunnelAuthorizer } from 'better-compute/iam';
import { handleTunnel } from 'better-compute/server';

const tunnels = {
  sql: compute.context.sql,
  issuer, // the same ExecTicketIssuer passed to betterCompute({ tunnels })
  issuerName: 'https://compute.example.com',
  relay: gatewayTunnelRelay({ sql: compute.context.sql, tls: internalClientTls }),
  authorize: tunnelAuthorizer(compute.context.iam),
};

const server = createServer(compute.nodeHandler);
server.on('upgrade', (req, socket, head) => {
  void handleTunnel(req, socket, head, tunnels).then((handled) => {
    if (!handled) socket.destroy();
  });
});
```

`handleTunnel` also takes `basePath` (default `/api/compute`), `jtiCache`, `reauthorizeMs`, `authorizeTimeoutMs`
(10 s), `iamOutageMs` (2 min), `maxDurationMs` (12 h), `maxMessageBytes` (1 MiB), `pingIntervalMs` (30 s) and
`onClose`.

## Open a tunnel [#open-a-tunnel]

`tunnels.open` is decided as `compute:instances:exec` on the instance, or as `compute:nodes:console` on the node
when the input names a node. See [the reference](/docs/reference/api/tunnels#open).

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant acting.', required: true },
  instanceId: { type: 'string', description: 'An instance (inst_...). Give exactly one of instanceId and nodeId.' },
  nodeId: { type: 'string', description: 'A node (node_...), for its console.' },
  op: {
    type: &#x22;'exec' | 'console' | 'ssh' | 'tcp'&#x22;,
    description: &#x22;A node takes console only (a root shell on the host). An instance takes exec, ssh, tcp or console (its serial console, 798e710).&#x22;,
    required: true,
  },
  port: { type: 'number', description: 'For tcp only, and required there: 1 to 65535.' },
}"
/>

It refuses what cannot be tunneled:

| Situation                                                                                                                                                                       | Error                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| Both or neither of `instanceId` and `nodeId`, a wrong op for the target, a `port` without `tcp`                                                                                 | `INVALID_INPUT`                                                           |
| An unknown instance, a draft, another tenant's instance, or a node the tenant cannot see                                                                                        | `NOT_FOUND`                                                               |
| A provider instance, which has no node agent to tunnel through                                                                                                                  | `UNSUPPORTED`                                                             |
| An instance that is not `running`, or a node that is not connected to a gateway                                                                                                 | `INVALID_STATE`                                                           |
| A [sandbox's](/docs/guides/vps/sandboxes#run-code-inside) instance or a warm sandbox instance, for any op: use `sandboxes.exec`                                                 | `INVALID_STATE` with `details.reason: 'sandbox-managed'`                  |
| An instance `ssh`, `tcp` or `console` tunnel whose node's current session does not advertise the `tunnel.instance-dial` capability (`node_sessions.capabilities`, from `Hello`) | `UNSUPPORTED` with `details: { reason: 'daemon-capability', capability }` |

The capability check exists because a daemon that cannot dial the instance's own address would reach the host's
loopback services instead. The daemons that advertise it are also the ones that serve an instance `console` from the
runtime; an older daemon refuses every instance ticket, and none answers one with a host shell. `exec` and node
consoles do not need it.

An instance's serial console is decided like the instance's other ops, as `compute:instances:exec` on the instance
(never `compute:nodes:console`), and its ticket carries the instance (`inst`), so bcd attaches to that domain's serial
port and never starts a host shell:

```bash
better-compute api tunnels.open tenantId=acme instanceId=inst_01jzq3v0m6y8x7w5t4s3r2p1ia op=console
```

The result carries the ticket and where to use it:

```json title="tunnels.open result"
{
  "tunnelId": "tun_01jzq3v0m6y8x7w5t4s3r2p1n0",
  "nodeId": "node_01jzq3v0m6y8x7w5t4s3r2p1na",
  "instanceId": "inst_01jzq3v0m6y8x7w5t4s3r2p1ia",
  "op": "exec",
  "ticket": "eyJhbGciOiJFUzI1NiIsInR5cCI6ImJjLWV4ZWMrand0Ii...",
  "expiresAt": 1790251260000,
  "path": "tunnels/eyJhbGciOiJFUzI1NiIsInR5cCI6ImJjLWV4ZWMrand0Ii..."
}
```

The server adds the `tunnels` group at run time, so the typed client does not list it. Call it by path, over HTTP,
or from the CLI:

  **TypeScript:**

    ```ts
    const opened = await client.$request<{ tunnelId: string; path: string }>('tunnels/open', {
      tenantId: 'acme',
      instanceId: 'inst_01jzq3v0m6y8x7w5t4s3r2p1ia',
      op: 'exec',
    });
    ```
  
  **HTTP:**

    ```bash
    curl -s https://compute.example.com/api/compute/tunnels/open \
      -H "authorization: Bearer $BETTER_COMPUTE_TOKEN" \
      -H 'content-type: application/json' -H 'x-better-compute: 1' \
      -d '{"tenantId":"acme","instanceId":"inst_01jzq3v0m6y8x7w5t4s3r2p1ia","op":"exec"}'
    ```
  
  **CLI:**

    ```bash
    better-compute api tunnels.open tenantId=acme instanceId=inst_01jzq3v0m6y8x7w5t4s3r2p1ia op=exec
    ```

    The CLI prints the ticket; it does not open the WebSocket.
  
Each call publishes `tunnel.issued` (subject `instance/<id>` or `node/<id>`) with the tunnel id, node, op,
identity, expiry and port, and writes an audit row like every mutation.

## Exec tickets [#exec-tickets]

An exec ticket is a compact ES256 JWT with header `typ: bc-exec+jwt`, so no other JWT
(a session JWT, a workload token) passes as one.

| Claim                 | Value                                                    |
| --------------------- | -------------------------------------------------------- |
| `iss`                 | The issuer name, the deployment's origin                 |
| `aud`                 | `node:<nodeId>`: the only daemon that may accept it      |
| `sub`                 | The caller's Better IAM identity id                      |
| `sid`                 | The tunnel id (`tun_...`), also `tunnelId` in the result |
| `op`                  | `exec`, `console`, `ssh` or `tcp`                        |
| `jti`                 | A single-use id (`tkt_...`)                              |
| `iat`, `exp`          | Seconds; at most 60 apart                                |
| `tnt`, `inst`, `port` | Tenant, instance and TCP port, when they apply           |
| `ses`                 | The caller's IAM session, for re-authorization           |
| `act`                 | The action `tunnels.open` decided, re-decided as is      |

`verifyExecTicket` is the check the api, the gateway and the daemon share. It refuses with `ACCESS_DENIED` and a
`details.reason`: `malformed`, `algorithm must be ES256`, `not an exec ticket`, `unknown key`, `bad signature`,
`ticket for another node`, `foreign issuer`, `lifetime too long`, `issued in the future`, `expired`,
`op not allowed` or `replayed`. Clock tolerance is 5 seconds, and a ticket id is recorded only after every other
check passed.

`issuer.rotate()` starts signing with a new key and keeps the previous `keepPrevious` keys in the JWKS, so tickets
and NodeSpecs in flight stay valid across a rotation. The ticket travels in the URL path, but it is single use and
expires within a minute, so one that shows up in a proxy log is already spent.

## Connect [#connect]

Open `wss://<host>{basePath}/tunnels/<ticket>` within 60 seconds. `tunnelUrl` builds it from the base URL and the
`path` in the result, and `connectTunnel` speaks the frames, in browsers and in Node.js 22 and later:

```ts title="exec.ts"
import { connectTunnel, tunnelUrl } from 'better-compute/client';

const tunnel = connectTunnel({
  url: tunnelUrl('https://compute.example.com', '/api/compute', opened.path),
});
tunnel.onData((data, channel) => (channel === 1 ? process.stderr : process.stdout).write(data));
await tunnel.ready; // the node opened its end
tunnel.resize(40, 120);
tunnel.write('nvidia-smi\n');
tunnel.close(); // the node reports its exit status, then the socket closes
const { exitCode, reason } = await tunnel.exited;
```

Each binary WebSocket message is one protobuf `bettercompute.agent.v1.TunnelFrame`. Its body is `opened` (the
node's first frame), `data` (channel 0 is stdout or the byte stream, 1 is stderr), `resize` (rows and columns) or
`closed` (exit code and reason). The server stamps the ticket's tunnel id on everything the caller sends.

The upgrade is refused with a plain HTTP answer before any WebSocket exists:

| Status | Code                 | When                                                       |
| ------ | -------------------- | ---------------------------------------------------------- |
| 426    | `METHOD_NOT_ALLOWED` | Not a WebSocket version 13 upgrade                         |
| 400    | `INVALID_INPUT`      | The ticket in the path cannot be decoded                   |
| 403    | `ACCESS_DENIED`      | The ticket fails verification (the reason is not revealed) |
| 409    | `INVALID_STATE`      | The node has no session on any gateway                     |
| 502    | The gateway's code   | The gateway refused or could not open the tunnel           |
| 500    | `INTERNAL_ERROR`     | Anything unexpected                                        |

Once open, the socket closes with one of these:

| Close code | Reason                     | Why                                              |
| ---------- | -------------------------- | ------------------------------------------------ |
| 1000       | `closed`, `caller-closed`  | The node ended the stream, or the caller hung up |
| 1003       | `binary TunnelFrames only` | The caller sent a text message                   |
| 1007       | `not a TunnelFrame`        | A binary message that does not decode            |
| 1008       | `revoked`                  | A re-check answered no                           |
| 1008       | `max-duration`             | 12 hours passed                                  |
| 1009       | `message too large`        | A message over `maxMessageBytes` (1 MiB)         |
| 1011       | `iam-unreachable`          | No successful re-check for 2 minutes             |
| 1011       | `relay-error`              | The gateway stream failed                        |

The server pings every 30 seconds and drops a peer that has been silent for two periods. Backpressure runs end to
end: a full socket on one side pauses reading on the other.

### From a terminal [#from-a-terminal]

The CLI does all of the above for you. `better-compute console --node ID` (op `console` on a node: a root shell),
`better-compute console --instance ID` (op `console` on an instance: its serial console; 798e710) and
`better-compute exec --instance ID` (op `exec`) call `tunnels.open` as the token, open the WebSocket on the server
named by `--url`, and attach your terminal in raw mode until the remote program exits. `console` takes exactly one of
`--node` and `--instance` (a usage error otherwise):

```bash
better-compute console --url https://compute.example.com --tenant acme --node node_01k5w…
better-compute console --url https://compute.example.com --tenant acme --instance inst_01k5w…
```

Every key goes to the remote, Ctrl-C included; Ctrl-] detaches. The window size is sent at start and on every
resize. The command exits with the remote program's status, and with 255 when the tunnel breaks, like ssh. Both
need `--url`: without it they fail with `UNSUPPORTED` before spending a ticket. See the
[CLI guide](/docs/guides/clients/cli#interactive-access).

## Re-authorization and revocation [#re-authorization-and-revocation]

While a tunnel is open, the api re-checks the caller every 60 seconds, with a 10-second timeout per check.

* **A no** closes the tunnel with `revoked`.
* **An error or a timeout** counts as IAM unreachable; once 2 minutes pass without a successful check, the tunnel
  closes with `iam-unreachable`, even while a check still hangs.
* **12 hours** is the hard maximum, whatever the answers.

How the check is made is described in [access](/docs/guides/access): with the upgrade request's credential when it
carried one, otherwise by the identity and session recorded in the ticket. A delegated or agent session cannot be
re-decided that way and closes at its first re-check.

The gateway keeps its own registry of the tunnels it relays and can close them without the api:

* revoking a node's identity closes every tunnel to that node (reason `revoked`);
* `POST /v1/tunnels/close` on its internal control API, with `{ tunnelId?, nodeId?, subject?, agentId?,
  delegationId?, sandboxId?, tenantId?, anyAgent?, reason }`, closes the matching tunnels and answers `{ closed }`.
  The filter must narrow: one of the id keys, or `tenantId` with `anyAgent: true`; a tenant alone is refused.
  `sandboxId` matches the tunnels a `sandboxes.exec` ticket opened into that sandbox.
  `gatewayControlClient(...).closeTunnels(filter, reason)` calls it, and `POST /v1/tunnels/reauthorize` takes the
  same filter;
* a gateway that stops closes all of its tunnels (`gateway-stopped`).

### Closing on IAM revocation [#closing-on-iam-revocation]

IAM revocations close tunnels at once, without waiting for the 60-second re-check. The IAM bridge's inbox routes
`delegation:revoke`, `agent:suspend` and `agent:suspend-all` to its `agent` handler, and
`agentStreamCloser(gatewayStreamCloser(() => gatewayClients))` from `better-compute/iam` turns each into one close
request sent to every gateway's control API:

| Event               | Tunnels closed                                                             |
| ------------------- | -------------------------------------------------------------------------- |
| `delegation:revoke` | Those opened under the delegation (and its agent, when the event names it) |
| `agent:suspend`     | Those the agent opened                                                     |
| `agent:suspend-all` | Every agent's tunnels in the tenant                                        |

The request goes out as soon as the `iam.inbox` job (every 2 seconds) drains the event, without waiting for the inbox
transaction, and a gateway that is away never fails the drain: close failures go to `onError`. When a revoked
delegation had live hand-offs, IAM revokes them without naming them, so the tenant's agent tunnels are re-authorized
at once (`POST /v1/tunnels/reauthorize`) instead of at their next check. The 60-second re-check stays the backstop.
`tests/e2e/access.test.ts` holds the v0.3 exit: on a real embedded Better IAM, `delegations.revoke` closes the agent's
tunnel in under 5 seconds with the revocation's reason, and the administrator's own console stays open.

In `serve`'s host these closes, and the SSH-certificate revocation of a departed identity, run before any module's IAM
reaction (b3f7a81): each module reaction runs under its own savepoint, so a reaction that fails is undone alone and
never takes the revocation with it (see [SSH](/docs/guides/access/ssh)).

`better-compute serve` wires this for you with `host.iam`:

```ts title="better-compute.config.ts"
export const host = defineHost(async () => ({
  iam: { bridge }, // the bridge whose modules and port the configuration uses
}));
```

`installIamHandlers` then installs the default reactions before the workers start: `agent` closes the agent's
tunnels on every gateway that heartbeated in the last 2 minutes (`gatewayStaleMs`), over its `internal_url` with the
TLS of `host.tunnels.control` (h2c only without TLS), and drops the agent's cached authentications. Better IAM
names the identity, not the session, in its session events, so:

* an offboarded, expired or deleted identity closes its tunnels on every gateway and its streams in this process's
  `host.iam.streams` (such as `GET /watch` streams);
* `auth:session:revoke`, `auth:session:revoke-others` and `identity:revoke-sessions` close the identity's local
  streams (clients reconnect and are re-authorized) and make every gateway re-authorize the identity's tunnels at
  once: the revoked sessions' shells close, the others keep running;
* a `binding:deactivate` (a role activation ended) makes every gateway re-check the holder's tunnels, and revokes
  the holder's break-glass SSH certificates ([SSH](/docs/guides/access/ssh#revoke-a-certificate));
* role- and tenant-wide revocations are left to the periodic re-check. Your own `host.iam.handlers` replace the
  default of the same name. `better-compute doctor` warns `host.iam` when the IAM module is registered without it. See
  [delegated confirmation](/docs/guides/iam/delegated-confirmation).

## The gateway half [#the-gateway-half]

The gateway verifies every ticket itself before it asks the node. `createGateway({ tunnels })` takes `jwks` (a
function returning the issuer's JWKS, read on every open), `issuer`, `jtiCache`, an optional `reauthorize`,
`reauthorizeMs` (60 s), `iamOutageMs` (2 min), `maxDurationMs` (12 h) and `openTimeoutMs` (10 s, how long the node
has to open its `Tunnel` stream). Opening fails with `ACCESS_DENIED` for a bad, foreign or replayed ticket,
`NOT_FOUND` when the node has no session on that gateway, `CONFLICT` when the tunnel id is already open, and
`INVALID_STATE` when the node did not open in time, or at once when the node is connected over the WebSocket
fallback, which carries Session frames only. A `Tunnel` stream from a node that no gateway asked for, or one
that answers for another node's tunnel, is refused.

`serve`'s gateway role wires `reauthorize` itself when the IAM port can re-authorize sessions: every 60 seconds, and
on `POST /v1/tunnels/reauthorize`, it decides the ticket's action (`compute:instances:exec` or
`compute:nodes:console`) on its instance or node again for the ticket's identity and session, the same decision
the api side makes. A `sandboxes.exec` ticket carries the sandbox as `sbx`, which the gateway keeps as
`TunnelInfo.sandboxId` and passes back, so its tunnel is re-decided as `compute:sandboxes:exec` on the sandbox, never
on the instance it drives. A ticket without a session, with an unexpected action, or naming
`compute:sandboxes:exec` without a sandbox, fails closed.

On the api side, `gatewayTunnelRelay` looks up the gateway holding the node's session (`node_sessions.gateway_id`,
then `gateways.internal_url`, cached 30 s) and calls that gateway's `gatewayControlClient(...).openTunnel`, an
HTTP/2 stream (`POST /v1/tunnels`) on the internal mTLS control API. An unregistered
gateway, or one that takes longer than 10 seconds, is `UNAVAILABLE`. When the caller hangs up, the node gets
5 seconds to report its exit status. See [the gateway](/docs/daemon/gateway) and
[the protocol](/docs/daemon/protocol).

## The node half [#the-node-half]

bcd answers `OpenTunnel` itself and trusts nothing the gateway says about the ticket
(`daemon/internal/agent/tunnels.go`, `daemon/internal/pki/tickets.go`):

1. `pki.VerifyExecTicket`, the Go port of `verifyExecTicket`, checks the header `typ` `bc-exec+jwt`, ES256 under the
   key with the ticket's `kid` in the JWKS pinned by its NodeSpec (`access.exec_ticket_jwks_json`, kept by the
   [access converger](/docs/guides/access/ssh#what-the-node-needs)), `aud` `node:<its own id>`, a lifetime of at most
   60 s that is neither in the future nor expired (5 s leeway, against server time), a known op, and a `sid` and a
   `jti`. Tickets that `@better-compute/pki` issues are interop fixtures of the Go tests.
2. The `jti` goes into the journal (at most 10,000 live ids, expired ones pruned), so a restart does not forget a
   spent ticket. A known `jti`, or a full set, is refused as `replayed`.
3. The ticket's `sid` must equal the tunnel id in `OpenTunnel`, so a gateway cannot reuse a ticket for another
   stream.
4. bcd dials the `Tunnel` RPC, sends `opened` first, and starts the program for the op.

A refused ticket opens nothing: bcd logs `tunnel refused` with the reason, and the gateway's open times out after
10 seconds, exactly as if the node had not answered. A node without a pinned JWKS refuses every ticket, and at most
16 tunnels are open at once.

A ticket that names an instance (`inst`) must reach that instance and nothing of the host, whose loopback (its sshd,
admin ports, other tenants' forwards) is off limits to a tenant. So bcd runs different programs for node and instance
tickets. When a program cannot start, the tunnel opens and closes at once with exit status 127 and the reason.

| Op        | Node ticket                                                                                                                                                                                                                                                     | Instance ticket                                                                                                                                                                                                             |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `console` | A login shell (`/bin/bash -l`, or `/bin/sh -l`) as root in `/`, on a pseudo-terminal opened through `/dev/ptmx` without cgo. `resize` sets the window size, a hang-up sends `SIGHUP` to the shell's process group, and `closed` carries the shell's exit status | The instance's serial console, from its runtime, for a running, managed instance on this host (`libvirt-kvm` opens the serial port's pty, `cloud-hypervisor` dials the VM's serial socket; f8273ef); never the host's shell |
| `exec`    | Refused: `exec` is into an instance; a host shell is `console`, decided as `compute:nodes:console`                                                                                                                                                              | Refused: it needs a guest agent                                                                                                                                                                                             |
| `ssh`     | A TCP connection to the node's own sshd on `127.0.0.1:22`                                                                                                                                                                                                       | A TCP connection to port 22 of the instance's own address                                                                                                                                                                   |
| `tcp`     | A TCP connection to the ticket's `port` on `127.0.0.1`                                                                                                                                                                                                          | A TCP connection to the ticket's `port` on the instance's own address                                                                                                                                                       |

For an instance ticket bcd looks the instance up in its runtimes. It must be a running domain that bcd manages on
this host (a stopped instance or one on another host is refused), and bcd dials the first of its reported addresses
that is not loopback, link-local, multicast, unspecified or an address of the host itself, so an instance ticket
never reaches the host.

`tunnels.open` issues `ssh`, `tcp` and `exec` only for instances, and `console` for both: a node's root shell, or an
instance's serial console (798e710). The API guards instance `ssh`, `tcp` and `console` with a capability: the node's
current session must advertise `tunnel.instance-dial` in its `Hello` (stored in `node_sessions.capabilities`; see
[sessions](/docs/daemon/sessions#hello)), or `tunnels.open` answers `UNSUPPORTED` before any ticket is minted. bcd
advertises it only when it runs an instance runtime: `libvirt-kvm` on a host build with `virsh`, `cloud-hypervisor`
on one with the `cloud-hypervisor` binary, the file-backed
runtime of a fakehw build, or a bcx runtime plugin; on a host without one a node `console` is the only op that runs.
An instance `exec` needs no capability, and bcd refuses it as above, so the serial console is the way into an
instance on a real host. Sandbox and warm sandbox instances are refused for every op, `console` included
(`sandbox-managed`); `sandboxes.exec` stays their only way in.

## The web terminal [#the-web-terminal]

The console (`apps/console`) has a terminal on every instance (`/instances/<id>/terminal`, op `exec`), the
instance's serial console (`/instances/<id>/console`, op `console`, linked as **Serial console** beside **Open a
terminal** on the instance page; 191898b), both decided as `compute:instances:exec`, and a console on every
connected node (`/nodes/<id>/console`, decided as `compute:nodes:console`; the node page shows the link only while
the node is connected). Since bcd refuses instance `exec` until guests have an agent, the serial console page is the
instance terminal that works on real hosts. Keys, pastes and resizes go straight to the target; **Reconnect** asks
for a new ticket.

The pages use `createTerminalSession` from `@better-compute/console-kit`, which you can use in your own UI:

```ts title="terminal.ts"
import { createTerminalSession } from '@better-compute/console-kit';

const session = createTerminalSession(api, {
  tenantId: 'acme',
  instanceId: 'inst_01jzq3v0m6y8x7w5t4s3r2p1ia', // or nodeId for a node's console
  op: 'console', // the instance's serial console; or 'exec', 'ssh'
  baseURL: 'https://compute.example.com',
});
session.subscribe(() => render(session.get())); // status: idle | opening | open | closed | error
session.onOutput((data) => terminal.write(data));
await session.open(); // tunnels.open, then the WebSocket
session.write('ls\n');
```

The store keeps the last 256 KiB of output (`maxBufferedBytes`) and the exit status once the session ends.

The console serves the upgrade from `node server.mjs`, because Next.js route handlers cannot. It reads the ticket
key from `CONSOLE_EXEC_TICKET_KEY_FILE`, the SSH CA from `CONSOLE_SSH_CA_KEY_FILE`, and its mTLS identity toward
gateways from `CONSOLE_GATEWAY_TLS_CERT`, `_KEY` and `_CA`. In development it generates the two keys; in
production, without the key file, tunnels are off. See [the console](/docs/guides/clients/console).

## Limits and defaults [#limits-and-defaults]

| Setting                                     | Value                                                           |
| ------------------------------------------- | --------------------------------------------------------------- |
| Ticket lifetime                             | 60 s at most, single use at the api, the gateway and the daemon |
| Clock tolerance on tickets                  | 5 s                                                             |
| Re-authorization                            | Every 60 s, 10 s per check                                      |
| IAM outage before closing                   | 2 min                                                           |
| Tunnel lifetime                             | 12 h at most                                                    |
| WebSocket message                           | 1 MiB at most; ping every 30 s                                  |
| Node open timeout (gateway)                 | 10 s                                                            |
| Tunnels open at once on one bcd node        | 16                                                              |
| Spent ticket ids a bcd node remembers       | 10,000, until each expires                                      |
| Exit status grace after the caller hangs up | 5 s                                                             |
| TCP ports                                   | 1 to 65535, instances only                                      |


# BMC drivers and credentials (/docs/guides/bare-metal/bmc)

> The driver concerns metal adds, how BMC operations reach the site agent, enrolling with sealed BMC credentials, power control and two-phase BMC password rotation.



A BMC is the one part of a server that works when the server does not: it can
power the machine, pick its boot device, mount media and report hardware. Metal drives every machine
through its BMC, from a site agent on the site's management network,
because the control plane never dials into a site. This page covers the driver contract, the BMC
operations metal sends, and how BMC credentials are stored, delivered and rotated.

## Driver concerns [#driver-concerns]

Importing `@better-compute/metal` merges ten driver kinds into `DriverKinds` of
`@better-compute/core` by declaration merging, so `defineDriver({ implements: { power: … } })`
type-checks wherever the package is imported. Each kind is one Ironic-style
driver concern:

| Concern         | Operations                                                                                            | Built-in drivers arrive |
| --------------- | ----------------------------------------------------------------------------------------------------- | ----------------------- |
| `power`         | `getPower()`, `setPower(action)`                                                                      | v0.2                    |
| `management`    | `setBootDevice(device, { persistent, uefi })`, optional `rotateCredential(newPassword, { username })` | v0.2                    |
| `boot`          | `prepareRamdisk(spec)`, `prepareInstance(boot)`, `cleanUp()`                                          | v0.2                    |
| `deploy`        | `deploy(spec)`, `tearDown({ wipe })`, both streaming progress                                         | v0.2                    |
| `inspect`       | `inspect()`, returning a `NodeInventory`                                                              | v0.2                    |
| `virtual-media` | `insert(iso)`, `eject()`                                                                              | v0.2                    |
| `console`       | `open()`, a byte stream (Serial-over-LAN)                                                             | draft until v0.3        |
| `bios`          | `getSettings()`, `applySettings(settings)`                                                            | draft until v0.5        |
| `raid`          | `applyLayout(layout)`, `clear()`                                                                      | draft until v0.5        |
| `firmware`      | `components()`, `update(component, image)`                                                            | draft until v0.5        |

`metalDriverKinds` holds the same table as data. One driver may implement several concerns, which
is what Ironic calls a hardware type:

```ts title="drivers/redfish.ts"
import { defineDriver, field, manifest } from 'better-compute/core';
import 'better-compute/metal'; // merges the metal driver kinds into DriverKinds

export const redfish = defineDriver({
  id: 'redfish',
  version: '1.0.0',
  runsOn: 'site',
  config: manifest({ address: field.url(), credentials: field.secret() }),
  implements: {
    power: (config) => ({ getPower: async () => 'on', setPower: async (action) => {} }),
    management: (config) => ({ setBootDevice: async (device, options) => {} }),
    'virtual-media': (config) => ({ insert: async (iso) => {}, eject: async () => {} }),
  },
  retrySafe: { power: ['getPower', 'setPower'], management: ['setBootDevice'] },
});
```

The built-in drivers (Redfish, IPMI and vendor BMCs through bmclib, PXE, iPXE and UEFI HTTP boot,
image streaming and kexec) run inside `bcd site` and `bcd ramdisk`, the Go daemon's v0.2 modes.
Today `bcd site` has Redfish built in, with the operations in [Redfish in bcd site](#redfish-in-bcd-site), and runs
every other BMC driver as a bcx plugin it hosts (51806ab). The repository has one such plugin, `bcx-ipmi`, for
[IPMI](#ipmi-through-bcx-ipmi); vendor BMCs through bmclib are not built. An operation for a driver the agent does not
host answers `unsupported-driver`. Out-of-process drivers in other languages come with bcx and
`@better-compute/driver-sdk` (v0.5). See [drivers](/docs/extending/drivers).

## How BMC operations run [#how-bmc-operations-run]

Metal's BMC steps are control-plane steps: only the control plane knows the machine's BMC address
and credential reference, so each step builds a `bettercompute.site.v1.BmcOperationRequest` and sends
it to the site agent of the machine's site:

```json title="BmcOperationRequest (proto3 JSON)"
{
  "driver": "redfish",
  "address": "https://10.0.0.10",
  "credentialsRef": "secret://root/…",
  "operation": "power.setPower",
  "argsJson": "{\"action\":\"cycle\"}",
  "tlsSpkiSha256": "<base64, when the BMC certificate is pinned>"
}
```

A BMC enrolled with `bmc.driverOptions` also carries them in `driverOptionsJson` (site/v1 field 7, 51a3dfe), as
canonical JSON with sorted keys, so a batch leaf bound to the payload keeps the same bytes; the built-in Redfish driver
takes none and answers `invalid-config` if a request has any. The agent fetches the credentials by reference (below),
runs `<concern>.<operation>` on the driver
and answers a `BmcOperationResult` (`ok`, `resultJson`, `errorCode`, `message`). The command kinds
are all on the `control` lane and accepted only by agents with role `site`:

| Kind                  | Used by                                                                                                         | Retry-safe | Flags                                   | Timeout |
| --------------------- | --------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------- | ------- |
| `bmc.query`           | `power.getPower`, `inspect.oob`, firmware checks, reading BIOS settings and the RAID layout                     | yes        | none                                    | 2m      |
| `bmc.boot`            | `management.setBootDevice`, `media.insert`, `media.eject`, applying BIOS settings (pending until the next boot) | yes        | destructive                             | 2m      |
| `bmc.power`           | `power.setPower`, `metal.power`                                                                                 | no         | destructive                             | 5m      |
| `bmc.rotate`          | `metal.rotateBmcCredential`                                                                                     | no         | attested by the site agent, destructive | 5m      |
| `bmc.firmware`        | firmware reflash                                                                                                | no         | destructive                             | 1h      |
| `bmc.storage`         | applying or clearing a RAID layout (a v0.5 draft)                                                               | no         | destructive                             | 1h      |
| `boot.prepareRamdisk` | boot entries for a PXE boot                                                                                     | yes        | none                                    | 2m      |
| `boot.cleanUp`        | removing a node's boot entries                                                                                  | yes        | none                                    | 2m      |
| `site.httpProxy`      | one HTTP request to an agentless machine                                                                        | no         | destructive (0d42de8)                   | 1m      |

A power cycle is not idempotent, so `bmc.power` is never retried blindly. Once its trust bundle holds
command-signing roots, an agent refuses every command that lacks a valid worker signature over the command body,
whatever its kind, so a compromised gateway cannot forge one (0d42de8; before, only destructive kinds needed one).
The destructive flag decides what falls under the v1.0 batch quorum. `bmc.boot` is destructive because the boot
device, virtual media and BIOS settings decide what a machine boots (with any later power cycle, that is a reimage),
and `bmc.rotate` because a changed password locks the platform out (9c14146, 3ff8131). `site.httpProxy` is
destructive because a request can change the machine it reaches, and bcd never lets it reach a BMC: the site agent
remembers every BMC host it has driven with a `bmc.*` command (`data_dir/bmc-hosts.json`) and refuses those hosts,
by name and by resolved address, and any `/redfish` path (`forbidden-target`), so a BMC is reached only through the
`bmc.*` kinds and their quorum. See [security](/docs/operations/security).

## Redfish in bcd site [#redfish-in-bcd-site]

`bcd site` answers every `bmc.*` kind with its Redfish client (`daemon/internal/redfish`). For each command it
fetches the credential with a `SecretRequest` bound to the command's workflow step, reads it as the enrolled
`{ username, password }` JSON or as a bare password (with the step's `username` argument as the account; the same
fallback fills a JSON without a username), and runs the operation:

| Operation                                     | Redfish request                                                                                                                                                                                                                                                                                 | Result                                                                                 |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `power.getPower`                              | `GET` the address's system (the one it names, else the first `ComputerSystem`)                                                                                                                                                                                                                  | `{ power: 'on' \| 'off' \| 'unknown' }`                                                |
| `power.setPower`                              | `POST .../Actions/ComputerSystem.Reset` with the `ResetType` of [power](#power)                                                                                                                                                                                                                 | `{}`                                                                                   |
| `management.setBootDevice`                    | `PATCH` the system's `Boot`: `BootSourceOverrideTarget` `Pxe`, `UefiHttp`, `Hdd` or `Cd` for `pxe`, `http`, `disk` and `cd`; `Once`, or `Continuous` when `persistent`; `UEFI`, or `Legacy` when `uefi` is false                                                                                | `{}`                                                                                   |
| `virtual-media.insert`, `virtual-media.eject` | `InsertMedia` (an http(s) image URL) or `EjectMedia` on the first virtual media device that takes a CD or DVD, of the system's own manager (its `Links.ManagedBy`), else of the first manager                                                                                                   | `{}`                                                                                   |
| `inspect.inspect`                             | the system and its `EthernetInterfaces`                                                                                                                                                                                                                                                         | `macs`, `ipAddresses`, `manufacturer`, `model`, `serialNumber`, `productUuid`, `power` |
| `firmware.components`                         | `UpdateService/FirmwareInventory`                                                                                                                                                                                                                                                               | `components: [{ id, name, version }]`                                                  |
| `management.rotateCredential`                 | `PATCH` the `AccountService` account with the new password (fetched by `newCredentialRef`), then a login with it                                                                                                                                                                                | `{ verified }`                                                                         |
| `bios.getSettings`                            | `GET` the system's `Bios` resource                                                                                                                                                                                                                                                              | `{ settings }`, every attribute as text                                                |
| `bios.applySettings`                          | `PATCH` the BIOS settings object (`@Redfish.Settings.SettingsObject`, else `.../Bios/Settings`) with `Attributes`, each value converted to the type the BIOS reports (boolean, integer or string); an attribute the BIOS lacks or a mistyped value is `invalid-args` before anything is written | `{ pendingReboot: true, changed }`                                                     |
| `raid.getLayout`                              | the volumes of the system's first storage subsystem that has any (the RAID controller)                                                                                                                                                                                                          | `{ volumes: [{ level, sizeGiB }] }`                                                    |
| `raid.applyLayout`                            | checks the layout first (levels 0, 1, 5, 6, 10; member disks named by serial number or WWN on that controller), then `DELETE`s every existing volume and `POST`s the layout's volumes                                                                                                           | `{ volumes }`                                                                          |
| `raid.clear`                                  | `DELETE` every volume of the controller                                                                                                                                                                                                                                                         | `{ deleted }`                                                                          |
| `firmware.update`                             | `UpdateService.SimpleUpdate` on the named component of `FirmwareInventory`, with the site cache's copy of the image as `ImageURI`                                                                                                                                                               | `{ started: true, component }`                                                         |

* **Transport.** HTTPS with basic authentication only; a plain-http address is refused (`invalid-address`). With
  `tlsSpkiSha256` the BMC's certificate must match that SPKI hash, since BMCs mostly present self-signed
  certificates; without it the system roots verify it. Redirects are never followed, and a request times out after
  60 seconds.
* **The address.** `bmc.address` is a host (`10.20.0.5`, `bmc.example:8443`), an `https://` or `redfish://` URL,
  or such a URL naming one system of a multi-system service (sushy-tools in front of libvirt, a chassis manager):
  `https://192.168.122.1:8000/redfish/v1/Systems/<id>`. Requests always go over HTTPS to the URL's host. Without a
  system path the client drives the first member of `/redfish/v1/Systems`. Any other path, credentials in the URL, a
  query or a fragment is refused (`invalid-address`). Virtual media uses the system's own manager, from its
  `Links.ManagedBy`, and falls back to the first member of `/redfish/v1/Managers`.
* **Paths stay on the BMC.** Resource paths come from the BMC's own answers (`@odata.id`), so every request path
  must start with `/redfish/v1` and contain no `@`, `..`, query, backslash or whitespace; anything else fails with
  `bmc-error`. A value such as `@evil.example/…` appended to the base would otherwise have sent the credentials to
  another host.
* **Failures.** A BMC that refuses or fails does not fail the command: it succeeds with `ok: false` and a stable
  `errorCode`, as the FakeDaemon's fake site answers. The codes are `unreachable` (including a pin mismatch),
  `auth-failed` (401 or 403), `unsupported` (404, 405, or an operation the client does not have), `bmc-error`,
  `invalid-args`, `invalid-address`, `invalid-config` (driver options on a Redfish BMC, 7753df0),
  `credentials-unavailable` and `unsupported-driver`.
* **Each kind carries only its own operations.** The kind decides the lane, retry safety and whether the batch
  quorum applies, so bcd runs an operation only under the kind metal assigns it: `bmc.query` (`power.getPower`,
  `inspect.inspect`, `firmware.components`, `bios.getSettings`, `raid.getLayout`), `bmc.boot`
  (`management.setBootDevice`, `virtual-media.insert` and `eject`, `bios.applySettings`), `bmc.power`
  (`power.setPower`), `bmc.rotate` (`management.rotateCredential`), `bmc.firmware` (`firmware.update`) and
  `bmc.storage` (`raid.applyLayout`, `raid.clear`). Anything else fails `invalid-payload`, so a `bmc.query`, which
  needs no batch, cannot power a machine off or rebuild its RAID.
* **Firmware comes from the site cache.** The BMC fetches the image itself and cannot check its digest, so the site
  agent first pulls it into its SHA-256-verified image cache (the step's `url` and `sha256`) and hands the BMC the
  boot server's `/cache/t/<grant>/<sha256>` URL, whose grant serves that one image for 2 hours (b7ed8a2; see
  [the image cache](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site)). An agent without a boot server has no cache to serve from and refuses the
  update (`ok: false`, `image-unverified`), as it does a missing or malformed digest.
* **Rotation.** `bmc.rotate` is attested: bcd signs its result with the site agent's node key, which is what the
  worker checks before promoting the new password.

## IPMI through bcx-ipmi [#ipmi-through-bcx-ipmi]

A machine whose BMC speaks IPMI rather than Redfish is enrolled with `bmc.driver: 'ipmi'`. `bcd site` hands its
`bmc.*` operations to the hosted bcx driver `ipmi`, which `daemon/cmd/bcx-ipmi` serves over `ipmitool -I lanplus`
(51806ab), with RMCP+ cipher suite 17 unless the BMC's `bmc.driverOptions` name another (933c51b, 7753df0):

| Operation                  | Kind        | `ipmitool`                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| -------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `power.getPower`           | `bmc.query` | `chassis power status`, answered `{ power: 'on' \| 'off' }`                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `power.setPower`           | `bmc.power` | `chassis power on`, `off`, `cycle`, `reset` or `soft`                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `management.setBootDevice` | `bmc.boot`  | `chassis bootdev pxe`, `disk`, `cdrom` or `bios`, with `options=persistent` when `persistent`                                                                                                                                                                                                                                                                                                                                                                                            |
| `inspect.inspect`          | `bmc.query` | `fru print 0` and `chassis power status`, plus `mc info`, `mc guid`, `lan print` and `sdr list` when they answer (7818f19): `manufacturer`, `model`, `serialNumber` and `power`, with empty `macs` and `ipAddresses`, `productUuid` from the system GUID when the BMC has one (7cd1dd2; lower-cased, canonical form only, never the all-zero GUID), plus `board`, the BMC's own `bmc` identity and a `sensors` summary. See [bcx-ipmi](/docs/extending/drivers#bcx-ipmi-the-ipmi-driver) |

* The operation allowlist per kind, the batch quorum and the site agent's BMC memory apply exactly as for Redfish.
* The site agent fetches the credential as for Redfish and hands the plugin the username and password only as
  secrets; the plugin passes the password to `ipmitool` through a pipe (`-f /dev/fd/3`), never on its command line
  or in its environment. An IPMI 2.0 password has at most 20 bytes and, since `ipmitool -f` keeps only the first word,
  no whitespace (`credentials-unavailable` otherwise).
* The BMC address is a host or IP address with an optional port (`10.20.0.14`, `10.20.0.14:623`).
* IPMI has no TLS: enroll an `ipmi` BMC without `bmc.tlsSpkiSha256`. With a pin, every operation fails
  `pin-unsupported` rather than running unpinned.
* Everything else (virtual media, firmware, BIOS, RAID, rotation) is Redfish only; a catalog ISO boots over PXE
  instead of virtual media (see [sanboot](#sanboot-instead-of-virtual-media)). Metal knows this and refuses a
  verb that would need the rest before any workflow starts (a3ec8d0; see
  [what metal checks before a workflow](#what-metal-checks-before-a-workflow)). Should such an operation still reach
  the site agent, it answers `unsupported-driver` for a concern `ipmi` does not list (`virtual-media`, `bios`, ...),
  and the plugin answers `unsupported-operation` for an unknown operation of a concern it lists; either fails the
  step with `DRIVER_ERROR`.

An `ipmi` machine enrolls and commissions to `ready` (a3ec8d0, 3f19859, f62f9e9). The rules that get it there:

* **Out-of-band inspection is optional.** `inspect.oob` runs `bcx-ipmi`'s FRU inspection when the agent's plugin
  serves `inspect`, and records what it reports. When the driver lacks `inspect`, or the site agent answers
  `unsupported-driver` or `unsupported-operation` (a `bcx-ipmi` from before 7818f19), the step is skipped with a
  `skipped` result, and the ramdisk inspects in band later.
* **PXE MACs: given, or discovered by UUID and confirmed.** IPMI inspection reports no host MACs, so a machine in a
  pool with an agent (one the ramdisk boots) needs another way to them (3d26428, 7cd1dd2, 8de6e2f):

  * enroll it with its `macs`; or
  * enroll it with its `productUuid` (the SMBIOS UUID), and let PXE discovery propose the MAC. `metal.ensureRamdisk`
    sets one-time PXE and power-cycles the machine with no boot entry, so its site agent lists it; a MAC first seen
    under that UUID (in any byte order) within 10 minutes of the cycle becomes the machine's candidate, shown as
    `discoveredMac` (see [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines)). The
    step then waits, polling every 5 seconds, until an operator binds a MAC with
    [`metal.confirmMac`](/docs/reference/api/metal#confirmmac) (from the CLI, `metal discovered --site …` to see the
    candidate, then `metal confirm-mac <node> --mac …`, 002dd95; or **Confirm** on the console's
    [PXE discovery](/docs/guides/clients/console#pxe-discovery) page, 15c11ee), and boots the machine into the ramdisk
    as usual. A
    pool with `policy.trustProvisioningNetwork: true` binds the candidate without confirmation, and then checks the
    ramdisk's own UUID and NICs against the machine.

  With neither `macs` nor a usable `productUuid` (a firmware placeholder UUID does not count), `metal.enroll` is
  `UNSUPPORTED` with reason `bmc-driver-unsupported` ("enroll the machine with macs, or with its productUuid for PXE
  discovery"). Since 7cd1dd2 `bcx-ipmi`'s inspection also returns the BMC's system GUID as `productUuid`, which
  `inspect.oob` records in place of the enrolled one, so discovery matches the UUID the BMC reports. Discovery never
  enrolls a machine and never binds a MAC from the DHCP data alone. An inspection that reports no MACs keeps the
  enrolled ones; one that reports MACs records them with `macSource: 'inspection'`.
* **Agentless machines take their addresses at enrollment.** In a pool with `agent: 'none'`, pass the machine's own
  IPs as `addresses` (IP literals, at most 16, deduplicated). The site's agents get them as
  `SiteSpec.proxy_targets` at once, so `site.httpProxy` can reach the machine. Out-of-band inspection replaces them
  only when it reports some, and IPMI reports none, so they stand.
* **Driver options.** `bmc.driverOptions` is a JSON object of at most 1 KiB that metal sends on every BMC operation
  of the machine (`driverOptionsJson` of the request, canonical JSON with sorted keys). `bcx-ipmi` takes one key,
  `cipherSuite`: 17 by default, or 3, 8 or 12 for BMCs without 17. Keys that look like credentials are
  `INVALID_INPUT`, and the `redfish` driver takes no options (`INVALID_INPUT` at enrollment). See
  [bcx-ipmi](/docs/extending/drivers#bcx-ipmi-the-ipmi-driver).
* **What stays Redfish only.** A hardware profile with BIOS settings or a RAID layout, the dedicated-root teardown
  (`metal-tenant-root-release`, which rotates the BMC password and checks firmware), and a deploy profile whose
  `media.insert` mounts a bare `url` or a `fromStep` ISO are refused for an `ipmi` machine with `UNSUPPORTED`. A
  `media.insert` that names a catalog ISO with `image` is accepted since 3d26428: the machine boots it over PXE
  instead (see [sanboot instead of virtual media](#sanboot-instead-of-virtual-media)).

```ts title="An IPMI machine"
await compute.api.metal.enroll(credential, {
  tenantId: 'root',
  siteId: 'site_…',
  poolId: 'pool_…', // a pool with an agent: the ramdisk boots it by its PXE MAC
  bmc: {
    address: '10.20.0.14',
    driver: 'ipmi',
    credentials: 'secret://root/bmc-bm-7', // no tlsSpkiSha256: IPMI has no TLS
    driverOptions: { cipherSuite: 3 }, // an older BMC without suite 17
  },
  macs: ['b8:ce:f6:00:00:07'], // or, instead: productUuid: '4c4c4544-0042-3810-8051-b2c04f4e3232'
});
```

None of this has run against a real BMC.

### Sanboot instead of virtual media [#sanboot-instead-of-virtual-media]

A driver without virtual media can still boot an ISO (3d26428), because the site agent can serve one over PXE with
iPXE `sanboot` (see [the boot server](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site)). The ISO must be a
catalog image of format `iso`, named in the step's `image` (at most 200 characters), since only a catalog image has
a digest the site's cache can verify:

```json title="deploy profile step"
{ "use": "media.insert", "with": { "image": "img_01k5w…" } }
```

`media.insert` resolves the image in the tenant's catalog (`NOT_FOUND` otherwise, `INVALID_INPUT` when it is not an
`iso`) and verifies it for use. Then:

1. **A driver with virtual media** (the table says so, or does not know the driver) mounts it as before: `url`,
   `fromStep`'s URL, or else the image's `https://` source. When the site agent answers that the driver has no
   virtual media (`DRIVER_ERROR` with an unsupported code), a step with `image` falls through to sanboot; one
   without fails.
2. **Without virtual media**, metal pins the image at the machine's site for the workflow, so the site agents cache
   it (`cacheImages`), and waits (polling every 10 seconds) until the site's `image_locations` row is `ready`. It then
   sends the machine's boot entries with `sanbootSha256` set and no discovery token, and the step returns
   `{ sanboot: <sha256>, imageId, entries }`. The step's timeout is 30 minutes, to leave time for the cache.
3. **A later `management.setBootDevice` step to `cd`** sets `pxe` instead while the workflow's last media step is a
   sanboot `media.insert`, and returns `sanboot: true`: the "CD" is on the network.
4. **`media.eject`** after a sanboot sends `boot.cleanUp` to the site agent, which drops the entries, then releases
   the workflow's image pin and returns `{ ejected: true, sanboot: true }`. On a driver without virtual media and no
   sanboot to undo, it returns `ejected: false` with a `skipped` reason instead of failing.

The check before a workflow counts this fallback (`unmetBmcNeeds`): a `media.insert` with `image` needs only
`management.setBootDevice` on such a driver, so an `ipmi` machine takes the profile, while a bare `url` or `fromStep`
still needs `virtual-media.insert` and is refused.

### What metal checks before a workflow [#what-metal-checks-before-a-workflow]

Metal keeps a table of the BMC operations each driver serves (`builtinBmcDrivers` in
`packages/metal/src/capabilities.ts`): `redfish` serves every operation metal sends, plus `inspect.macs` (its
inspection reports the PXE MACs); `ipmi` serves `power.getPower`, `power.setPower`, `management.setBootDevice` and
`inspect.inspect`. Each metal step needs certain operations: `metal.ensureRamdisk` needs the boot device and power,
`media.insert` virtual media (or, with a catalog `image`, only the boot device, for
[sanboot](#sanboot-instead-of-virtual-media)), `metal.firmwareCheck` `firmware.components`, `metal.rotateBmcCredential`
`management.rotateCredential`, and `bios.apply`, `bios.verify` and `raid.apply` count only when the node's hardware
profile sets BIOS settings or a RAID layout. `inspect.oob` is optional, and a step with `onFailure: 'continue'`
never counts.

Before the workflows start, metal checks every step of the profiles a call will run against the node's driver, and
refuses with `UNSUPPORTED`:

```json title="error details"
{
  "reason": "bmc-driver-unsupported",
  "driver": "ipmi",
  "unmet": [{ "profile": "metal-commission@2", "step": "bios.apply", "operations": ["bios.getSettings", "bios.applySettings"] }]
}
```

| Where                           | What is checked                                                                                                                                                                        |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metal.enroll`                  | The `enrolling`, `inspecting` and `commissioning` chain, and that the PXE MACs are knowable (given, found by inspection, or discoverable by a usable `productUuid` and then confirmed) |
| Every lifecycle verb            | The chain of workflows the verb starts; the verb is rolled back with it                                                                                                                |
| `metal.deploy`                  | A stored deploy profile (the tenant's or a public one), at the call                                                                                                                    |
| A module-shipped deploy profile | When the reconciler starts it: under a savepoint, rolled back, and the node fails with the same error                                                                                  |
| `metal.unrescue`, `metal.power` | The unrescue profile; `power.setPower`                                                                                                                                                 |

`metal({ bmcDrivers: { <id>: [operations] } })` declares the operations of other drivers, or replaces the built-in
entries. The site's agents narrow the table: each agent's Hello writes `bc.dev/driver-kind.<driver>.<kind>` labels
for the drivers it hosts (14ceb1e; see [labels](/docs/guides/nodes/inventory-and-health#labels)), and metal keeps
only the operations of the kinds they report. That also makes a driver the table does not know checkable. A driver
neither names is not checked in advance; its site agent answers at run time. Built-in Redfish is never in the labels.

The plugin is not built into bcd. A release ships it as `bcx-ipmi_<version>_linux_<arch>.tar.gz`, the api's
downloads route serves it, and `daemon install-script --install-plugins bcx-ipmi --release-manifest …` installs it
into `/usr/local/lib/bcd/plugins` with its pin, after checking the archive against the signed manifest (adf2ae9; see
[installing plugins from the release](/docs/daemon/install#installing-plugins-from-the-release)). Install `ipmitool`
on the agent's host; without it the plugin's health is `NOT_SERVING`. It has been tested against a fake `ipmitool` only. See [bcx-ipmi](/docs/extending/drivers#bcx-ipmi-the-ipmi-driver).

## Enrolling a machine [#enrolling-a-machine]

`metal.enroll` creates the node in `enrolling`, owned by the pool's tenant, with role `unassigned`,
`platform` trust and power `unknown`, and queues the reconciler. It is decided as
`compute:nodes:provision` on the pool. It publishes `node.enrolled`. The node's metal view carries `macSource`
(`operator` for MACs given at enrollment or with `metal.confirmMac`, `inspection`, or `discovered`) and, for a machine
still without MACs, the `discoveredMac` candidate awaiting confirmation (8de6e2f).

```ts
const node = await compute.api.metal.enroll(credential, {
  tenantId: 'root',
  siteId: 'site_…',
  poolId: 'pool_…',
  bmc: {
    address: 'https://10.0.0.10',
    driver: 'redfish',
    credentials: { username: 'admin', password: process.env.BMC_PASSWORD! }, // or 'secret://…'
    tlsSpkiSha256: '<64 hex digits>',
  },
  macs: ['b8:ce:f6:00:00:01'],
  uefi: true,
});
// node.bmc = { driver: 'redfish', address: 'https://10.0.0.10', credentialsSet: true }
// node.addresses = [] until inspection reports some (the view adds bmc.driverOptions when set)
```

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The pool owner.', required: true },
  siteId: { type: 'string', description: &#x22;The pool's site. Another site is INVALID_INPUT.&#x22;, required: true },
  poolId: { type: 'string', description: 'A pool the tenant owns; otherwise NOT_FOUND.', required: true },
  'bmc.address': { type: 'string', description: 'The BMC endpoint, at most 512 characters: a host, or an https:// URL that may name one system (/redfish/v1/Systems/<id>) of a multi-system service.', required: true },
  'bmc.driver': { type: 'string', description: 'A driver id such as redfish or ipmi.', required: true },
  'bmc.credentials': { type: &#x22;'secret://…' | { username, password }&#x22;, description: &#x22;One of the tenant's own secrets (secret://<tenantId>/<id>; another tenant's is ACCESS_DENIED, and vault:// or env:// references are for root callers only), or inline credentials stored at once and never echoed.&#x22;, required: true },
  'bmc.tlsSpkiSha256': { type: 'string', description: &#x22;64 hex digits: the SHA-256 of the BMC certificate's SubjectPublicKeyInfo, pinned.&#x22; },
  'bmc.driverOptions': { type: 'object', description: 'Options for the BMC driver, sent on every BMC operation: a JSON object of at most 1 KiB, never credential-like keys (e.g. bcx-ipmi { cipherSuite: 3 }). The redfish driver takes none. Stored in metal_nodes.bmc_driver_options (metal/0006).' },
  macs: { type: 'string[]', description: 'Up to 16 PXE MACs. Redfish out-of-band inspection finds them when left out; IPMI inspection does not, so an ipmi machine in a pool with an agent is UNSUPPORTED without them or a usable productUuid.' },
  addresses: { type: 'string[]', description: &#x22;Up to 16 IP literals: the machine's own addresses, the only hosts site.httpProxy may reach for an agentless machine. Out-of-band inspection replaces them when it reports some.&#x22; },
  productUuid: { type: 'string', description: 'The SMBIOS product UUID (what the BMC reports), at most 64 characters. A UUID in any case or with braces is stored in canonical form; any other string is stored lower-cased and never matches (894f6f9). Without macs, PXE discovery proposes a MAC from it for metal.confirmMac (3d26428, 8de6e2f).' },
  arch: { type: &#x22;'x86_64' | 'arm64'&#x22;, description: 'Selects the boot architecture of the PXE entry.', default: &#x22;'x86_64'&#x22; },
  uefi: { type: 'boolean', description: 'UEFI or legacy BIOS boot.', default: 'true' },
  hardwareProfile: { type: 'string', description: &#x22;Overrides the pool's policy.hardwareProfile for this node.&#x22; },
}"
/>

From the command line, `metal enroll` (171d9be) takes the credentials as a reference, or as a username and a
password file; there is no password flag, so a password never lands in shell history or the process list:

```bash
better-compute metal enroll --tenant root --site site_01k5w… --pool pool_01k5w… \
  --bmc-address https://10.0.0.10 --bmc-driver redfish --bmc-credentials secret://root/bmc-bm-1 \
  --bmc-tls-spki 9f3e… --macs b8:ce:f6:00:00:01
better-compute metal enroll --tenant root --site site_01k5w… --pool pool_01k5w… \
  --bmc-address 10.0.0.11 --bmc-driver redfish --bmc-username admin --bmc-password-file ./bmc-pass.txt
```

`--bmc-credentials` must be a `secret://<tenant>/<id>` reference of the node's own tenant (the server refuses
another tenant's with `ACCESS_DENIED`, and takes `vault://` and `env://` references from root callers only, since
the site agent presents the secret to the address you name); otherwise pass `--bmc-username` with
`--bmc-password-file` (one trailing newline is dropped, and an empty file is refused), never both forms. `--legacy-bios`
sends `uefi: false`, and `--arch`, `--product-uuid` and `--hardware-profile` map to the fields below. The command has
no flags for `bmc.driverOptions` or `addresses` yet: enroll a machine that needs them through the API. See
[metal enroll](/docs/reference/cli#metal-enroll).

`bmc.driver`, `bmc.tlsSpkiSha256` and every entry of `macs` must be JSON strings: a missing driver,
`true`, or a number for the pin or a MAC is `INVALID_INPUT`, never coerced to text. The pin may use
upper-case hex and a MAC `-` separators; both are stored lower case, MACs with `:`.

The first workflow, `metal-enroll`, proves the credentials work (`power.getPower`) and inspects the
machine out of band (`inspect.oob`), recording the PXE MACs, SMBIOS UUID and IP addresses the BMC
reports (Redfish reports all three; `bcx-ipmi` reports only the UUID, from the system GUID, and the enrolled MACs
and addresses are kept). A reported UUID is what [PXE discovery](/docs/guides/bare-metal/sites#discovering-new-machines)
later matches for a machine without MACs. A driver without out-of-band
inspection skips that step. A machine whose BMC does not answer fails in `enrolling`. Before the node is created,
`metal.enroll` checks that its driver can serve enrollment through commissioning (see
[what metal checks before a workflow](#what-metal-checks-before-a-workflow)).

## Credentials [#credentials]

Postgres never holds a BMC password, only a `secret://` reference:

* **Inline credentials** go to the secret store before anything else happens, so the pair never
  reaches a row, a log or an event. The answer says only `credentialsSet: true`.
* **A reference** comes from [`secrets.put`](/docs/reference/api/secrets#put), the write-only secret
  API (v0.2). Store the pair as JSON, the shape inline credentials are stored in, and use
  `scope: 'fleet'` for platform machines:

```ts
const { ref } = await compute.api.secrets.put(credential, {
  tenantId: 'root', // fleet secrets belong to the root tenant
  name: 'bmc-bm-1', // a second put under the same name makes a new version
  value: JSON.stringify({ username: 'admin', password }), // at most 64 KiB, never returned
  scope: 'fleet', // decided on compute-fleet/global; 'tenant' (default) on compute-tenant/{t}
  idempotencyKey: 'bmc-bm-1-v1',
});
```

The CLI's `secrets put` (573b0b0) reads the value only from a file (`--value-file`) or standard input (`--stdin`),
never from an argument, so it stays out of shell history and the process list. Exactly one of the two is required,
one trailing newline is dropped, and an empty value is refused before anything is sent:

```bash
# bmc-bm-1.json holds {"username":"admin","password":"…"}, mode 0600
better-compute secrets put --tenant root --name bmc-bm-1 --scope fleet --value-file ./bmc-bm-1.json
vault read -field=bmc secret/site-fsn1 | better-compute secrets put --tenant root --name bmc-bm-1 --scope fleet --stdin
better-compute secrets delete --tenant root --name bmc-bm-1 --scope fleet
```

`secrets.put` and `secrets.delete` are decided as `compute:secrets:manage`. `secrets.put` records the scope it
decided on in `secret_refs.scope` (migration `core/0025_secret_scope`) and answers it in the secret's info, so a
fleet secret stays distinguishable from the root tenant's own: a put or delete of a `fleet` secret without
`scope: 'fleet'` is `INVALID_STATE`, since it was decided on the tenant and the secret is guarded by the fleet.
Secrets stored before the migration read `tenant`, and a fleet put may claim one. No method reads a value
back, and the `security.secret.put` and `security.secret.deleted` events carry only the ref, name
and version. Unless you pass other stores in the `secrets` option, the server uses the
`local-sealed` store, which keeps ciphertext in `sealed_secrets`, sealed under the deployment
secret.

### How the site agent gets the password [#how-the-site-agent-gets-the-password]

A command carries only the reference. The agent asks for the value with a `SecretRequest` signed by
its node key and naming a fresh one-time key, and the API releases it only when all of these hold:

1. the request is signed by the node's current, unrevoked certificate and names the session's node;
2. the certificate is a platform identity, never a tenant's `node-tenant` certificate;
3. the reference is bound to work the node is doing right now: a live command of that workflow step,
   addressed to it, that carries the reference;
4. the request id was not used before.

The answer is a JWE (ECDH-ES+A256KW, A256GCM) to the one-time key that expires within five
minutes, so the gateway relays only ciphertext.

## Power [#power]

[`metal.power`](/docs/reference/api/metal#power) sends one `bmc.power` command to the site agent and
returns `{ commandId }` without waiting for the BMC. It is decided as `compute:nodes:power`. `bmc.power` is
destructive, so when the deployment keeps batches it first creates a batch of one leaf (subject
`bmc:<site agent>/<BMC address>`, kind `bmc.power`, reason `power <action> <node>`) and enqueues the command in it
(478025a); the answer then carries `batchId` and `held`. With approver keys configured the command stays held until
approvers sign that batch (see [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)). A held command
lives as long as its batch waits for approvals (24 hours by default), not the five minutes an unheld one has to
reach the BMC, so a slow approval does not find it expired.

```bash
better-compute metal power node_01k5w… --tenant root --action graceful-restart
```

| `action`           | Redfish `ResetType` |
| ------------------ | ------------------- |
| `on`               | `On`                |
| `off`              | `ForceOff`          |
| `graceful-off`     | `GracefulShutdown`  |
| `reset`            | `ForceRestart`      |
| `graceful-restart` | `GracefulRestart`   |
| `cycle`            | `PowerCycle`        |
| `nmi`              | `Nmi`               |

* It is `INVALID_STATE` while a workflow controls the machine's power (`inspecting`,
  `commissioning`, `deploying`, `releasing`), when the node has no BMC credentials, or when no site
  agent of the node's site is connected. Any other action name is `INVALID_INPUT`.
* The command expires five minutes after it is queued and is keyed by the request id, so a retried
  HTTP request does not power-cycle twice.
* Power is its own axis: it never changes the provision state. The power steps record the last
  known state on the node.

## Rotating BMC passwords [#rotating-bmc-passwords]

A BMC password that a tenant may have seen, or that has simply lived too long, has to change without
ever locking the platform out. `metal.rotateBmcCredential` rotates in two phases, and the old
password stays current until the new one is proven:

<Mermaid
  chart="sequenceDiagram
  participant W as Worker
  participant V as Secret store
  participant S as Site agent
  participant B as BMC
  W->>V: store a generated password
  W->>W: record a pending version
  W->>S: bmc.rotate, refs only
  S->>W: SecretRequest for the new ref
  W-->>S: JWE to a one-time key
  S->>B: set the password, then log in with it
  S-->>W: result signed by the site agent
  W->>W: promote pending to current"
/>

1. The worker generates a 20-character password with upper case, lower case, a digit and a symbol
   (no quotes, backslashes or look-alike characters), stores it in the secret store, and records a
   `pending` version in `bmc_credential_versions`. The enrollment credential is recorded first as
   version 1, `current`. The new secret keeps the enrollment shape, `{ username, password }` JSON,
   so everything that resolves the reference still gets the account name: the username is the
   step's `username` argument, else the one in the current credential (resolved on the server). A
   bare password is stored only when neither names a username.
2. `bmc.rotate` sends only references. The site agent fetches the new value, applies it through the
   driver's `management.rotateCredential`, logs in with it, and signs the result.
3. Only a result signed by a `site` certificate that says `verified: true` promotes the version:
   pending becomes `current`, the old one `superseded`, and the node's credential reference and
   driver bindings name the new secret. The node's `BmcCredentialRotated` condition turns `True`.
4. On any failure the old credential stays current, the version becomes `failed` (keeping its
   reference, since the BMC may hold the new password if only the answer was lost), the condition
   turns `False` with the reason, and the step fails with `DRIVER_ERROR`.

A retry of the step reuses the pending version rather than minting another password. Superseded
secrets that rotation itself made are deleted from the store once no pending or in-flight command and
no other machine still names them. The secret a machine was enrolled with (version 1) is never
deleted: it is yours, and one rack-wide reference may serve many machines. A node has at most one
current and one pending version at a time.

Today the tenant-root teardown runs the rotation after every dedicated-root lease, and a failed
rotation quarantines the machine (see [dedicated-root metal](/docs/guides/bare-metal/dedicated-root)).
You can add it to your own profiles:

```ts
{ use: 'metal.rotateBmcCredential', on: 'site', with: { username: 'admin' } }
```

PLAN.md also rotates at enrollment and every 90 days; neither is scheduled automatically yet.
Password rules, lockouts and account names differ per vendor, and the driver is responsible for
meeting or refusing them.

## Firmware checks [#firmware-checks]

`metal.firmwareCheck` asks the BMC for every firmware component (`firmware.components`) and compares
it with the pool's `policy.firmwareBaseline` (component id to exact version). Drift fails the step
with `DRIVER_ERROR` and details `code: 'FIRMWARE_DRIFT'`, listing each component's expected and found
version, unless the pool sets `firmwareRemediation: 'reflash'` and has an https image with a sha256
in `firmwareImages` for every drifted component. Then `bmc.firmware` writes each image, and the check
runs again. The teardown of a dedicated-root lease uses it.

## Retirement [#retirement]

`metal.retire` erases the disks (crypto-erase), powers off and runs `metal.revokeIdentity`, which
revokes the node's certificates, bumps its identity generation, clears its driver bindings, drops
its BMC credential reference, and frees the hardware subjects bound to it, so the machine can join
again later as a new node.

## Operating guidance [#operating-guidance]

PLAN.md sets these rules for the management network, which are deployment practice rather than
something the control plane can enforce:

* BMC passwords are random per machine and never shared across machines.
* The management VLAN is reachable only from site agents.
* Vendor default accounts and IPMI cipher suite 0 are disabled (verify per vendor).
* Pin the BMC's certificate with `tlsSpkiSha256` wherever the BMC serves a stable one.


# Dedicated-root metal (/docs/guides/bare-metal/dedicated-root)

> Lease whole machines to tenants with root, under tenant-controlled trust and node-tenant certificates, and take them back with a BMC-driven teardown.



Some tenants need the whole machine: their own kernel, their own OS image, root on the host. The
moment a tenant has root, nothing running on that machine can be trusted by the platform again: not
its agent, not its inventory, not its reports. A dedicated-root lease is designed around that. For
the lease, the machine's trust changes to `tenant-controlled` and the platform gives up its identity
on it. Afterwards, the machine comes back only through a teardown driven entirely from the BMC side.

Dedicated-root leases are a v0.5 feature of PLAN.md. They are implemented in metal
(`metal/0003_tenant_root`) and capacity (whole-node placement), and tested end to end against the
simulated site. The contract between the two modules is rows only; neither imports the other.

## Set up a tenant-root pool [#set-up-a-tenant-root-pool]

Whole-node leases are served only by pools that opt in with `policy.tenantRoot`. Give the pool a
firmware baseline too: the teardown verifies it, and fails without one.

```ts
await compute.api.pools.create(credential, {
  tenantId: 'root',
  siteId,
  class: 'metal-dedicated',
  policy: {
    hardwareProfile: 'gpu-redfish',
    tenantRoot: true,
    firmwareBaseline: { bmc: '1.12.0', bios: '2.4.1', 'gpu-vbios': '96.00.5f' },
    firmwareRemediation: 'reflash', // or 'quarantine', the default
    firmwareImages: {
      bios: { url: 'https://firmware.site-a/bios-2.4.1.bin', sha256: '<64 hex digits>' },
    },
  },
});
```

<TypeTable
  type="{
  tenantRoot: { type: 'boolean', description: &#x22;Reservations may use binding: 'node' here; metal leases the pool's ready nodes whole.&#x22; },
  firmwareBaseline: { type: 'Record<string, string>', description: 'Component id to the exact version every component must report after a lease. Required by the teardown.' },
  firmwareRemediation: { type: &#x22;'quarantine' | 'reflash'&#x22;, description: 'On drift: fail with maintenance on, or write the baseline images and check again.', default: &#x22;'quarantine'&#x22; },
  firmwareImages: { type: 'Record<string, { url, sha256 }>', description: 'Baseline images for reflash: https URLs and 64-hex digests; anything else is ignored.' },
  allowUnverifiedRelease: { type: 'boolean', description: 'Agentless pools only: let a release reach ready without an attested wipe.' },
}"
/>

Offer the pool to the tenants who may lease from it, as with any pool (see
[pools](/docs/guides/capacity/pools)). Nodes under tenant control add no capacity to the pool's
rollups while they are leased.

## Reserve a whole machine [#reserve-a-whole-machine]

The tenant creates a reservation with `binding: 'node'` and names
the metal profile to deploy:

```ts
const reservation = await client.reservations.create({
  tenantId: 'acme',
  poolId,
  devices: [{ class: 'gpu', count: 8 }],
  duration: '7d',
  binding: 'node',
  node: { profileId: 'acme-root-os', profileVersion: 1 },
  idempotencyKey: 'acme-root-2026-09',
});
```

* `binding: 'node'` needs `node.profileId`, and `node` needs `binding: 'node'` (`INVALID_INPUT`
  otherwise). Always pass `profileVersion`: metal refuses to start a lease whose reservation names no
  version, publishes `node.tenant-root.refused`, and leaves the node `ready`.
* A node binding cannot be `spot`, and a pool without `policy.tenantRoot` refuses it.
* Placement takes nodes that are `ready`, not in maintenance, not cordoned, with `platform` trust and
  no live allocation in the window, whose healthy atoms cover `devices`. Other NoSchedule reasons do not
  stop a lease, with one exception: a node tainted `dedicated=<tenant>` is leased only by that tenant
  (see [taints](/docs/guides/nodes/operating-nodes#taints)). The same check guards the lease's commit and
  BookingRepair. The allocation books every
  bookable atom of the node, so nothing else can land on it. `gang` leases several such nodes on one
  topology value.
* The profile resolves in the lessee's tenant: the lessee's own profile, or a public one. The image
  it deploys resolves in the machine owner's tenant, so the lessee's profile has to name a public
  (promoted) image.
* If metal has not finished deploying the node within 2 hours, the lease fails and is not placed
  again elsewhere.
* A bound lease bills `node.metal.hours` per lease-hour, plus the GPU meter for the GPUs that were
  healthy at the start of each hourly window. See [billing and usage](/docs/guides/iam/billing-and-usage).

## How a lease starts [#how-a-lease-starts]

<Mermaid
  chart="sequenceDiagram
  participant C as Capacity
  participant M as Metal
  participant J as JoinService
  participant T as Lessee OS
  C->>C: book every atom of a ready node
  C->>M: queue metal work at the start
  M->>M: trust tenant-controlled, new identity generation
  M->>M: deploy the lessee's profile
  T->>J: join with the provisioned token
  J-->>T: node-tenant certificate
  M->>C: allocation bound, reservation queued
  C->>C: reservation active"
/>

1. When the reservation reaches its start (`binding`), capacity queues the node's `metal` work.
2. On a `ready` node of a tenant-root pool, not in maintenance, metal records the lease, marks the
   machine `tenant_root`, sets its release wipe to `secure-erase` and its trust to
   `tenant-controlled`, and starts a new identity generation, which revokes every platform certificate
   of the node before any tenant code runs. It then applies `deploy` with the lessee's profile and
   publishes `node.tenant-root.leased` (allocation, reservation, lessee tenant, profile and version).
3. The deploy workflow runs in the lessee's tenant. When the installed OS joins, JoinService sees the
   node's `tenant-controlled` trust and issues a `node-tenant`
   certificate instead of a `node` one.
4. When the deploy completes, the node is `deployed` with the profile's role (usually `dedicated`).
   Metal marks the allocation bound and queues capacity's reservation work, which moves the
   reservation to `active`, and publishes `node.tenant-root.bound`.

## What the tenant gets, what the platform keeps [#what-the-tenant-gets-what-the-platform-keeps]

A `node-tenant` certificate runs a session as a node, but carries none of a platform node's
authority:

|                                                    | Lessee                                                 | Platform                 |
| -------------------------------------------------- | ------------------------------------------------------ | ------------------------ |
| Root on the OS, its disks and kernel               | yes                                                    | no                       |
| Session through the gateway, telemetry and `leave` | yes                                                    |                          |
| Secrets from the broker                            | never: the broker refuses `node-tenant` certificates   |                          |
| Destructive commands                               | never: the gateway rejects them as `tenant-controlled` |                          |
| Results that count (wipe, diagnostics)             | never: only platform identities attest                 | the platform ramdisk     |
| BMC, power and boot device                         | no                                                     | yes, from the site agent |
| The teardown                                       | no                                                     | always                   |

The attestation check accepts only `ramdisk` and `node` certificates by default, so even a wipe
result the lessee's OS signs with its own key for this very node is refused.

The lessee's inventory and health reports are advisory. While the node is `tenant-controlled`, an
inventory it sends is appended as a snapshot (`node.inventory.changed` with `advisory: true`) but
changes none of the node's devices, device health, labels, conditions, image cache or hardware key, and
its XIDs and host pressure are ignored. A GPU the lessee reports missing or unhealthy therefore stays
healthy and on the lease's bill. `metal.exitTenantRoot` ends the advisory period, so the first report
under platform trust applies for real. See
[trust](/docs/guides/nodes/inventory-and-health#trust-tenant-controlled-nodes-are-advisory).

PLAN.md also disables in-band BMC access where the vendor supports it. That is not implemented yet,
which is one reason the teardown rotates the BMC password.

## The teardown [#the-teardown]

When capacity releases the allocation (a release, the end of its time or a lapsed lease), metal
moves the deployed node to `releasing` (`node.tenant-root.ending`) and runs
`metal-tenant-root-release@1` instead of the ordinary release. Every step is driven from the site
agent or the control plane:

| # | Step                        | Why                                                                                           |
| - | --------------------------- | --------------------------------------------------------------------------------------------- |
| 1 | `power.setPower` off        | stop the tenant's OS                                                                          |
| 2 | `metal.fenceTenant`         | revoke the lessee's certificates first, so nothing left behind can sign for this node         |
| 3 | `metal.rotateBmcCredential` | the lessee may have learned the BMC password; see [BMC rotation](/docs/guides/bare-metal/bmc) |
| 4 | `metal.ensureRamdisk`       | boot the platform ramdisk over PXE                                                            |
| 5 | `disk.wipe` secure-erase    | an attested wipe of every disk                                                                |
| 6 | `diag.dcgm` level 1         | GPU reset and ECC check                                                                       |
| 7 | `metal.firmwareCheck`       | every component must match `policy.firmwareBaseline`                                          |
| 8 | `metal.exitTenantRoot`      | a new identity generation, trust back to `platform`, the lease cleared                        |
| 9 | `power.setPower` off        | park the machine                                                                              |

The cleanup step `boot.cleanUp` removes the boot entries. As with every release, the node reaches
`ready` only if the wipe's result is node-signed and every disk passed read-back. The v0.5 exit
criterion holds in the tests: a lease and its release yield a new identity generation.

Agentless pools (`agent: 'none'`) run `metal-tenant-root-release-oob@1` instead: power off, fence,
rotate, eject media, firmware check, exit. It has no wipe, so it reaches `ready` only with
`policy.allowUnverifiedRelease`.

## When the teardown fails [#when-the-teardown-fails]

Every failure quarantines the machine: the node is `failed` with `failedFrom: 'releasing'` and
maintenance on, and it stays `tenant_root` with `tenant-controlled` trust, because only a completed
teardown clears them.

* **Firmware drift** fails the check with `code: 'FIRMWARE_DRIFT'`, listing each drifted component.
  With `firmwareRemediation: 'reflash'` and an image for every drifted component, the site agent
  writes the images (`bmc.firmware`) and the check runs again; drift that survives the reflash still
  quarantines.
* **A rotation the BMC refuses** fails the teardown, and the old credential stays current.
* **A wipe not signed by a platform identity** never reaches `ready`.
* **A pool without `firmwareBaseline`** fails the check with `INVALID_CONFIG`.

`metal.recover` sends the node back to `releasing`, and since it is still `tenant_root`, the teardown
runs again from the start. The same holds for a lessee deploy that failed: recovering it runs the
tenant-root teardown, not the ordinary release. Maintenance stays on after recovery; see
[failure and recovery](/docs/guides/bare-metal).

## When the lessee's tenant is deleted [#when-the-lessees-tenant-is-deleted]

When Better IAM deletes the lessee's tenant, compute's tenancy teardown ends its reservations through
capacity (`capacityTenantDeleted`), so each lease is released now and runs the tenant-root teardown above.
Metal's own `tenantDeleted` hook retires the machines the deleted tenant owns, not the platform machines it
leased. See [tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants).

## Limits [#limits]

* PLAN.md offers dedicated-root only on vetted hardware until TPM measured boot exists, since a
  tenant with root could otherwise persist below the OS in ways a firmware version check does not see.
* Reservation lifecycle, quotas and offerings work as for any reservation; see
  [reservations](/docs/guides/capacity/reservations).


# Deploying machines (/docs/guides/bare-metal/deploying)

> Walkthroughs that deploy machines on an emulated QEMU/KVM site, install Ubuntu over virtual media, and run Talos on an agentless pool.



This page walks through the three deployments the repository proves: a GPU host commissioned and
deployed over PXE on an emulated site, Ubuntu installed by its own installer
from a per-machine ISO, and Talos on machines that never run `bcd`. Each follows the same lifecycle from
[bare metal](/docs/guides/bare-metal); they differ in how the OS gets onto the disk and how the
control plane learns it is up.

> **Status.** 
  None of these has run on real hardware or VMs yet. They need `bcd site` and `bcd ramdisk`, the Go daemon's v0.2
  modes, which are now written (see [what the ramdisk does](#what-the-ramdisk-does) and
  [the boot server](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site)) but not released, and tested only by
  the Go unit tests. The Ubuntu and Talos walkthroughs run in the test suite against simulated site agents and
  machines (`tests/extensions/`), and the emulated-site configuration is checked to load and pass
  `better-compute doctor`.

## Adopt or PXE [#adopt-or-pxe]

|                 | Adopt (v0.1)                          | Enroll and deploy (v0.2)                                           |
| --------------- | ------------------------------------- | ------------------------------------------------------------------ |
| You start with  | a machine running an OS you installed | a machine with a BMC, whatever is on its disks                     |
| You provide     | a join token and the install script   | the BMC address and credentials, a site agent, an image, a profile |
| Reimage         | never                                 | on every deploy                                                    |
| On the way back | no platform wipe                      | an attested wipe before `ready`                                    |
| States          | `adopting`, `adopted`                 | `enrolling` through `ready`, `deploying`, `deployed`               |

Adoption is covered on [joining nodes](/docs/guides/nodes/joining). The state table lets an adopted
node with a BMC be released, but metal acts only on machines it enrolled, so an adopted node has no
metal record and `metal.undeploy` answers `NOT_FOUND` for it today.

## Before you start [#before-you-start]

A deployment that provisions metal needs, beyond the usual control plane:

* `metal({ hardwareProfiles })` and `provisionModule({ modules: [bareMetal], userData })`, as on
  [bare metal](/docs/guides/bare-metal). The examples' shared configuration
  (`examples/shared/compose.ts`) registers both modules but no `userData` options and no attestors,
  so add them for profiles that render user data and for agents that join over PXE.
* A PKI bundle (`better-compute pki init`), so JoinService can admit site agents, ramdisks and
  installed machines, and the gateway role for them to connect to. See [PKI](/docs/operations/pki).
* The `discovery` and `provisioned` attestors, wired as on
  [sites](/docs/guides/bare-metal/sites).
* A site with a connected site agent, a pool on it whose `policy.hardwareProfile` names your hardware
  profile, an image in the catalog, and a deploy profile.

## Walkthrough: an emulated site [#walkthrough-an-emulated-site]

`examples/emulated-site` stands in for a rack: QEMU/KVM virtual machines are the servers,
`sushy-emulator` gives each a Redfish BMC over libvirt, and a `bcd site` agent on the libvirt bridge
runs proxy DHCP, iPXE and the image cache. It needs Linux with `/dev/kvm`, QEMU, libvirt, sushy-tools
and a `bcd` build, plus the iPXE builds and a ramdisk image, which the repository does not build. Nobody has run
it end to end yet. Its configuration registers one hardware profile:

```ts title="examples/emulated-site/better-compute.config.ts"
export const hardwareProfiles = {
  'emulated-redfish': {
    description: 'QEMU/KVM machines behind sushy-tools (Redfish), PXE-booted by the site agent',
    drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream' },
  },
};
```

      ### Create the machines and their BMCs [#create-the-machines-and-their-bmcs]

    ```bash
    virt-install --name bm-1 --memory 8192 --vcpus 4 --disk size=40 --pxe --network network=bc-site --noautoconsole
    virt-install --name bm-2 --memory 8192 --vcpus 4 --disk size=40 --pxe --network network=bc-site --noautoconsole
    sushy-emulator --libvirt-uri qemu:///system --port 8000
    ```
  
      ### Start the control plane [#start-the-control-plane]

    ```bash
    cd examples/emulated-site
    better-compute pki init --trust-domain site.example.test
    better-compute serve
    ```

    The gateway role needs a serving certificate (`better-compute pki issue-gateway`); see
    `examples/postgres` for the environment variables. Calls below run as `BETTER_COMPUTE_TOKEN`.
  
      ### Create a site, its agent and a pool [#create-a-site-its-agent-and-a-pool]

    ```bash
    export BETTER_COMPUTE_TENANT=root
    better-compute sites create --kind shared --region lab-1 --key lab-1
    better-compute site-tokens create --site site_… --ttl 3600 --key lab-1-agents --out ./site-token
    echo '{"hardwareProfile":"emulated-redfish"}' > policy.json
    better-compute pools create --site site_… --class metal-host --policy-file policy.json
    ```

    `site-tokens create --out` writes the one-time token to a new 0600 file instead of printing it.

    Join a site agent on the `bc-site` network with the site token (`BCD_JOIN_TOKEN=bcjt_… bcd join --role site`,
    then `bcd site --config /etc/bcd/config.toml`). Its `config.toml` needs a `[boot]` section with `proxy_dhcp`, `tftp`
    and `tftp_root` (the iPXE builds), `server_ip` and `url` on the bridge, and the ramdisk's `kernel` and `initrd`; see
    [installing bcd](/docs/daemon/install#a-site-agent). It becomes the site's boot server once the SiteBoot controller
    gives it the lease.

    One `sushy-emulator` serves both VMs, so each enrollment names its own system in the BMC address:
    `https://192.168.122.1:8000/redfish/v1/Systems/<id>`, where `<id>` is the libvirt domain's UUID
    (`virsh domuuid bm-1`, or the `Members` of `GET /redfish/v1/Systems` on the emulator). bcd's Redfish client then
    drives that system and its own manager for virtual media; an address without a system path would drive the first
    system for both machines. See [Redfish in bcd site](/docs/guides/bare-metal/bmc#redfish-in-bcd-site).
  
      ### Register the image and the profile [#register-the-image-and-the-profile]

    ```bash
    better-compute api images.register --data @image.json
    better-compute profiles create ./ubuntu-gpu-host.json
    better-compute profiles validate ubuntu-24.04-gpu-host --profile-version 1
    ```

    `images register` (with `--ref`, `--source`, `--disk-format` and `--sha256`) is the CLI form of the first call when
    you would rather pass flags than a JSON body.

    The profile is the GPU host from [profiles and images](/docs/guides/bare-metal/profiles-and-images):
    render user data, boot the ramdisk, stream the image, set boot to disk, kexec.
  
      ### Enroll the machines [#enroll-the-machines]

    ```json title="enroll-bm-1.json"
    {
      "tenantId": "root",
      "siteId": "site_…",
      "poolId": "pool_…",
      "bmc": {
        "address": "https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>",
        "driver": "redfish",
        "credentials": { "username": "admin", "password": "…" }
      }
    }
    ```

    ```bash
    better-compute api metal.enroll --data @enroll-bm-1.json
    better-compute workflows list --node node_… --query "items[].profileId"
    ```

    The same enrollment with flags, the password read from a file (see [from the command line](#from-the-command-line)):

    ```bash
    better-compute metal enroll --site site_… --pool pool_… \
      --bmc-address 'https://192.168.122.1:8000/redfish/v1/Systems/<libvirt domain uuid>' --bmc-driver redfish \
      --bmc-username admin --bmc-password-file ./bmc-password
    ```

    The node is `enrolling`, and the reconciler carries it through `metal-enroll`, `metal-inspect` and
    `metal-commission` to `ready` with no further calls. The example's README also lists `metal.manage`,
    `metal.inspect` and `metal.provide` here: you do not need them after an enrollment. They re-run
    phases of a machine that is already `ready`.
  
      ### Deploy, then release [#deploy-then-release]

    ```bash
    better-compute metal deploy node_… --profile ubuntu-24.04-gpu-host --profile-version 1
    better-compute metal undeploy node_… --wipe quick --yes
    ```

    The deploy ends when the installed OS has joined as the same node and the profile's acceptance
    steps passed: the node is `deployed` with role `host`. The undeploy wipes and powers off, and the
    node is `ready` again only after the wipe is attested.
  
### What the machine sees [#what-the-machine-sees]

In the lifecycle test, enrollment to `ready` sends exactly these commands. BMC work goes to the site
agent, disk and diagnostics work to the machine's ramdisk:

| #  | Command               | Sent by                                                   |
| -- | --------------------- | --------------------------------------------------------- |
| 1  | `bmc.query`           | `power.getPower` (the BMC answers with these credentials) |
| 2  | `bmc.query`           | `inspect.oob` (PXE MACs, SMBIOS UUID, IP addresses)       |
| 3  | `bmc.boot`            | one-time PXE                                              |
| 4  | `boot.prepareRamdisk` | a boot entry with a MAC-bound discovery token             |
| 5  | `bmc.power`           | cycle into the ramdisk                                    |
| 6  | `inventory.refresh`   | in-band inventory                                         |
| 7  | `boot.cleanUp`        | inspection's cleanup; the ramdisk stays up                |
| 8  | `diag.dcgm`           | level 2, node-signed                                      |
| 9  | `disk.wipe`           | quick, node-signed                                        |
| 10 | `bmc.power`           | off                                                       |
| 11 | `boot.cleanUp`        | commissioning's cleanup                                   |

The deploy then sends `deploy.image` (the catalog source, digest, format and compression, plus a
reference to the rendered user data) and `boot.kexec`. The user data travels only as a reference, and
the deploy's ramdisk boot gets its own discovery token.

A machine whose BMC cannot report its MACs (IPMI) and that was enrolled with only its `productUuid` adds a step
before 4 (3d26428, 8de6e2f): `metal.ensureRamdisk` power-cycles it into PXE with no boot entry, a MAC first seen under
its UUID within 10 minutes becomes its candidate, and nothing more is sent until an operator confirms a MAC with
`metal.confirmMac` (`better-compute metal confirm-mac <node> --mac …`, after `metal discovered --site …` shows the
candidate, 002dd95; or **Confirm** on the console's [PXE discovery](/docs/guides/clients/console#pxe-discovery) page,
15c11ee), or at once on a pool with `policy.trustProvisioningNetwork` (whose `inventory.refresh` then checks
the ramdisk's UUID and NICs against the machine). See
[discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines).

### What the ramdisk does [#what-the-ramdisk-does]

`bcd ramdisk` runs these kinds (`daemon/internal/agent/ramdisk.go`) under metal's names. Disks are named by WWN,
else serial, never by kernel name, since kernel names are not stable across boots.

| Kind           | What bcd does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `disk.wipe`    | The listed disks, or every non-removable one. `quick` (the default) zeroes the first and last MiB of each disk, where partition tables (with the GPT backup), filesystem, LVM, RAID and ZFS signatures live, and reads them back. `secure-erase` and `crypto-erase` use the drive's own command: `nvme format --ses=1` or `--ses=2` for NVMe, the ATA security erase through `hdparm` for SATA (no crypto erase there), else `blkdiscard --secure`. The result lists each disk's method and whether it was verified; a disk that was not wiped and verified fails the whole command (`wipe-unverified`), so `ready` rests on a verified wipe. A listed disk that is missing is `disk-missing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `deploy.image` | Picks the root disk from the hint (WWN, serial, device, or a minimum size; otherwise the smallest non-removable disk of at least 4 GiB), streams the image from the site cache by digest when `bc.cache` is set, else from its source URL, through SHA-256 (of the stored bytes) and gzip or zstd straight onto the disk, then re-reads the partition table. A digest mismatch zeroes the disk's ends again (`checksum-mismatch`), so a substituted image never boots. qcow2 is refused (`unsupported-format`). User data is fetched with a `SecretRequest` bound to the step and written as a cloud-init NoCloud seed on the root filesystem, or as Ignition's `config.ign` on the partition labelled `boot`. It then installs the ramdisk's own join configuration into the OS (on the partition holding `/etc`, mode 0600): `/etc/bcd/config.toml` with the gateway, server name, site and the three pins, and `/etc/bcd/join-encryption.jwk`, so the installed `bcd join` needs nothing else from the user data but the token; a failure there is `user-data-failed`. The image is untrusted while it is written into, since it may be a tenant's and a crafted filesystem attacks whatever kernel mounts it. An ext2, ext3 or ext4 partition holding `/etc` (most images' root) is never mounted (1ae1789): bcd writes the NoCloud seed and its join files with e2fsprogs' `debugfs`, which reads and writes the filesystem in userspace, making parent directories, unlinking whatever the image has at a target path (a file, a symlink, a device node) without following it, writing each file root-owned 0600, and reading every file back to check it is a regular file with the bytes written; only plain absolute paths are accepted. Ignition's `config.ign` goes onto the partition labelled `boot` in userspace too (d9a900f): `debugfs` for ext (Fedora CoreOS) and `mtools` for FAT, each read back. Only xfs and btrfs, or a ramdisk without the tools, fall back to a `nodev,nosuid` mount, where every file goes through `os.Root` (a path that resolves outside the mounted filesystem is refused) and whatever is at a target path is replaced with a file created exclusively, never followed or written through; a directory there is refused. The ramdisk image therefore needs e2fsprogs (`debugfs`) and `mtools` |
| `boot.kexec`   | Journals `running{boot_id}`, loads the installed kernel and initrd from the root disk (the given paths, `/boot/vmlinuz` with `/boot/initrd.img`, or the newest `vmlinuz-*` with its `initrd.img-`, `initramfs-` or `initrd-` file). On an ext root it never mounts the image (d9a900f): `debugfs` lists `/boot`, resolves symlinks inside the image (at most 8 hops; an absolute target is the image's own path, never the ramdisk's) and copies the kernel and initrd, regular files of at most 512 MiB, into the ramdisk's tmpfs, which kexec loads; xfs and btrfs are mounted read-only, `nodev,nosuid`. The kernel command line is `root=UUID=<root> ro` plus the extra arguments, and bcd runs `systemctl kexec` (or `kexec -e`). Self-terminating: it completes when the installed OS rejoins                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `diag.dcgm`    | `dcgmi diag -r <level> -j` (levels 1 to 4, default 1), for the listed GPUs or all; `Pass` and `Skip` pass, anything else fails with its warnings as the message. Attested, so the result is node-signed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

`inventory.refresh`, `diag.nvml` and `logs.collect` run in the ramdisk too. Under the `fakehw` build these act on
disk image files, so the Go tests run the same logic on sparse multi-terabyte files.

## Walkthrough: Ubuntu autoinstall over virtual media [#walkthrough-ubuntu-autoinstall-over-virtual-media]

This is walkthrough 1 of PLAN.md §5.6: a third-party plugin, with no change to `@better-compute/*`,
installs Ubuntu with its own installer from an ISO mounted through the BMC, with a single-use
`provisioned` join token inside the autoinstall configuration so `bcd` joins afterwards.

```ts title="acme-autoinstall.ts"
import { createHash } from 'node:crypto';
import { definePlugin, manifest } from 'better-compute/core';
import { defineProfile, defineStep, renderTemplate } from 'better-compute/provision';

const autoinstall = `#cloud-config
autoinstall:
  version: 1
  identity: { hostname: {{hostname}} }
  late-commands:
    - curtin in-target -- sh -c "echo {{joinToken}} > /etc/bcd/join-token && chmod 600 /etc/bcd/join-token"
    - curtin in-target -- systemctl enable bcd
`;

const renderAutoinstallIso = defineStep({
  id: 'autoinstall.renderIso',
  runsOn: 'control-plane',
  timeout: '5m',
  retrySafe: false,
  input: manifest({}),
  async run(_input, context) {
    const node = context.node!;
    const { token } = await context.sql.transaction((tx) =>
      context.pki!.issueJoinToken(tx, { tenantId: 'root', role: 'provisioned', siteId: node.siteId, ttlSeconds: 7200, maxUses: 1 }),
    );
    const userData = renderTemplate(autoinstall, { hostname: node.id.replace(/_/g, '-'), joinToken: token });
    const digest = createHash('sha256').update(userData).digest('hex');
    await publishIso(digest, userData); // your ISO build and storage behind the site cache
    return { url: `https://cache.site-a/autoinstall/${digest}.iso` }; // unguessable, per node
  },
});

const ubuntuAutoinstall = defineProfile({
  id: 'ubuntu-24.04-autoinstall',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'autoinstall.renderIso' },
    { use: 'media.insert', on: 'site', with: { fromStep: 'autoinstall.renderIso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.agentJoined', with: { role: 'node' } },
    { use: 'media.eject', on: 'site' },
  ],
});

export const acmeAutoinstall = definePlugin({
  id: 'acme-autoinstall',
  extensions: [renderAutoinstallIso],
  profiles: [ubuntuAutoinstall],
});
```

Pass the plugin to the workflow engine next to metal, so it resolves the plugin's step and profile:
`provisionModule({ modules: [bareMetal, acmeAutoinstall] })`. `media.insert` with `fromStep` reads the
`url` the render step returned and mounts it through the BMC's virtual media.

**On a BMC without virtual media** (IPMI), this profile is refused before the workflow starts, because a
`fromStep` ISO has no digest the site cache can verify. An ISO registered in the catalog works there (3d26428): name
it with `{ use: 'media.insert', with: { image: 'img_…' } }`, and metal pins it at the site, boots it with iPXE
`sanboot` from the site cache, turns the later `setBootDevice` to `cd` into `pxe`, and has `media.eject` remove the
boot entry. A catalog ISO is the same for every machine, so a per-node join token cannot live inside it. See
[sanboot instead of virtual media](/docs/guides/bare-metal/bmc#sanboot-instead-of-virtual-media).

What the tests assert:

* `profiles.validate` resolves every step of the plugin profile.
* Enroll, `ready`, deploy, `deployed` with role `host`, undeploy, `ready`; the ISO is ejected and the
  token appears only inside the ISO, never in a command.
* The v0.2 acceptance in simulation: 20 consecutive unattended deploy and release loops, a fresh
  single-use token per install, and every transition into `ready` preceded by exactly one attested
  wipe in that workflow. A release whose wipe is not attested ends `failed` with
  `failedFrom: 'releasing'` and maintenance on.

> **Joining as the same node.** 
  In the simulation the installed OS rejoins as the deploying node. The real `provisioned` attestor
  re-keys an existing node only for tokens it can trace to a running workflow, and the resolver that
  ships (`provisionedTargetResolver`) traces tokens minted by `userdata.render`. A token a plugin mints
  itself must at least carry `expect: { productUuid }`, and then joins as a new node.

## Walkthrough: Talos on an agentless pool [#walkthrough-talos-on-an-agentless-pool]

Talos manages itself through its own API and never runs `bcd`. This is walkthrough 3 of PLAN.md §5.6:
a pool with `agent: 'none'`, inventory from Redfish, and a third-party step that reaches the Talos
API through the site agent, because the control plane never talks to machines directly.

```ts title="acme-talos.ts"
import { field, manifest } from 'better-compute/core';
import { httpViaSite } from 'better-compute/metal';
import { defineProfile, defineStep, stepNotReady } from 'better-compute/provision';

const talosReady = defineStep({
  id: 'wait.talosReady',
  runsOn: 'control-plane',
  relays: ['site'],
  sends: ['site.httpProxy'], // the kinds it relays: the workflow's batch scope includes them
  timeout: '20m',
  retrySafe: true,
  input: manifest({ port: field.int({ min: 1, max: 65535 }).default(50000) }),
  async run(input, context) {
    const { rows } = await context.sql.query<{ addresses: string[] }>(
      'SELECT addresses FROM metal_nodes WHERE node_id = $1', [context.node!.id],
    );
    const address = rows[0]?.addresses[0];
    if (!address) throw stepNotReady(10_000);
    const response = await httpViaSite(context, {
      url: `https://${address}:${input.port}/machine/version`,
      tlsSpkiSha256: talosCaSpki, // Talos serves its own CA: pin it
      poll: true, // each call returns the newest answer and sends the next request
    });
    if (response.status !== 200) throw stepNotReady(5_000);
    return { talos: response.json<{ version: string }>().version };
  },
});

const talos = defineProfile({
  id: 'talos-1.8',
  version: 1,
  target: 'metal',
  role: 'host',
  steps: [
    { use: 'media.insert', on: 'site', with: { url: 'https://cache.site-a/talos-1.8-metal-amd64.iso' } },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'cd' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.talosReady' },
  ],
});
```

The pool opts in to what an agentless machine cannot do:

```ts
await compute.api.pools.create(credential, {
  tenantId: 'root',
  siteId,
  class: 'metal-host',
  agent: 'none',
  policy: { allowUnverifiedRelease: true, provisioningCidrs: ['10.30.0.0/16'] },
});
```

* **Enrollment** runs `metal-enroll`, then the out-of-band profiles: Redfish inspection records the
  machine's MACs and IP addresses, and commissioning only powers it off. No ramdisk ever boots.
* **Deploy** mounts the Talos ISO, boots it and polls the Talos API until it answers. The node is
  `deployed` with role `host`, and every command went to the site agent.
* **Release** ejects the media and powers off. It reaches `ready` without an attested wipe only
  because the pool set `allowUnverifiedRelease`, and metal publishes `node.release.unverified`.
  Without it the release fails with `UNVERIFIED_WIPE`.
* **Rescue** needs a ramdisk and fails the node with `UNSUPPORTED`.
* The node never connects, so it is never schedulable for `hold` reservations. PLAN.md has such
  nodes serve `binding: 'node'` leases from v0.5.

### What `site.httpProxy` allows [#what-sitehttpproxy-allows]

The site agent is a trusted proxy on the provisioning network, so `httpViaSite` refuses anything
beyond talking to the machine itself, with `INVALID_INPUT`, before a command exists:

* The host must be an IP literal among the machine's recorded addresses. Names are refused, so DNS
  rebinding cannot redirect it.
* `https` only, unless the address is inside one of the pool's `policy.provisioningCidrs` (IPv4).
* No credentials in the URL, no `Host` or hop-by-hop headers, no line breaks in header values, and
  secret headers only as `secret://` references that the agent resolves and never logs.
* Methods `GET`, `POST`, `PUT` and `DELETE`, with a timeout from 1 to 60 seconds (30 by default).

A redirect is `DRIVER_ERROR` rather than followed, and so is a response over 1 MiB.

`site.httpProxy` is a destructive kind (0d42de8, f66e236): a request can change the machine it reaches, so with
approver keys configured it runs only in an approved batch, under the leaf `node:<target node>`. The step must
declare `sends: ['site.httpProxy']` as above: the engine records the kinds a workflow can send when it starts, and
metal scopes the workflow's batch from them, so with approver keys set a relaying step without it sends commands the
queue refuses at enqueue: `INVALID_INPUT` (not in the batch's scope), or `INVALID_STATE` with reason `batch-required`
when the workflow has no batch at all. See [who creates batches](/docs/operations/pki#who-creates-batches).

`bcd site` enforces its own limits on each request, whatever the command says, and fails anything else with
`forbidden-target`: `http` or `https` without credentials; `GET`, `POST`, `PUT` or `DELETE`; a host that resolves only
to site-internal addresses (private, CGNAT `100.64.0.0/10` or unique-local; never loopback, link-local and so metadata
services, multicast or public addresses), with the connection made to the address it checked, never a second lookup;
no redirects; a timeout of at most a minute (30 seconds by default); the pinned SPKI when given; secret headers
fetched with a `SecretRequest`. It truncates the response at 1 MiB and says so. It never reaches a BMC: every BMC host
it has driven with a `bmc.*` command is remembered with every address it resolved to (`data_dir/bmc-hosts.json`) and
refused, by name and by resolved address, and so is any `/redfish` path and any address in the site agent's own
`[site] bmc_networks` (local configuration the control plane cannot change; `daemon install-script --bmc-networks`
writes it, and `better-compute doctor` warns about site agents without it). Under the v1.0 quorum a request other
than a `GET` is a write, which runs only under a batch leaf bound to its exact payload; readiness polls such as
Talos's are `GET`s.

The host must also belong to the command's `target_node_id` (a2a853d). Metal puts the site's agentless machines
(nodes of `agent: 'none'` pools that are not retired) into the site agent's spec as `SiteSpec.proxy_targets`, each
with the addresses Redfish inspection recorded or `metal.enroll`'s `addresses` gave (3f19859; the only source for an
IPMI machine, whose inspection reports none), and never its BMC host. The list is recomputed when such a machine is
enrolled with addresses, when inspection records addresses and when the machine is retired (8391d74). bcd then requires an IP literal to be one of the target node's listed
addresses, and a name to match a listed name (case-insensitive, trailing dot ignored) or to resolve only to listed
addresses; a target node the list does not name is refused too, all with `forbidden-target`. The check applies once
the site's spec lists any target, which it does whenever the site has an agentless machine.

## From the command line [#from-the-command-line]

The `metal` commands (171d9be) are the metal methods with flags, each decided as the token's identity:

```bash
export BETTER_COMPUTE_TENANT=root
better-compute metal enroll --site site_01k5w… --pool pool_01k5w… --bmc-address 10.20.0.14 \
  --bmc-driver redfish --bmc-credentials secret://root/bmc-r12-14 --macs b8:ce:f6:00:00:01
better-compute metal deploy node_01k5w… --profile ubuntu-24.04-gpu-host --profile-version 1
better-compute metal power node_01k5w… --action cycle --yes
better-compute metal undeploy node_01k5w… --wipe secure-erase --yes
better-compute metal abort node_01k5w… --reason 'wrong profile'
better-compute metal bulk --pool pool_01k5w… --verb retire --nodes-file retire.txt --reason 'EOL rack r12'
```

* **Credentials** are a `secret://<tenant>/<id>` reference of the node's own tenant (`--bmc-credentials`; `vault://`
  and `env://` references only as the root tenant), or `--bmc-username` with `--bmc-password-file`. There is no
  password flag. See [enrolling a machine](/docs/guides/bare-metal/bmc#enrolling-a-machine).
* **Destructive commands ask first**: `metal retire` (the disks are erased and the node is retired for good),
  `metal undeploy` (the installed system and its disks are wiped) and `metal power` with `off`, `reset` or `cycle`
  (power is cut without asking the running system) go ahead only with `--yes`, or after a yes at an interactive
  terminal; in a script or CI job without `--yes` they stop with a usage error before calling the API, as
  `nodes delete` does. `metal bulk` is not asked this way: its `--reason` and the batch's approvers are its check.
* **One verb per node**: `manage`, `inspect`, `provide`, `rescue`, `unrescue`, `recover` and `retire` take only the
  node; `deploy` takes `--profile` and `--profile-version`, `undeploy` takes `--wipe`, `abort` takes `--reason`, and
  `power` takes `--action` (`on`, `off`, `graceful-off`, `reset`, `graceful-restart`, `cycle`, `nmi`).
* **Many nodes at once**: `metal bulk` runs one `--verb` (`manage`, `inspect`, `provide`, `undeploy`, `rescue` or
  `retire`) on the nodes of `--pool` listed in `--nodes` (comma-separated) or `--nodes-file` (one per line,
  deduplicated) as one `metal.bulk` batch. `--reason` is required: it is what the quorum's approvers see when they
  review the batch with [`batch approve`](/docs/guides/clients/cli#destructive-batches). `--wipe` is for
  `--verb undeploy` only.
* **PXE discovery**: `metal discovered --site ID` lists what the site's agents saw PXE-booting from unknown MACs,
  and `metal confirm-mac NODE --mac MAC` binds the MAC of a machine enrolled without one (002dd95), which unblocks an
  IPMI machine waiting in `metal.ensureRamdisk`. The console's
  [PXE discovery](/docs/guides/clients/console#pxe-discovery) page does the same from a browser (15c11ee). See
  [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines).

`workflows list --node`, `workflows get`, `workflows retry --from-step` and `commands list --node` follow what a
verb started. Provision profiles have their own commands (`profiles create` from a JSON document, `validate`,
`render`, `update --visibility`, `delete`). See [command groups](/docs/guides/clients/cli#command-groups).

## Release and retire [#release-and-retire]

* `metal.undeploy` takes `wipe`: `quick` (the default), `secure-erase`, `crypto-erase`, or `none`.
  Every mode but `none` must come back node-signed with every disk's read-back passed before the
  machine is `ready`.
* `metal.retire` works from `ready`, `failed`, `enrolling`, `adopting` or `adopted`, so undeploy a
  deployed machine first. It crypto-erases, powers off and revokes the machine's identity; `retired`
  is final.

  - [Profiles and images](/docs/guides/bare-metal/profiles-and-images): The steps these profiles use, user data templates and the image catalog.

  - [Extension walkthroughs](/docs/extending/walkthroughs): The same walkthroughs from the plugin author's side.

  - [Dedicated-root metal](/docs/guides/bare-metal/dedicated-root): Leasing a whole machine, with root, to one tenant.


# Bare metal (/docs/guides/bare-metal)

> Enroll machines by their BMCs, commission them over PXE, deploy signed images and wipe them on release, all driven by one provision state machine.



Bare metal is how Better Compute takes a server you own from a powered-off box with a
BMC to a running, accepted node, and back again with its disks wiped. Nobody
has to touch the machine: the control plane drives it through the BMC and network boot, and every
step is recorded, retried and resumable.

There are two ways a machine becomes a node:

* **Adopt** (v0.1). The machine already runs an OS. You install `bcd` with a join token, it joins,
  and the pool's acceptance checks move it to `adopted`. No BMC, no reimage. See
  [joining nodes](/docs/guides/nodes/joining).
* **Enroll** (v0.2). You give the control plane the machine's BMC address and credentials. It
  inspects the machine out of band, PXE-boots it into a ramdisk, burns it
  in, wipes it and parks it in `ready`. A deploy then streams an image to disk and the installed OS
  joins as the same node.

This section covers the second path: the `metal` module in `@better-compute/metal` (also
`better-compute/metal`), built on the workflow engine of
`@better-compute/provision`.

> **What runs today.** 
  The control plane side is implemented and tested: the `metal` API group, the metal reconciler, the
  workflows, steps and profiles of every provision state, and the site and ramdisk command kinds. The
  agents that touch hardware are modes of the Go daemon `bcd`, and both exist but are not complete.
  `bcd site` runs BMC operations over Redfish, or through a bcx plugin it hosts such as `bcx-ipmi` (a driver it does
  not host answers `unsupported-driver`),
  `site.httpProxy`, boot entries and the pre-warmed image cache. While it holds the site boot lease it
  answers PXE and UEFI HTTP boot for prepared machines: proxy DHCP (it never hands out addresses),
  TFTP for the iPXE builds, and iPXE scripts over plain HTTP. `bcd ramdisk` runs `disk.wipe`, `deploy.image`, `boot.kexec`
  and `diag.dcgm`, and joins with the boot parameters on its kernel command line. bcd is not released, only its Go
  unit tests run these modes, and nothing has run them against the control plane or a real machine; the iPXE builds
  and the ramdisk image (a kernel and initramfs with `bcd` and its tools) are yours to supply. The test suite
  still drives the whole lifecycle with simulated site agents and ramdisks
  (`tests/extensions/metal-sim.ts`). See [what bcd does today](/docs/daemon#what-bcd-does-today).

## Register the module [#register-the-module]

`metal()` contributes the verbs, the reconciler, its steps, profiles, command kinds and the
`metal/*` migrations. It declares `requires: ['provision']`, and the workflow engine has to know
metal's steps and profiles, so pass the same instance to both:

```ts title="compute.ts"
import { betterCompute } from 'better-compute';
import { metal } from 'better-compute/metal';
import { provisionModule } from 'better-compute/provision';

const bareMetal = metal({
  hardwareProfiles: {
    'gpu-redfish': {
      description: 'Redfish BMC, iPXE boot, image streaming from the ramdisk',
      drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream', inspect: 'agent' },
    },
  },
});

export const compute = betterCompute({
  // database, secret, baseURL, iam and pki as usual
  modules: [
    provisionModule({
      modules: [bareMetal], // the engine resolves metal's steps and profiles
      userData: {
        controlPlaneUrl: 'https://gw.example.com', // the gateway the installed bcd joins through
        bcdUrl: 'https://compute.example.com/api/compute/downloads/bcd/0.2.0/linux-amd64',
        bcdSha256: '<hex digest of that binary>',
      },
    }),
    bareMetal,
  ],
});
```

* `hardwareProfiles` maps a name to per-concern driver bindings. A pool picks one with
  `policy.hardwareProfile`, and `metal.enroll` can override it per node. See
  [profiles and images](/docs/guides/bare-metal/profiles-and-images).
* `verifyAttestation` decides whether a node-signed result (a wipe, a DCGM run) is genuine. The
  default, `nodeSignedResults()`, rebuilds the signing input from the stored command row and checks
  the node's signature against `node_certs`. `acceptIngestedAttestation` exists for tests only.
* The server finds the module that exposes `workflowEngine()` (the provision module) and puts its
  engine on `ctx.workflows`, so you do not wire it yourself.
* The server also gives every step the secret broker (`ctx.secrets`, for BMC credentials and
  rendered user data) and the join-token port (`ctx.pki`, for discovery and `provisioned` tokens).
  For agents to join with those tokens, configure the `pki` option (JoinService) and the attestors
  shown on [sites](/docs/guides/bare-metal/sites).

## The provision state machine [#the-provision-state-machine]

Every metal node has a provision state. The legal moves live in
one table in `@better-compute/core` (`provisionStateTable`), so the adopt path of v0.1 needs no
metal package, and every verb, API method and reconciler checks against the same data. Illegal
moves fail with `INVALID_STATE`.

<Mermaid
  chart="stateDiagram-v2
  [*] --> adopting: join
  [*] --> enrolling: enroll
  adopting --> adopted: complete
  adopting --> adopting: adopt
  adopted --> adopting: adopt
  adopted --> releasing: release
  enrolling --> inspecting: complete
  inspecting --> commissioning: complete
  commissioning --> ready: complete
  ready --> deploying: deploy
  ready --> enrolling: manage
  ready --> inspecting: inspect
  ready --> commissioning: provide
  deploying --> deployed: complete
  deployed --> releasing: release
  deployed --> rescue: rescue
  rescue --> deployed: unrescue
  rescue --> releasing: release
  releasing --> ready: complete
  enrolling --> failed: fail or abort
  inspecting --> failed: fail or abort
  commissioning --> failed: fail or abort
  deploying --> failed: fail or abort
  releasing --> failed: fail or abort
  rescue --> failed: fail
  retiring --> failed: fail
  failed --> enrolling: recover
  failed --> releasing: recover
  failed --> deployed: recover
  failed --> retiring: recover
  adopting --> retiring: retire
  adopted --> retiring: retire
  enrolling --> retiring: retire
  ready --> retiring: retire
  failed --> retiring: retire
  retiring --> retired: complete
  retired --> [*]"
/>

| State           | What is happening                                                                                              | Workflow                              |
| --------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| `adopting`      | A machine that joined with its own OS runs the pool's acceptance checks                                        | none (server)                         |
| `adopted`       | Accepted, running its own OS; schedulable                                                                      | none                                  |
| `enrolling`     | BMC access is verified; out-of-band inspection finds the PXE MACs                                              | `metal-enroll`                        |
| `inspecting`    | The machine boots the ramdisk and reports its inventory in band                                                | `metal-inspect`                       |
| `commissioning` | DCGM level 2, a quick wipe, power off                                                                          | `metal-commission`                    |
| `ready`         | Clean and powered off, waiting for a deploy                                                                    | none                                  |
| `deploying`     | The profile named in `metal.deploy` runs                                                                       | the tenant's profile                  |
| `deployed`      | Running the installed OS; schedulable                                                                          | none                                  |
| `releasing`     | Wipe with the chosen mode, power off                                                                           | `metal-release`                       |
| `rescue`        | The rescue ramdisk is booted on a deployed machine; after `metal.unrescue`, the machine boots back into its OS | `metal-rescue`, then `metal-unrescue` |
| `failed`        | A workflow failed or an operator aborted; `failedFrom` says where                                              | none                                  |
| `retiring`      | Crypto-erase, power off, revoke the identity                                                                   | `metal-retire`                        |
| `retired`       | Terminal: identity revoked, BMC credentials forgotten                                                          | none                                  |

Only `adopted` and `deployed` nodes can be scheduled (`schedulableProvisionStates`).
Maintenance, cordon, power and connectivity are separate axes: a node can be `deployed` and cordoned,
or `ready` and powered on. Pools whose machines never run `bcd` (`agent: 'none'`) use out-of-band
variants of these profiles; see [Talos on an agentless pool](/docs/guides/bare-metal/deploying).

## The metal API group [#the-metal-api-group]

Every method is a mutation decided as `compute:nodes:provision` on the node, except `enroll`
(decided on the pool, since the node does not exist yet), `bulk` (decided on the pool, and each node on its own as
the verb it applies) and `power` (`compute:nodes:power`).

| Method                                           | Verb                                                                           | Legal from                                                            | Moves to                                  |
| ------------------------------------------------ | ------------------------------------------------------------------------------ | --------------------------------------------------------------------- | ----------------------------------------- |
| [`enroll`](/docs/reference/api/metal#enroll)     | creates the node                                                               | none                                                                  | `enrolling`                               |
| [`manage`](/docs/reference/api/metal#manage)     | `manage`                                                                       | `ready`                                                               | `enrolling` (re-verifies the BMC)         |
| [`inspect`](/docs/reference/api/metal#inspect)   | `inspect`                                                                      | `ready`                                                               | `inspecting`                              |
| [`provide`](/docs/reference/api/metal#provide)   | `provide`                                                                      | `ready`                                                               | `commissioning`                           |
| [`deploy`](/docs/reference/api/metal#deploy)     | `deploy`                                                                       | `ready`                                                               | `deploying`                               |
| [`undeploy`](/docs/reference/api/metal#undeploy) | `release`                                                                      | `deployed`, `rescue`                                                  | `releasing`                               |
| [`rescue`](/docs/reference/api/metal#rescue)     | `rescue`                                                                       | `deployed`                                                            | `rescue`                                  |
| [`unrescue`](/docs/reference/api/metal#unrescue) | `unrescue`                                                                     | `rescue`                                                              | `deployed`, once the installed OS is back |
| [`abort`](/docs/reference/api/metal#abort)       | `abort`                                                                        | `enrolling`, `inspecting`, `commissioning`, `deploying`, `releasing`  | `failed`                                  |
| [`recover`](/docs/reference/api/metal#recover)   | `recover`                                                                      | `failed`                                                              | the recover target below                  |
| [`retire`](/docs/reference/api/metal#retire)     | `retire`                                                                       | `adopting`, `adopted`, `enrolling`, `ready`, `failed`                 | `retiring`                                |
| [`power`](/docs/reference/api/metal#power)       | none (power axis only)                                                         | any state but `inspecting`, `commissioning`, `deploying`, `releasing` | unchanged                                 |
| [`bulk`](/docs/reference/api/metal#bulk) (v1.0)  | `manage`, `inspect`, `provide`, `undeploy`, `rescue` or `retire` on many nodes | as that verb                                                          | as that verb                              |

Under the v1.0 [destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10), every workflow's
destructive commands (wipes, deploys, `bmc.*`, `site.httpProxy`) run in a batch approvers signed, and metal creates
those batches itself (f66e236): a workflow gets a one-node batch of its own, `metal.power` a one-leaf batch
(478025a), and `metal.bulk({ tenantId, poolId, verb, nodeIds, wipe?, reason })` applies one verb to up to 10,000
nodes of one pool under a single batch that covers the verb's whole chain, so approvers sign once for the lot. The
`reason` is what approvers read. See [who creates batches](/docs/operations/pki#who-creates-batches).

Because the reconciler chains finished workflows, `manage`, `inspect` and `provide` each run the
rest of the chain back to `ready`: `manage` re-verifies the BMC, re-inspects and re-commissions,
`provide` only re-commissions. Every chain ends with a wipe. To retire a deployed machine, undeploy
it first.

Each method returns the node's metal view, never its credentials:

```ts
const node = await compute.api.metal.deploy(credential, {
  tenantId: 'root',
  nodeId: 'node_01k…',
  profileId: 'ubuntu-24.04-gpu-host',
  profileVersion: 1,
});
// {
//   nodeId, tenantId, siteId, poolId, provisionState: 'deploying', maintenance: false,
//   power: 'off', bmc: { driver: 'redfish', address: 'https://10.0.0.10', credentialsSet: true },
//   macs: ['b8:ce:f6:00:00:01'], workflowId?, failedFrom?, lastError?
// }
```

`metal.undeploy` takes an optional `wipe` (`none`, `quick`, `secure-erase` or `crypto-erase`;
default `quick`), and `metal.abort` an optional `reason` of up to 512 characters. The inputs of
`enroll` and `power` are on [BMC drivers and credentials](/docs/guides/bare-metal/bmc).

### Rescue and unrescue [#rescue-and-unrescue]

`metal.rescue` boots the rescue ramdisk on a deployed machine (`metal-rescue`, one `metal.ensureRamdisk` step), for
repairs from outside the installed OS. `metal.unrescue` brings it back without a reinstall: it starts the
`metal-unrescue` workflow (set the boot device to disk, persistently; power-cycle; wait up to 20 minutes for the
installed OS to rejoin with role `node`; clean up boot entries) and publishes `node.provision.unrescuing`. The node
stays in `rescue` while that runs. The metal reconciler applies `unrescue` and moves it to `deployed` only once the
workflow succeeded; if it failed, the node goes to `failed` with maintenance on, and `metal.recover` takes it back to
`deployed`.

`unrescue` outside `rescue` is `INVALID_STATE`, a second call while the workflow runs returns the node unchanged, and
a call while the rescue ramdisk is still booting is `CONFLICT`. It needs the provision module (`INVALID_CONFIG`
otherwise).

## How a verb becomes work [#how-a-verb-becomes-work]

A verb does not touch the machine. In one transaction it locks the node, checks the move against the
state table, applies it as a guarded update (`WHERE version = $v`, else `CONFLICT`), refreshes the
node's authorization projection, queues the `metal` reconciler for the node and publishes
`node.provision.<state>`.

The metal reconciler (work kind `metal`, sharded by node) is level-triggered:

1. For a node in a workflow-driven state (`enrolling`, `inspecting`, `commissioning`, `deploying`,
   `releasing`, `retiring`, `rescue`), it makes sure exactly that state's workflow runs, started with
   `owner: 'metal'`.
2. When the workflow ends, the engine queues the reconciler again. A succeeded workflow applies
   `complete`; a failed or cancelled one applies `fail`.
3. It repeats (up to eight transitions per run), so `enrolling → inspecting → commissioning → ready`
   chains without a caller. While a workflow runs, it looks again every 30 seconds (`metalPollMs`) in
   case a wake-up was missed.

A finished deploy sets the node's role to the profile's `role` (`host` unless it says `dedicated`).

## Failure and recovery [#failure-and-recovery]

`fail` (a workflow failed) and `abort` (an operator stopped it with `metal.abort`) both land in
`failed`, record `failedFrom`, store the error in `lastError` and turn maintenance on. `metal.abort`
also aborts the running workflow, which still runs its cleanup steps.

`metal.recover` moves a failed node to the last stable point for where it failed, never straight to
`ready`:

| `failedFrom`                               | `recover` goes to                        |
| ------------------------------------------ | ---------------------------------------- |
| `enrolling`, `inspecting`, `commissioning` | `enrolling`                              |
| `deploying`, `releasing`                   | `releasing` (which wipes before `ready`) |
| `rescue`                                   | `deployed`                               |
| `retiring`                                 | `retiring`                               |

* `ready` is reachable only through commissioning or through a release whose wipe was verified. A
  release whose `disk.wipe` result is not node-signed, or whose disks did not all pass read-back,
  fails with `UNVERIFIED_WIPE` instead of reaching `ready`.
* `recover` clears `failedFrom` but leaves maintenance on. Clearing maintenance is a separate
  operator decision, and the API has no method for it yet.
* Leave rescue with `metal.unrescue` to keep the installed OS, or with `metal.undeploy` to wipe and release the
  machine. See [rescue and unrescue](#rescue-and-unrescue).
* `retired` is terminal. `recover` on a retired node is `INVALID_STATE`.

## Events [#events]

Metal publishes into the node owner's tenant:

* `node.enrolled`, then `node.provision.<state>` on every move (`node.provision.ready`,
  `node.provision.failed`, …);
* `node.provision.workflow` when a state's workflow starts, with its `workflowId` and profile;
* `node.provision.unrescuing` when `metal.unrescue` starts its workflow, with the `workflowId`;
* `node.release.unverified` when an agentless pool released without an attested wipe;
* `node.tenant-root.leased`, `.bound`, `.ending` and `.refused` for
  [dedicated-root leases](/docs/guides/bare-metal/dedicated-root).

The workflow engine adds `workflow.*` and `workflow.step.*` events. See
[workflows](/docs/guides/bare-metal/workflows) and [events](/docs/guides/events).

## In this section [#in-this-section]

  - [Sites and site agents](/docs/guides/bare-metal/sites): Boot domains, bcd site, site tokens and the one active boot server per site.

  - [Provisioning workflows](/docs/guides/bare-metal/workflows): Persisted steps on the control plane, site, node or ramdisk; retries, rejoin and cancel.

  - [Profiles and images](/docs/guides/bare-metal/profiles-and-images): Versioned profiles, the ones metal ships, driver bindings, user data and the image catalog.

  - [BMC drivers and credentials](/docs/guides/bare-metal/bmc): Driver concerns, power, sealed BMC credentials and two-phase password rotation.

  - [Deploying machines](/docs/guides/bare-metal/deploying): Walkthroughs: an emulated site, Ubuntu autoinstall and Talos.

  - [Dedicated-root metal](/docs/guides/bare-metal/dedicated-root): Whole machines leased to a tenant with root, and the teardown that takes them back.


# Profiles and images (/docs/guides/bare-metal/profiles-and-images)

> Versioned metal profiles that name steps, drivers, an image and user data; the profiles metal ships for each provision state; and the signed image catalog.



A profile is the recipe for one machine: which steps run, which drivers
do each job, which image goes on the disk, what user data it boots with and
what host configuration the installed OS converges to. Profiles are versioned documents, and a
version becomes immutable the first time a workflow uses it, so you can always tell exactly what
was run on a machine.

## Defining a profile [#defining-a-profile]

`defineProfile(document)` checks a profile and returns a deeply frozen copy. This GPU host profile
follows the one the metal test suite deploys over PXE, with host config, acceptance and cleanup
added:

```ts title="profiles/ubuntu-gpu-host.ts"
import { defineProfile } from 'better-compute/provision';

export const ubuntuGpuHost = defineProfile({
  id: 'ubuntu-24.04-gpu-host',
  version: 1,
  target: 'metal',
  role: 'host',
  image: { ref: 'ubuntu-24.04-server@2026.09', format: 'raw', compression: 'zstd' },
  userData: { kind: 'cloud-init', template: 'bcd-host.yaml.hbs' },
  hostConfig: { 'nvidia-driver': { branch: '580' }, 'fabric-manager': { when: 'nvswitch' } },
  steps: [
    { use: 'userdata.render' },
    { use: 'boot.prepareRamdisk', on: 'site' },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'pxe' } },
    { use: 'power.setPower', on: 'site', with: { action: 'cycle' } },
    { use: 'wait.agentJoined', with: { role: 'ramdisk' } },
    { use: 'deploy.deploy', on: 'ramdisk', timeout: '30m' },
    { use: 'management.setBootDevice', on: 'site', with: { device: 'disk' } },
    { use: 'boot.kexec', on: 'ramdisk', await: 'rejoin', timeout: '15m' }, // completes when the OS joins
  ],
  acceptance: [{ use: 'diag.dcgm', with: { level: 1 } }],
  cleanup: [{ use: 'boot.cleanUp', on: 'site' }],
});
```

<TypeTable
  type="{
  id: { type: 'string', description: 'Lower-case letters, digits, dots, underscores and hyphens; at most 128 characters.', required: true },
  version: { type: 'number', description: 'A positive whole number. Change the document, bump the version.', required: true },
  target: { type: &#x22;'metal' | 'vps'&#x22;, description: 'Metal fields are refused on vps profiles and the other way round.', required: true },
  description: { type: 'string', description: 'At most 1024 characters.' },
  role: { type: &#x22;'host' | 'dedicated'&#x22;, description: 'The role the node takes when a deploy of this profile completes.', default: &#x22;'host'&#x22; },
  entry: { type: &#x22;'adopt' | 'deploy'&#x22;, description: &#x22;An adopt profile may not carry image, userData or steps. Nothing starts adopt-entry profiles yet: adoption acceptance comes from the pool's policy.acceptance.&#x22; },
  drivers: { type: 'Record<concern, string | { driver, config }>', description: 'Drivers for the driver operations this profile invokes (driver.invoke).' },
  image: { type: '{ ref, format?, compression? }', description: &#x22;A catalog reference name@version. format ('raw', 'qcow2', 'iso') and compression ('none', 'zstd', 'gzip') override the catalog's.&#x22; },
  userData: { type: &#x22;{ kind: 'cloud-init' | 'ignition' | 'talos', template }&#x22;, description: 'What userdata.render renders; template names a registered template.' },
  hostConfig: { type: 'Record<unit, settings>', description: 'Config units that become the node desired state after the workflow succeeds.' },
  steps: { type: 'ProfileStep[]', description: 'The main phase, at most 256 steps.' },
  acceptance: { type: 'ProfileStep[]', description: 'Run after the main steps, in the main phase.' },
  cleanup: { type: 'ProfileStep[]', description: 'Run last, whatever the outcome.' },
}"
/>

`defineProfile` rejects unknown fields, malformed steps, ids and durations, values that are not
JSON, and fields that belong to the other target, with `INVALID_CONFIG` whose `details.path` names
the first offending field (for example `steps[2].timeout`). Through the API the same problems are
`INVALID_INPUT`. The step fields (`on`, `with`, `timeout`, `retries`, `await`, `onFailure`,
`fallback`, `when`) are described on [provisioning workflows](/docs/guides/bare-metal/workflows).

## Storing and checking profiles [#storing-and-checking-profiles]

Profiles live in two places: those your code ships through `provisionModule({ profiles })` or a
module's `profiles` (read-only, visible to every tenant, listed under the tenant `builtin`), and
stored ones in the `profiles` table, per tenant and version. A lookup prefers the tenant's own
profile, then a public stored one, then a shipped one.

| Method                                                       | Action                    | What it does                                                                                              |
| ------------------------------------------------------------ | ------------------------- | --------------------------------------------------------------------------------------------------------- |
| [`profiles.create`](/docs/reference/api/profiles#create)     | `compute:profiles:manage` | stores a new version with `visibility` `private` (default) or `public`; an existing version is `CONFLICT` |
| [`profiles.get`](/docs/reference/api/profiles#get)           | `compute:profiles:read`   | one version, the latest when `profileVersion` is left out                                                 |
| [`profiles.list`](/docs/reference/api/profiles#list)         | `compute:profiles:read`   | the tenant's, public and shipped profiles, up to 200 a page                                               |
| [`profiles.validate`](/docs/reference/api/profiles#validate) | `compute:profiles:read`   | `{ valid, issues }`: every `use` that does not resolve and every `with` its step rejects                  |
| [`profiles.render`](/docs/reference/api/profiles#render)     | `compute:profiles:read`   | the expanded steps a workflow of this profile would run                                                   |
| [`profiles.update`](/docs/reference/api/profiles#update)     | `compute:profiles:manage` | replaces an unused version's document, or changes visibility; a used version is `INVALID_STATE`           |
| [`profiles.delete`](/docs/reference/api/profiles#delete)     | `compute:profiles:manage` | hides one version, or all, from new workflows; running workflows keep their steps                         |

Run `validate` before the first deploy: it reports every problem at once with its path, where a
deploy would fail on the first.

```bash
better-compute profiles create ./ubuntu-gpu-host.json --tenant root            # --public for every tenant
better-compute profiles validate ubuntu-24.04-gpu-host --tenant root --profile-version 1
better-compute profiles render ubuntu-24.04-gpu-host --tenant root              # the latest version
```

The `profiles` commands (171d9be) default `--profile-version` to the latest version where the method allows it;
`profiles update` takes `--file` (a replacement document) or `--visibility`, and `--if-version` for the row's
version.

## The profiles metal ships [#the-profiles-metal-ships]

Metal runs one shipped profile per workflow-driven state. Deploying is the exception: it runs the
profile named in `metal.deploy`.

| Profile              | State                                 | Main steps                                                                                                                | Cleanup        |
| -------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------- |
| `metal-enroll@1`     | `enrolling`                           | `power.getPower`, `inspect.oob`                                                                                           | none           |
| `metal-inspect@2`    | `inspecting`                          | `metal.ensureRamdisk`, `inventory.refresh`                                                                                | `boot.cleanUp` |
| `metal-commission@1` | `commissioning`                       | `metal.ensureRamdisk`, `diag.dcgm` level 2, `disk.wipe` quick, `power.setPower` off                                       | `boot.cleanUp` |
| `metal-release@1`    | `releasing`                           | `metal.ensureRamdisk`, `disk.wipe` with the undeploy's mode, `power.setPower` off                                         | `boot.cleanUp` |
| `metal-retire@1`     | `retiring`                            | `metal.ensureRamdisk`, `disk.wipe` crypto-erase, `power.setPower` off (each continues on failure), `metal.revokeIdentity` | `boot.cleanUp` |
| `metal-rescue@1`     | `rescue`                              | `metal.ensureRamdisk`                                                                                                     | none           |
| `metal-unrescue@1`   | `rescue`, started by `metal.unrescue` | `management.setBootDevice` disk (persistent), `power.setPower` cycle, `wait.agentJoined` role `node` (20m)                | `boot.cleanUp` |

Pools whose machines never run `bcd` (`agent: 'none'`) cannot boot a ramdisk, so they get
out-of-band variants: `metal-inspect-oob@1` (`inspect.oob`), `metal-commission-oob@1` (power off),
`metal-release-oob@1` (`media.eject`, then power off) and `metal-retire-oob@1` (power off, then
`metal.revokeIdentity`). Rescue on such a pool fails the node with `UNSUPPORTED`. The two tenant-root
teardown profiles are on [dedicated-root metal](/docs/guides/bare-metal/dedicated-root), and provision
ships `remediate-reboot@1` and `remediate-gpu-reset@1` for health remediation.

## Steps you can use [#steps-you-can-use]

| Step                                  | Runs                          | Input                                                                              | Timeout |
| ------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------- | ------- |
| `power.getPower`                      | relays to the site agent      | none                                                                               | 2m      |
| `power.setPower`                      | relays to the site agent      | `action`: `on`, `off`, `graceful-off`, `reset`, `graceful-restart`, `cycle`, `nmi` | 5m      |
| `management.setBootDevice`            | relays to the site agent      | `device`: `pxe`, `http`, `disk`, `cd`; `persistent`                                | 2m      |
| `inspect.oob`                         | relays to the site agent      | none                                                                               | 5m      |
| `media.insert`                        | relays to the site agent      | `url` (https), `fromStep` or `image` (a catalog `iso`)                             | 30m     |
| `media.eject`                         | relays to the site agent      | none                                                                               | 5m      |
| `boot.prepareRamdisk`, `boot.cleanUp` | relays to the site agent      | none                                                                               | 2m      |
| `metal.ensureRamdisk`                 | control plane                 | none                                                                               | 20m     |
| `inventory.refresh`                   | relays to the ramdisk or node | none                                                                               | 10m     |
| `diag.dcgm`                           | relays to the ramdisk or node | `level` 1 to 4 (default 1)                                                         | 1h      |
| `disk.wipe`                           | relays to the ramdisk         | `mode`: `none`, `quick`, `secure-erase`, `crypto-erase`                            | 6h      |
| `deploy.deploy`                       | relays to the ramdisk         | `image` (defaults to the profile's)                                                | 1h      |
| `boot.kexec`                          | ramdisk, `await: 'rejoin'`    | none                                                                               | 15m     |
| `userdata.render`                     | control plane                 | none                                                                               | 2m      |
| `wait.agentJoined`                    | control plane                 | `role`: `node`, `ramdisk` or `site`                                                | 15m     |
| `wait.generationObserved`             | control plane                 | none                                                                               | 10m     |

* `metal.ensureRamdisk` does nothing when the machine's ramdisk is already connected. Connected means
  since metal last cut the machine's power (`metal_nodes.power_cut_at`: the last `off`, `graceful-off`,
  `reset`, `graceful-restart` or `cycle` a step or `metal.power` sent): a session from before a hard
  power-off still looks connected until the gateway's ping times out, and is not taken for a running
  ramdisk. Otherwise it
  sets one-time PXE, sends the boot entries and power-cycles the machine, then waits for the ramdisk
  to join. A machine known only by its product UUID is first power-cycled into PXE with no entry, so
  its site agent lists it; the step then waits until the MAC [PXE discovery](/docs/guides/bare-metal/sites#discovering-new-machines)
  proposes is confirmed with `metal.confirmMac`, or bound at once on a pool with `policy.trustProvisioningNetwork`
  (3d26428, 8de6e2f).
* `inventory.refresh` checks that the machine that booted is the one enrolled (8de6e2f): a product UUID in the
  ramdisk's `system` section must be a spelling of the enrolled one, and for a MAC PXE discovery bound it waits for
  the ramdisk's `system` and `nics` sections and requires the NICs to include that MAC. A mismatch fails the workflow
  with `DRIVER_ERROR` (reason `machine-identity-mismatch`) before anything is written.
* `diag.dcgm` fails with `DRIVER_ERROR` unless the result is node-signed and every test passed.
* `disk.wipe` succeeds with `attested: false` when the signature or a disk's read-back check fails;
  metal then refuses to release the node to `ready` (`UNVERIFIED_WIPE`).
* `deploy.deploy` resolves the image in the machine owner's tenant (its own image first, then a
  public one), refuses an `iso` (`INVALID_INPUT`), sends `deploy.image` with the source URL, digest,
  format, compression and a user-data reference, and fails with `DRIVER_ERROR` when the digest the
  ramdisk wrote differs from the catalog's.
* `media.insert` with `fromStep` mounts the `url` an earlier step of the same workflow returned: how
  a plugin hands a per-node ISO to the BMC (see [deploying](/docs/guides/bare-metal/deploying)).
* `media.insert` with `image` (3d26428) names an `iso` of the catalog: virtual media mounts its `https://` source,
  and a BMC without virtual media boots it with iPXE `sanboot` from the site cache instead; a later
  `management.setBootDevice` to `cd` then sets `pxe`, and `media.eject` removes the boot entry. See
  [sanboot instead of virtual media](/docs/guides/bare-metal/bmc#sanboot-instead-of-virtual-media).

## Driver bindings from the hardware profile [#driver-bindings-from-the-hardware-profile]

Machines in a pool usually share their BMC and boot hardware, so driver choices live in a hardware
profile rather than on each node:

```ts
metal({
  hardwareProfiles: {
    'gpu-redfish': { drivers: { power: 'redfish', management: 'redfish', boot: 'ipxe', deploy: 'image-stream', inspect: 'agent' } },
  },
});
await compute.api.pools.create(credential, {
  tenantId: 'root', siteId, class: 'metal-host', policy: { hardwareProfile: 'gpu-redfish' },
});
```

At enrollment, `resolveDriverBindings` builds the node's bindings in three layers:

1. The BMC given to `metal.enroll` binds `power`, `management` and `virtual-media`, with its address
   and credential reference as config.
2. The hardware profile (the node's `hardwareProfile`, else the pool's `policy.hardwareProfile`)
   overrides per concern.
3. Per-node overrides win.

An unknown hardware profile is `INVALID_INPUT`. The bindings are stored on the node and served in its
NodeSpec. Metal's own BMC steps use the BMC driver the node was enrolled with; a profile's `drivers`
apply to the driver operations it invokes directly.

## User data [#user-data]

A deployed OS has to join as the same node, so it needs a join token in its first-boot
configuration. `userdata.render` builds that configuration from the profile's `userData`:

1. It mints a single-use `provisioned` join token for the node's pool and site (valid 2 hours, or
   `tokenTtlSeconds`).
2. It renders the named template with the variables `nodeId`, `hostname` (the node id with `_`
   turned into `-`), `joinToken`, `controlPlaneUrl`, `bcdUrl` and `bcdSha256`.
3. It stores the document in the secret store and returns only `{ userDataRef, kind, joinTokenId }`.
   `deploy.deploy` passes the reference, and the ramdisk fetches the document with a SecretRequest.

Two templates ship: `bcd-host.yaml.hbs` (cloud-init) and `bcd-host.ign` (Ignition 3.4.0, built as
JSON so values cannot change its structure). The cloud-init one writes the token to a 0600 file and a
`bcd.service` unit, downloads `bcd`, checks its digest, joins, then starts the node agent:

```yaml title="bcd-host.yaml.hbs"
#cloud-config
hostname: {{hostname}}
write_files:
  - path: /etc/bcd/join-token
    permissions: '0600'
    content: {{joinToken}}
  - path: /etc/systemd/system/bcd.service
    permissions: '0644'
    content: |
      [Unit]
      Description=better-compute node agent
      After=network-online.target bcd-join.service
      Wants=network-online.target
      [Service]
      ExecStart=/usr/local/bin/bcd node
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target
runcmd:
  - [sh, -c, "curl -fsSL '{{bcdUrl}}' -o /usr/local/bin/bcd && echo '{{bcdSha256}}  /usr/local/bin/bcd' | sha256sum -c - && chmod 0755 /usr/local/bin/bcd"]
  - [/usr/local/bin/bcd, join, --server, {{controlPlaneUrl}}, --token-file, /etc/bcd/join-token, --node-id, {{nodeId}}, --serving-ca-pin, {{servingCaPin}}, --join-signing-pin, {{joinSigningPin}}, --join-encryption-pin, {{joinEncryptionPin}}]
  - [systemctl, enable, --now, bcd.service]
```

(The template is a function; the snippet shows its output with placeholders. Without pins the three `--…-pin`
arguments are left out.)

The Ignition one does the same with two units: `bcd-join.service`, a oneshot that runs only while the token file
exists, and `bcd.service` after it. Both run exactly `bcdJoinArgv()` and `bcdNodeArgv()` (exported from
`@better-compute/provision`), which bcd's own tests check against its flags:

* `bcd join --server URL --token-file PATH --node-id ID` joins with the `provisioned` token and removes the file once
  joined. bcd refuses an identity other than the node the OS was installed for, and `--server` supplies the join
  encryption key from the control plane's well-known document when `join-encryption.jwk` is absent (it must match
  the pinned key).
* The ramdisk's `deploy.image` installs its pinned `config.toml` and `join-encryption.jwk` into the OS it wrote, so
  `bcd node` finds its configuration at the default path.
* The three pins are the trust anchors bcd checks the well-known document against: the gateway serving CA, the key
  that signs `JoinResponse`, and the JoinService key the token is sealed to. They come from the host's authenticated
  PKI bundle (`userData.pins`, see below), never from that document, so a forged document (DNS takeover plus a Web
  PKI certificate) cannot swap them. A real bcd refuses to bootstrap without all three.

`controlPlaneUrl` must be an http(s) URL without spaces, quotes or shell and systemd special characters (it is a plain
argument). `bcdUrl` is single-quoted for `sh -c`, so a presigned URL's `&` and `%` are fine; it may not hold quotes,
backslashes or whitespace. `bcdSha256` is 64 lower-case hex digits, and each pin is `sha256/` plus the base64 SPKI
digest. `provisionModule` refuses anything else with `INVALID_CONFIG`, including a partial set of pins.

* Configure it with
  `provisionModule({ userData: { controlPlaneUrl, bcdUrl, bcdSha256, pins?, templates?, tokenTtlSeconds? } })`, where
  `pins` is `{ servingCaPin, joinSigningPin, joinEncryptionPin }` from the host's bundle. Without the URLs and the
  digest, rendering fails with `INVALID_CONFIG`. Custom templates get the pins as the variables of the same names.
* A template is a string with `{{name}}` placeholders or a function of the variables. Every value
  must be a single line without `{{`, so a value can never add lines to the document; a placeholder
  with no value is `INVALID_CONFIG`.
* No template ships for `kind: 'talos'`; bring your own.
* A profile with `userData` whose deploy runs before any `userdata.render` step fails with
  `INVALID_CONFIG`. A re-run after a crash mints a new token; the unused one is single-use and
  expires, so it never becomes a second way in.
* The installed `bcd` joins through the `provisioned` attestor, which maps the token back to the
  running workflow's node (see [sites](/docs/guides/bare-metal/sites) for the wiring).

## Host config [#host-config]

`hostConfig` is not a step: when the workflow succeeds, its units become the node's desired state
(the NodeSpec `host` section), and bcd's host converger applies them and sets drift back on its next
pass (6c2343f). The built-in units, in the order bcd applies them:

| Unit                | Settings                                                                                                                                                |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kernel-modules`    | `load` (module names), `options` (module: `"key=value ..."`)                                                                                            |
| `sysctl`            | sysctl names with scalar values; keys that run programs as root or lower the host's defenses are `INVALID_INPUT`                                        |
| `hugepages`         | `pages2m`, `pages1g`, `transparent` (`always`, `madvise`, `never`), `perNuma`                                                                           |
| `cpu`               | `governor` (`performance`, `schedutil`, `ondemand`, `powersave`), `isolate` (a cpulist, never CPU 0; effective after a reboot)                          |
| `iommu`             | `passthrough`, default true (effective after a reboot)                                                                                                  |
| `nvidia-driver`     | `branch` (three digits), `open`                                                                                                                         |
| `fabric-manager`    | none                                                                                                                                                    |
| `gpu-settings`      | `powerLimitWatts` (1 to 10,000), `ecc` (effective after a reboot)                                                                                       |
| `mig-layout`        | `from: 'pool'` takes the pool's `policy.migLayout`, or a `layout`                                                                                       |
| `gpu-sharing`       | `mode` (`mps` or `timeslice`), `replicas` (1 to 48)                                                                                                     |
| `container-toolkit` | `cdi`, default true                                                                                                                                     |
| `k8s-join`          | `server`, `caCertSha256`, `tokenRef` (a `secret://` reference), `kubernetesVersion`, `nodeLabels`, `taints`, `gpu`, `gpuApi` (`device-plugin` or `dra`) |

Any unit takes `when`, a label expression. The pool can set units too, and a pool's unit replaces the profile's:
`policy.hugepages`, `policy.kernelModules`, `policy.cpu`, `policy.iommu`, `policy.gpu` and `policy.kubernetes`, and
`gpu-sharing` for a pool shared by `mps` or `timeslice`. [Host configuration](/docs/daemon/host-config) describes
what each unit does on the machine, the files it writes, what it refuses and when it reports `RebootRequired`, and
the [host config units reference](/docs/reference/host-units) lists each unit's settings field by field. Add
units with `provisionModule({ configUnits })`; bcd applies only the units it knows, and reports any other as a
terminal failure once the units it knows had their turn.

## The image catalog [#the-image-catalog]

Images are built outside the control plane (mkosi or Packer, as `raw.zst` or `qcow2`) and registered
by digest. The catalog records where the artifact lives, what it is, and whether a trusted signer
vouched for it. It never stores the image itself.

```ts title="register-image.ts"
import { sign } from 'node:crypto';
import { imageSigningInput } from 'better-compute/provision';

const image = {
  ref: 'ubuntu-24.04-server@2026.09',
  kind: 'os',
  format: 'raw',
  compression: 'zstd',
  sha256: '<64 hex digits>',
} as const;
const signature = sign(null, imageSigningInput(image), signingKey).toString('base64'); // Ed25519

await compute.api.images.register(credential, {
  tenantId: 'root',
  ...image,
  sizeBytes: 3_221_225_472,
  source: 'https://images.example.com/ubuntu-24.04-server-2026.09.raw.zst', // https:// or oci://
  signerId: 'platform-2026',
  signature,
});
```

<TypeTable
  type="{
  ref: { type: 'string', description: 'name@version, unique per tenant (a duplicate is CONFLICT).', required: true },
  kind: { type: &#x22;'os' | 'ramdisk'&#x22;, description: 'Ramdisks must be signed by a platform signer.', default: &#x22;'os'&#x22; },
  format: { type: &#x22;'raw' | 'qcow2' | 'iso'&#x22;, description: 'deploy.deploy writes raw and qcow2 to disk; an iso goes through media.insert (virtual media, or sanboot from the site cache).', required: true },
  compression: { type: &#x22;'none' | 'zstd' | 'gzip'&#x22;, description: 'How the artifact is compressed.', default: &#x22;'none'&#x22; },
  sha256: { type: 'string', description: '64 hex digits of the artifact as stored.', required: true },
  sizeBytes: { type: 'number', description: 'The artifact size, for cache planning.' },
  source: { type: 'string', description: 'Where caches pull it from: an https:// or oci:// URL.', required: true },
  signature: { type: 'string', description: 'Base64 signature over imageSigningInput(...). Goes with signerId.' },
  signerId: { type: 'string', description: &#x22;A signer on the tenant's allowlist or the platform's.&#x22; },
}"
/>

* **What is signed.** `imageSigningInput` is the UTF-8 text `bc-image-v1`, then the kind, ref,
  format, compression and lower-case digest, one per line. Sign it with Ed25519, or ECDSA P-256 over
  SHA-256 (IEEE P1363 or DER encoding).
* **Signers.** Allowlists are rows of `image_signers` per tenant, with SPKI PEM keys; the root
  tenant's signers are the platform's. Hosts add and revoke them with `addImageSigner` and
  `revokeImageSigner`; there is no API method. Images a revoked signer verified keep their flag, but no longer
  deploy (below).
* **Verification.** A signature must verify against the named signer of the tenant or the platform,
  else `INVALID_INPUT`. A verified image has `verified: true`. The default check is offline against
  `image_signers`; swap in Sigstore or a KMS with `provisionModule({ imageVerifier })`.
* **Visibility.** Unsigned images stay private to their tenant forever (a table `CHECK` enforces it).
  [`images.promote`](/docs/reference/api/images#promote) makes a verified image public to every tenant;
  on an unsigned one it is `INVALID_STATE`.
* **At deploy time.** Before metal relays `deploy.image`, `verifyImageForUse` re-verifies the image's stored
  signature against the signer allowlist it was verified against at registration. An unsigned image, a revoked
  signer or a tampered catalog row stops the deploy with `INVALID_STATE` (`details.reason` `unsigned` or
  `signature`), and the node fails out of `deploying` without any `deploy.image` sent. Pass the verifier you gave
  `provisionModule({ imageVerifier })` as `metal({ images: { verifier } })` when it is not the default allowlist.
  `images: { allowUnsigned: true }` lets a tenant-private unsigned OS image through, for development only; never a
  ramdisk, and never a signed image whose signature fails.
* **Reading.** `images.get` and `images.list` (filter by `kind`, up to 200 a page) return the tenant's
  own images and public ones. A profile's `image.ref` resolves to the tenant's own image first.

`images.register` and `images.promote` are decided as `compute:images:publish`, `get` and `list` as
`compute:images:read`.

From the command line (d1e83ca), `images register` takes the format as `--disk-format` (`--format` is the CLI's
output flag), and `--signature` and `--signer` only together:

```bash
better-compute images register --tenant root --ref ubuntu-24.04-server@2026.09 \
  --source https://images.example.com/ubuntu-24.04-server-2026.09.raw.zst \
  --disk-format raw --compression zstd --sha256 <64 hex digits> --size 3221225472 \
  --signer platform-2026 --signature "$SIGNATURE"
better-compute images list --tenant root --kind os
better-compute images promote img_01k5w… --tenant root
```

### Site caches [#site-caches]

The catalog also tracks which site caches hold which images (`image_locations`: `pulling`, `ready`,
`failed`, `evicting`). The rows are derived from the site agents' own caches: each agent reports its cache
in the `images` inventory section (`node_image_cache`), a change queues the `provision.site-cache`
reconciler for the site, and `syncSiteImageLocations` marks a catalog image whose digest a live site agent
holds as at the site (`ready` if any agent has it ready, else `pulling` or `evicting`), and drops the
locations no agent holds any more unless a deploy pinned them. `recordImageLocation` and
`touchImageLocation` record reports and use (and pin an image while a deploy streams it), `imagesCachedAt(siteId)` lists the ready images at a site (the
`image-cached` placement scorer reads nodes' own `node_image_cache` rows instead, see
[scheduler plugins](/docs/extending/scheduler-plugins#the-image-cached-scorer)), and `evictionCandidates(siteId, bytes, inUse)` picks the least recently used ready images,
never pinned or in-use ones. What to pre-warm is desired state today: a pool's `policy.prewarmImages` lists
`name@version` refs, and provision puts the ones that verify into its site agents' NodeSpec `site.cacheImages`
(see [sites](/docs/guides/bare-metal/sites)). `bcd site` pulls that list into its cache, verifies each digest before
the copy becomes visible, serves it to the site's ramdisks by digest and reports it in its `images` inventory
section; a ramdisk's `deploy.image` tries the cache first and falls back to the source (see
[the boot server](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site)). It does not pull on demand: an image
no pool pre-warms streams from its source.


# Sites and site agents (/docs/guides/bare-metal/sites)

> Sites are boot and failure domains. A bcd site agent in each one runs proxy DHCP, iPXE, the image cache and BMC access, with one active boot server per site.



A site is a boot and failure domain: a room, a cage or a lab whose machines
share a provisioning network and a BMC network. Everything the control plane does to a machine's
hardware (power, boot device, network boot, image streaming) goes through a
site agent on that site, because the control plane never dials into a
site. Pools, nodes and join tokens all belong to exactly one site.

## Sites [#sites]

A site has an owner, a kind and a region:

| Kind        | Owner                      | Created under          | Use it for                                   |
| ----------- | -------------------------- | ---------------------- | -------------------------------------------- |
| `shared`    | the root (platform) tenant | `compute-fleet/global` | platform hardware shared through offerings   |
| `dedicated` | the root tenant            | `compute-fleet/global` | platform hardware set aside for one customer |
| `custom`    | the tenant                 | `compute-tenant/{t}`   | a tenant's own hardware (bring your own)     |

Every `sites.*` mutation is decided as `compute:sites:manage`. Creating a `shared` or `dedicated`
site with a `tenantId` other than the root tenant is `INVALID_INPUT`.

```ts
const site = await compute.api.sites.create(credential, {
  tenantId: 'root',
  kind: 'shared',
  region: 'eu-central-1', // a region label
  iamRegion: 'eu-central', // optional: the IAM region name, when it differs
  idempotencyKey: 'site-fsn1',
});
// { id: 'site_…', ownerTenantId: 'root', kind: 'shared', region: 'eu-central-1', iamRegion: 'eu-central', version: 1 }
```

The id derives from the tenant, the caller and the idempotency key, so a retry returns the same site
and a different request under the same key is `CONFLICT`. The site's `region`, `iamRegion` and
`kind` are the attributes IAM conditions see on `compute-site`.

* [`sites.update`](/docs/reference/api/sites#update) (v0.2) changes `region` and `iamRegion`, or clears
  `iamRegion` with `clearIamRegion: true`. Pass the `version` you read to get `CONFLICT` instead of a
  lost update.
* [`sites.delete`](/docs/reference/api/sites#delete) (v0.2) refuses with `INVALID_STATE` while the site
  still has live pools or nodes. Otherwise it revokes every join token of the site (site tokens
  included), expires its open join requests, drops its boot lease, and removes the row, or tombstones
  it when anything ever referenced it. Deleting a tombstoned site again answers `deleted: false`.

## The site agent [#the-site-agent]

A site agent is `bcd` running in `site` mode on a machine next to the fleet, one or two per site. To
the control plane it is a node like any other: a `nodes` row with `role = 'site'`, no pool, a
`site` certificate and a session through the gateway. It runs the site's commands:

| Duty                                                                                    | When                                                      | In `bcd site` today                                                                                                                                                                                                              |
| --------------------------------------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proxy DHCP, so booting machines find the iPXE binary without replacing your DHCP server | only while holding the boot lease                         | yes, with `[boot] proxy_dhcp`                                                                                                                                                                                                    |
| TFTP and iPXE scripts, one per machine being booted                                     | only while holding the boot lease                         | yes: read-only TFTP for the iPXE builds, scripts over plain HTTP                                                                                                                                                                 |
| Vouching for ramdisks that PXE-booted from it (signed discovery assertions)             | only while holding the boot lease                         | yes                                                                                                                                                                                                                              |
| An image cache that ramdisks stream from                                                | always (a boot's grant only while holding the boot lease) | yes, by digest under a grant (`GET /cache/t/<grant>/<sha256>`, b7ed8a2)                                                                                                                                                          |
| BMC access (Redfish, IPMI, vendor drivers) on the management network                    | always                                                    | Redfish built in; IPMI power, boot device and inspection through the hosted `bcx-ipmi` plugin (51806ab, 7818f19), which `daemon install-script --install-plugins bcx-ipmi` installs from the release (adf2ae9); no vendor driver |

The command kinds it accepts are role-restricted to `site`: `bmc.query`, `bmc.boot`, `bmc.power`,
`bmc.rotate`, `bmc.firmware`, `bmc.storage`, `boot.prepareRamdisk`, `boot.cleanUp` and `site.httpProxy`, plus
the shared `inventory.refresh`, `logs.collect` and `node.reboot`. Their payloads are messages of
`bettercompute.site.v1`, which stays a draft package until v0.2 closes. Every `bmc.*` kind but `bmc.query`, and
`site.httpProxy` since 0d42de8, is destructive, so under the v1.0 quorum it runs only in an approved batch;
`site.httpProxy` also never reaches a BMC (the agent refuses the BMC hosts it drives, any `/redfish` path and its
local `[site] bmc_networks`), and reaches only the addresses of the agentless machine it names (`SiteSpec.proxy_targets`,
which metal fills from inspection; a2a853d, 8391d74). See
[BMC drivers and credentials](/docs/guides/bare-metal/bmc) and the
[command kinds reference](/docs/reference/command-kinds).

The agent's desired state is the `site` section of its NodeSpec (`SiteSpec`): the boot lease state (below) and,
since v0.2, `cacheImages`, the verified catalog images its cache pre-warms (`url`, `sha256`, `format`). The
provision module fills that list: every live pool of the agent's site can name images in `policy.prewarmImages`
(`name@version` refs, resolved in the pool owner's catalog), and only images that still verify (signed, signer not
revoked, row untampered) are listed, deduplicated by URL. The agent pulls each one and re-verifies its digest before
serving it. `images.register` and `images.promote` recompute the specs of the site agents whose pools name the ref,
and so does `pools.update` when `prewarmImages` changes. Modules contribute `site` fields independently, because
the section merges field by field (see [NodeSpec sections](/docs/extending/modules#nodespec-sections)).

> **Status.** 
  `bcd site` exists, and its contracts are in `proto/bettercompute/site/v1`. It runs BMC operations over
  Redfish, or through a bcx plugin it hosts for any other driver (such as `bcx-ipmi` for `ipmi`; see
  [IPMI through bcx-ipmi](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi); a driver it does not host answers
  `unsupported-driver`), `site.httpProxy` and boot entries. While it holds the boot lease,
  its boot server answers prepared machines only: proxy DHCP (boot server and file, never an address), TFTP for the iPXE
  builds, iPXE scripts and discovery assertions over plain HTTP (see [the boot server](#the-boot-server-in-bcd-site)).
  Its cache pre-warms `cacheImages`, serves them to the site's ramdisks by digest and reports them in its `images`
  inventory section, which becomes the site's `image_locations`. bcd is not released, and only its Go unit tests run
  these parts (against a fake Redfish BMC, a scripted session and real sockets): no test runs `bcd site` against the
  control plane, and no real site has run it. The dev stack joins site agents as FakeDaemons with role `site`
  (`createDevStack().joinSite()`), and the metal test suite answers site commands with a simulated agent.

## Enroll site agents with site tokens [#enroll-site-agents-with-site-tokens]

A site agent joins with a site token: a join token of role `site`, bound to one site and no pool.
[`siteTokens.create`](/docs/reference/api/site-tokens#create) is decided as `compute:sites:manage` on
`compute-site/{siteId}`, and only the site's owner can mint one.

```ts
const minted = await compute.api.siteTokens.create(credential, {
  tenantId: 'root',
  siteId: site.id,
  ttlSeconds: 3600, // 60 to 86 400
  maxUses: 2, // how many agents may enroll with it: 1 (default) to 16
  idempotencyKey: 'fsn1-site-agents',
});
// { id: 'jt_…', siteId, role: 'site', usesLeft: 2, expiresAt, token: 'bcjt_<id>.<secret>' }
```

* The bearer (`bcjt_<id>.<secret>`) is in the answer that created the token and nowhere else: only
  its hash is stored, and a replay of the same request omits it. The same key for another site is
  `CONFLICT`.
* A deleted site is `INVALID_STATE`; a site the tenant does not own is `NOT_FOUND`.
* [`siteTokens.revoke`](/docs/reference/api/site-tokens#revoke) takes `tenantId`, `siteId` and
  `tokenId`. It revokes only site tokens: a node join token is `NOT_FOUND` there.
* Both publish security events: `security.site-token.created` and `security.site-token.revoked`.

From the command line (573b0b0), where `--out` writes the one-time token to a new 0600 file instead of printing it
and `revoke` refuses a pasted token (anything with a `.`), so the secret is never sent as an id:

```bash
better-compute sites create --tenant root --kind shared --region eu-central-1 --iam-region eu-central --key site-fsn1
better-compute site-tokens create --tenant root --site site_01k5w… --ttl 3600 --max-uses 2 --out ./site-token
better-compute site-tokens revoke jt_01k5w… --tenant root --site site_01k5w…
```

`site-tokens create --ttl` is in seconds (60 to 86,400).

The agent joins through the same Join RPC as any daemon (see [joining nodes](/docs/guides/nodes/joining)),
and JoinService issues it a 24-hour certificate with role `site`. With bcd that is
`BCD_JOIN_TOKEN=bcjt_... bcd join --role site`, then `bcd site`; see
[installing bcd](/docs/daemon/install#a-site-agent).

## One active boot server per site [#one-active-boot-server-per-site]

Two agents answering DHCP for the same machine would race, and a stale agent vouching for a ramdisk
would be a way in. So exactly one agent per site is the boot server: the holder of the lease
`site:<siteId>:boot`.

* The lease has a 15-second TTL and a fencing epoch that grows on every change of holder.
* The SiteBoot controller renews it while its holder is a connected site agent. The holder learns
  that it is active, and at which epoch, from the `site` section of its NodeSpec (`BootLeaseState`).
* A holder that loses its session keeps the lease until it expires and self-fences after the TTL: it
  stops answering proxy DHCP, iPXE and discovery. Only then is the first connected standby (by node
  id) promoted, under a new epoch. A live lease never changes hands, so two agents can never both
  believe they are active at the same epoch.
* When no agent is left, the lease is released and the epoch still moves on.
* Every discovery assertion carries the epoch the agent held, and the control plane refuses an epoch
  that is not the lease's current one.

The lease logic is the server's SiteBoot controller (`tickSiteBoot`). `betterCompute()` registers it as the built-in
`site-boot` module: a singleton reconciler whose `tick` key runs every 5 seconds in the `controllers` role.

A failover does not restart a machine that was already booting. BMC operations are reroutable between a site's
agents: a power cycle the dead agent already ran is reused, not sent again, and one it never received is moved to
the new agent. Boot entries stay per agent, so the new lease holder gets its own entry (and its own single-use
discovery token) and serves the waiting machine. `tests/metal-site-failover.test.ts` kills the active agent right
after it power-cycled a machine: the standby is promoted when the dead agent's lease runs out, never earlier, the
machine boots from it within 30 seconds, the deploy completes, and the boot step's BMC commands ran once. With both
agents down the boot waits, and the first agent back serves it.

### Where site steps go [#where-site-steps-go]

`siteAgentFor(nodeId)` picks the agent for a machine's site steps: a connected site agent of the
machine's own site, the boot-lease holder first, otherwise the first connected agent by id. A machine's
site steps never go to another site's agent. When no agent is connected, a relaying step waits and
retries until its timeout, and a step that runs on `site` directly fails with `NO_SITE_AGENT` at its
deadline. PLAN.md also routes BMC and image-cache work to any agent that advertises the driver; that
is not implemented.

## Booting a machine into the ramdisk [#booting-a-machine-into-the-ramdisk]

Metal never lets a machine boot something it did not prepare. Before each network boot, the
`metal.ensureRamdisk` step sets a one-time PXE boot through the BMC, sends the site agent one boot
entry per known PXE MAC, and power-cycles the machine:

<Mermaid
  chart="sequenceDiagram
  participant W as Worker
  participant S as Site agent
  participant M as Machine
  participant J as JoinService
  W->>S: bmc.boot, one-time PXE
  W->>S: boot.prepareRamdisk, one entry per MAC
  W->>S: bmc.power, cycle
  M->>S: DHCP and iPXE script request
  S-->>M: iPXE script with the discovery token
  M->>J: Join with the token and a signed assertion
  J-->>M: ramdisk certificate, at most 2 h
  W->>M: inventory, diagnostics, wipe, image"
/>

* Each entry (`ProxyDhcpEntry`) names the MAC, the SMBIOS product UUID when known, the boot
  architecture (`BIOS`, `X86_64_EFI` or `ARM64_EFI`, from the node's `arch` and `uefi`), the workflow
  and a `notAfter` two hours out.
* Each entry carries a fresh single-use `discovery` join token bound to that MAC (`expect.mac`), valid
  for 900 seconds. The token is stored in the secret store and the entry carries only its reference;
  the site agent fetches the bearer with a SecretRequest for that workflow step and renders it into
  that MAC's iPXE script. No command body ever holds the token.
* A token is minted only when its entry's command is first sent, not on every poll, and the cleanup
  step `boot.cleanUp` removes the entries and deletes the stored tokens.
* A machine with no known MAC fails with `INVALID_STATE`: inspect it out of band first. The one exception
  (3d26428) is a machine enrolled with a usable `productUuid` and no MACs: `metal.ensureRamdisk` PXE-boots it once
  with no entry so a site agent lists it, and goes on once the MAC [discovery](#discovering-new-machines) proposes
  is confirmed with `metal.confirmMac` (8de6e2f), or bound at once on a pool that trusts its provisioning network.

### The boot server in bcd site [#the-boot-server-in-bcd-site]

`bcd site` keeps the boot entries it receives in `data_dir/boot-entries.json`, keyed by MAC, so a restart keeps
answering; `boot.cleanUp` drops a workflow's entries (or the listed MACs), and lapsed entries are dropped on their
own. With a `[boot]` section in its config (see [installing bcd](/docs/daemon/install#a-site-agent)) it serves them
in three stages:

<Mermaid
  chart="sequenceDiagram
  participant M as Machine firmware
  participant D as Site DHCP server
  participant S as bcd site
  M->>D: DHCP DISCOVER (PXEClient)
  D-->>M: an address
  S-->>M: proxy DHCP: boot server and iPXE file, no address
  M->>S: TFTP undionly.kpxe or ipxe.efi
  M->>S: DHCP again, now as iPXE
  S-->>M: proxy DHCP: /boot/chain.ipxe
  M->>S: GET /boot/ipxe?mac=&uuid=&ip=
  S->>S: SecretRequest for the discovery token, sign the assertion
  S-->>M: iPXE script: kernel, initrd, bc.* parameters"
/>

1. **Proxy DHCP** (UDP 67 and 4011, `proxy_dhcp = true`) answers `PXEClient` and `HTTPClient` requests beside your
   DHCP server and never hands out an address. It reads the MAC, the client architecture (option 93) and whether the
   client is iPXE already. Firmware gets the iPXE build for its architecture from TFTP at `server_ip`:
   `undionly.kpxe` for BIOS, `ipxe.efi` for x86-64 UEFI, `ipxe-arm64.efi` for arm64 UEFI. UEFI HTTP boot clients get
   the same build as a URL, `<url>/boot/files/<file>`. iPXE itself gets `<url>/boot/chain.ipxe`, which chains to
   `/boot/ipxe` with its MAC, SMBIOS UUID and IP.
2. **TFTP** (`tftp`, `tftp_root`) is read-only, octet mode, with block size and transfer size negotiation, and serves
   nothing outside its root. You supply the iPXE builds. Since 4d8b493 it runs at most 64 transfers at once and 4
   per source address; a request beyond either bound is dropped unanswered (answering would take a socket too), and
   the firmware retries.
3. **`GET /boot/ipxe`** renders the machine's script: the site's ramdisk `kernel` and `initrd`, and on the kernel
   command line the `bc.*` boot parameters `bcd ramdisk` joins with (the gateway, pins and join key, the machine's
   discovery token and a signed assertion, and `bc.cache` when `url` is set), then `cmdline`.

Proxy DHCP answers, and `/boot/ipxe` renders a script, only when both hold:

* the MAC has a live boot entry; an unknown machine gets no proxy DHCP answer and a 404 script that exits, so it
  falls through to its next boot device, and is listed as discovered (see
  [discovering new machines](#discovering-new-machines));
* the agent is the active boot server: its `site` section says it holds the lease at a non-zero epoch, and it has
  heard from the gateway within the lease TTL (15 seconds when the section gives none). A partitioned agent fences
  itself this way; its `/boot/ipxe` answers 503 with a script that waits 10 seconds and reboots.

The discovery token is fetched with a `SecretRequest` bound to the entry's workflow the first time the machine asks
for its script and reused if it asks again; while it is not available the script waits and retries. Each script
carries a fresh `DiscoveryAssertion` (the site, this agent's node id, the lease epoch, the MAC, the IP from the query
or the connection, the product UUID from the query or the entry, the workflow, `issued_at` in server time, and a
`not_after` at most 15 minutes out and never past the entry's), signed by the agent's node key under purpose
`discovery-assertion`.

**Sanboot of a cached ISO** (0056e1e, e763001). For a BMC without virtual media, such as `ipmi`, a boot entry may
carry `ProxyDhcpEntry.sanboot_sha256` instead of a ramdisk boot: `/boot/ipxe` then answers
`sanboot --no-describe <url>/cache/t/<grant>/<sha256>`, and iPXE reads the ISO from the image cache with HTTP range
requests. The ISO boots as it is, with no discovery token and no `bc.*` parameters. The agent serves it only when
the site spec names that digest as an `iso` cache image and the verified copy is cached. Otherwise the machine waits
(a 503 script that sleeps 30 seconds and chains again, never the ramdisk), and the `site` section fails naming the
entry. Since 3d26428 metal sets `sanboot_sha256`: a `media.insert` step that names a catalog ISO (`image`) on a
driver without virtual media pins the ISO at the site and, once the site's cache holds it verified, sends a boot
entry for it (see [sanboot instead of virtual media](/docs/guides/bare-metal/bmc#sanboot-instead-of-virtual-media)).

The image cache is served by the same HTTP server, and since b7ed8a2 only under grants: every cache URL the agent
hands out has the form `GET /cache/t/<grant>/<sha256 hex>`, with an unguessable 128-bit grant, and any other path,
a plain `/cache/<sha256>` included, is 404. The cache holds tenants' deploy images, so nothing is served to whoever
asks:

| Grant                 | Handed out as                                                  | Serves                                                                                                   | Valid                                                                                                                               |
| --------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| A boot                | The ramdisk's `bc.cache` in its iPXE script, and a sanboot URL | Any cached image (`bc.cache`; the ramdisk checks the digest its command names) or that one ISO (sanboot) | Until the boot entry's `not_after` (6 hours for an entry without one), and only while the agent holds the boot lease: otherwise 503 |
| A BMC firmware update | The URL the BMC fetches the firmware image from                | That one digest                                                                                          | 2 hours                                                                                                                             |

The agent pulls every `cacheImages` entry of its `site` section in the background into
`data_dir/cache/<sha256>`, and a copy becomes visible only after its bytes match the digest. The `site` section
stays `PROGRESSING` while images warm and fails for an image whose bytes do not match. With `[boot] cache_max_bytes`
set, each pass trims the cache to that budget, least recently used first, and never evicts an image the section
still names (the pools' pre-warm images plus every image a deploy at the site is streaming). A ramdisk booted with
`bc.cache` tries the cache first for `deploy.image` and falls back to the image's source URL; the digest is checked
on the stream either way, so a cache can never substitute an image. The boot server caps request headers at 8 KiB
(0442561), since any machine on the segment can reach it.

### The discovery attestor [#the-discovery-attestor]

The ramdisk joins with the discovery token plus the site agent's `SignedDiscoveryAssertion` that it
served this MAC. The `discovery` attestor admits it only when all of these hold:

* the token is a live, single-use `discovery` token for the same site, and its `expect.mac` matches;
* the vouching agent is a site agent of that site, holds the live boot lease, and the assertion's
  epoch is the lease's current epoch;
* the assertion is signed by that agent's key under a `site` certificate, at most 15 minutes old, and
  used for one join only;
* the ramdisk's claimed MACs include the asserted MAC, and its product UUID matches when asserted.

The result is a ramdisk certificate valid for at most two hours and
restricted to the commissioning command kinds. When the assertion names a running workflow, the
ramdisk joins as that workflow's node, so a known machine re-commissions as itself.

Hosts wire the attestors with provision's resolvers:

```ts title="compute.ts"
import { discoveryAttestor, provisionedAttestor } from 'better-compute/pki';
import { discoveryWorkflowNodeResolver, provisionedTargetResolver } from 'better-compute/provision';

export const compute = betterCompute({
  // …
  attestors: [
    discoveryAttestor({ resolveWorkflowNode: discoveryWorkflowNodeResolver() }),
    provisionedAttestor({ resolveTarget: provisionedTargetResolver() }),
  ],
});
```

Both resolvers answer only for workflows that are still running, so a token or assertion from a
finished or aborted workflow cannot re-key a node that has moved on. See
[attestation](/docs/guides/nodes/attestation) for the other attestors.

## Discovering new machines [#discovering-new-machines]

Today a machine enters metal by its BMC: `metal.enroll` with the BMC address, then the `inspect.oob`
step asks the BMC for the machine's PXE MACs, SMBIOS UUID and IP addresses and records them. That is
what makes the boot entries above possible.

PXE discovery closes the gap for a BMC that cannot name the machine's NICs, such as IPMI. It never enrolls anything:
it lets metal learn the PXE MAC of a machine that is already enrolled.

**The site agent's side** (0056e1e, e763001). A PXE or HTTP-boot DHCP request, or an iPXE script fetch, from a MAC
without a live boot entry, while the agent holds the boot lease, is never answered but listed in
`StatusReport.discovered`:

| Field                              | Value                                                           |
| ---------------------------------- | --------------------------------------------------------------- |
| `mac`                              | The machine's MAC                                               |
| `product_uuid`                     | The SMBIOS UUID from DHCP option 97, the one a BMC also reports |
| `arch`                             | The client architecture                                         |
| `first_seen`, `last_seen`, `count` | When it was first and last seen, and how often                  |
| `relay`                            | The DHCP relay (`giaddr`) it came through                       |

The list holds at most 256 machines (the oldest `last_seen` goes first), forgets each a day after it was last seen,
and drops a machine once it gets a boot entry. A new machine asks for a status report at most once every 30 seconds.
Since 0442561 the agent takes only well-formed fields from an unknown client, so none can grow the status report past
the frame limit: a MAC must parse, a UUID is kept only in canonical 36-character form, an arch only as
`[a-z0-9_-]{1,32}`, and a relay only as an IP address. While the agent runs proxy DHCP, an iPXE script fetch counts
only for a MAC that DHCP already showed, since anyone can fetch a URL.

**The control plane's side** (3d26428, 8de6e2f). Discovery data is unauthenticated DHCP: any host on the provisioning
network can claim any MAC and any UUID. So storing the list binds nothing, and a discovered MAC boots a machine only
after an operator confirms it.

* **Storing.** `metal()` declares `metalDiscoveryStatusHandler()` as one of its module status handlers
  (`ComputeModule.statusHandlers`, 8de6e2f), and `better-compute serve` and the dev stack run every registered
  module's handlers after the gateway's XID health handler and before the configuration's own `reports.status`. A host
  that builds its gateway with `createGateway` itself adds it to `reports.status`. It stores lists from live site
  agents only (other nodes' roles are cached per process for 10 minutes, so a host's reports cost no query), in
  `metal_discovered` (migration `metal/0007_discovered`), each agent's list replaced whole and written only when its
  digest changes (`metal_discovery_reports`).
* **Reading.** [`metal.discovered`](/docs/reference/api/metal#discovered) (`{ tenantId, siteId }`, decided as
  `compute:sites:read` on the site) lists the machines seen in the last 24 hours, merged over the site's agents,
  newest first, at most 256: `mac`, `productUuid`, `arch`, `firstSeen`, `lastSeen`, `count` (summed over agents),
  `relays`, `agentNodeIds`, `nodeId` when an enrolled machine has that MAC, and `candidateFor` when it is a machine's
  candidate awaiting confirmation. Only the tenant that owns the site sees it; any other tenant gets `NOT_FOUND`,
  since the list describes the site's network.
* **Matching, only right after metal's own power cycle.** Only `metal.ensureRamdisk` matches, for a machine enrolled
  with a usable product UUID and no MACs, right after it power-cycled the machine into PXE with its BMC. A MAC becomes
  the machine's candidate when all of these hold: it was first seen within 10 minutes after that power command was
  sent (with 30 seconds of clock slack before it); it is the only MAC the UUID was seen from in the last 24 hours, in
  any of three spellings (as given, with the first three fields byte-swapped as SMBIOS stores them, or fully reversed
  as IPMI's Get System GUID prints it); no other machine of the site shares the UUID; and no enrolled machine has the
  MAC. Firmware placeholder UUIDs (all zeros, all ones, `03000200-0400-0500-0006-000700080009` and
  `00020003-0004-0005-0006-000700080009`) never match. A new candidate publishes `node.pxe-mac.discovered` (`nodeId`,
  `mac`, `productUuid`, `siteId`, `agentNodeId`, `bound`), and the machine's metal view shows it as `discoveredMac`.
* **Confirming.** The candidate gets no boot entry and no discovery token until an operator binds it with
  [`metal.confirmMac`](/docs/reference/api/metal#confirmmac) (`{ tenantId, nodeId, mac }`, decided as
  `compute:nodes:provision` on the node), which takes the candidate or any MAC the operator knows to be the machine's.
  It publishes `node.pxe-mac.confirmed` (`nodeId`, `mac`, `discovered`, `requestId`) for the audit trail, answers
  `INVALID_STATE` for a node that already has MACs or is retired and `CONFLICT` for a MAC another machine of the site
  has, and wakes the reconciler and the workflow, so the waiting `metal.ensureRamdisk` goes on from that MAC. While
  it waits, the step reports why (`waiting for metal.confirmMac`, or the reason there is no candidate: `not-seen`,
  `ambiguous`, `shared-uuid`, `mac-enrolled`).
* **How long it waits.** Once there is a candidate, or a finding only an operator can settle (`ambiguous`,
  `shared-uuid`, `mac-enrolled`), the wait is yours and not the step's: it runs outside `metal.ensureRamdisk`'s
  20-minute timeout, for up to 7 days from the discovery power cycle. The node keeps its provision state, the
  machine keeps asking for PXE, and nothing power-cycles it again. The step's 20 minutes count again from the
  confirmation. A machine that discovery never saw (`not-seen`) still fails after 20 minutes, since that is a
  cabling, firmware or network fault rather than a pending decision.

> **policy.trustProvisioningNetwork.** 
  A pool whose policy sets `trustProvisioningNetwork: true` binds the candidate at once, without `metal.confirmMac`
  (`macSource: 'discovered'`). Use it only on a provisioning network no untrusted host can reach: any host on it can
  claim a machine's UUID from its own MAC and boot that machine's ramdisk with its discovery token. As a backstop,
  `inventory.refresh` then waits for the ramdisk's own `system` and `nics` sections, received after it connected, and
  fails the workflow with `DRIVER_ERROR` (reason `machine-identity-mismatch`) before anything is written to the machine
  unless the reported product UUID is a spelling of the enrolled one and the NICs include the MAC. For MACs from any
  other source, a UUID present on both sides is still compared.

The metal view records where a machine's MACs came from as `macSource` (`operator` for enroll and `metal.confirmMac`,
`inspection`, or `discovered`; migration `metal/0008_mac_source`). From the CLI (002dd95):

```bash
better-compute metal discovered --site site_01k5w… --format table
better-compute metal confirm-mac node_01k5w… --mac 3c:ec:ef:12:34:56
```

| Command                  | Flags                                                                       | Calls                                                                                                                                                                                       |
| ------------------------ | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metal discovered`       | `--site ID` (required), `--tenant`                                          | `metal.discovered`: what the site's agents saw PXE or HTTP boot from MACs with no boot entry, with the machine each is a candidate for; nothing in it is trusted or bound                   |
| `metal confirm-mac NODE` | `--mac MAC` (required, `aa:bb:cc:dd:ee:ff`; `-` also separates), `--tenant` | `metal.confirmMac`: the operator's decision that `--mac` is the machine's PXE MAC, usually the candidate `metal discovered` shows; a malformed MAC is a usage error before anything is sent |

In the console, the **PXE discovery** page (`/discovery`, 15c11ee) shows the same list for a site you pick, with a
**Confirm** button on each candidate and a form to bind a MAC by hand; both ask in a browser confirmation before
calling `metal.confirmMac`, so nothing is bound without the operator's click. See
[the console](/docs/guides/clients/console#pxe-discovery). See [IPMI](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi)
for the enrollment that uses this.

The discovery attestor also supports operator approval:

* `approval: 'never'` (the default) admits as described above;
* `approval: 'unknown-machines'` defers a join that no running workflow names;
* `approval: 'always'` defers every discovery join.

A deferred join becomes a pending join request (24 hours by default) that an operator decides with
[`joinRequests.approve`](/docs/reference/api/join-requests#approve) or `joinRequests.reject`; the
ramdisk retries every 30 seconds with the same CSR and consumes the approval once.

## Next [#next]

* [Provisioning workflows](/docs/guides/bare-metal/workflows): what runs on the site agent and when.
* [BMC drivers and credentials](/docs/guides/bare-metal/bmc): the BMC side of the site agent.
* [PKI](/docs/operations/pki): the node CA and JoinService keys the agents join against.


# Provisioning workflows (/docs/guides/bare-metal/workflows)

> The workflow engine expands a profile into persisted steps that run on the control plane, a site agent, the node or its ramdisk, and resume after any crash.



Commissioning or deploying a machine is a dozen operations on three different computers: the BMC
through the site agent, the ramdisk on the machine, then the installed OS. Any of them can take an
hour, and the worker that started them can crash in between. A workflow
makes that safe: the engine in `@better-compute/provision` expands a
profile into rows, runs one step at a time, and
records every transition, so whichever worker next owns the node's shard picks up exactly where the
last one stopped.

You rarely start workflows yourself. Metal starts one for each workflow-driven provision state (see
[bare metal](/docs/guides/bare-metal)), and the `health` reconciler starts remediation workflows. Your
part is writing profiles and steps, and watching or steering workflows through the API.

## Anatomy of a workflow [#anatomy-of-a-workflow]

`ctx.workflows.start(tx, { tenantId, profileId, profileVersion, nodeId, owner })` runs in the
caller's transaction. It loads the pinned profile version, expands it, and inserts a `workflows` row
(id `wf_…`) with one `workflow_steps` row per step:

* The profile document is snapshotted on the workflow, and the profile version is marked used, which
  makes it immutable from then on.
* A node (or an instance) has at most one active workflow: a second `start` is `CONFLICT`. An
  unknown profile or node is `NOT_FOUND`; a profile whose steps do not resolve is `INVALID_INPUT`
  listing every issue.
* `owner` names a work kind to queue when the workflow finishes. Metal passes `owner: 'metal'`, so
  its reconciler reacts without polling.

Steps belong to one of three phases, which run in this order:

| Phase        | Where its rows come from                                                             | Row ids                         |
| ------------ | ------------------------------------------------------------------------------------ | ------------------------------- |
| `main`       | the profile's `steps`, then its `acceptance` steps                                   | `main.1`, `main.2`, …           |
| `on-failure` | a step's `onFailure` when it names a step                                            | `main.3.on-failure`             |
| `cleanup`    | a step definition's own `cleanup` (last in, first out), then the profile's `cleanup` | `main.3.cleanup.1`, `cleanup.1` |

A workflow holds at most 512 steps across all phases. Only one step runs at a time.

## Where steps run [#where-steps-run]

Every step has a `runsOn`:

| `runsOn`        | Runs as                                                          | Example                               |
| --------------- | ---------------------------------------------------------------- | ------------------------------------- |
| `control-plane` | `run(input, context)` in the worker that owns the target's shard | `userdata.render`, `wait.agentJoined` |
| `site`          | exactly one command to the site agent of the node's site         | a driver operation with `on: 'site'`  |
| `node`          | exactly one command to the node's `bcd`                          | `node.reboot`                         |
| `ramdisk`       | exactly one command to the machine's ramdisk                     | `boot.kexec`                          |

A profile step's `use` resolves in this order:

1. A registered step with that id. If the profile also says `on:`, it must match the step's `runsOn`,
   or one of its `relays` (below).
2. Otherwise, with `on:` set to an agent, a driver operation `<concern>.<operation>` such as
   `power.setPower`, for the concerns `power`, `management`, `boot`, `deploy`, `inspect`,
   `virtual-media`, `console`, `bios`, `raid`, `firmware` and `config`. It becomes one
   `driver.invoke` command carrying the driver bound in the profile's `drivers` for that concern,
   with a default timeout of 10 minutes.
3. Otherwise `fallback`, resolved the same way.

### Relaying steps [#relaying-steps]

Most of metal's steps are control-plane steps that build a command only the control plane can build
(the BMC address and credential reference, the resolved image, a boot entry with its token) and
send it to an agent themselves with `awaitCommand`. They declare where they relay to (for example
`relays: ['site']`), so profiles may still write them as `{ use: 'power.setPower', on: 'site' }`.

`awaitCommand(context, { nodeId, kind, payload, key?, timeoutMs? })` sends a command once per step
attempt, under the key `wf:<id>:<stepId>#<attempt>[:<key>]`, and finds the same command on every
re-run. Until it settles, it throws `stepNotReady` and the engine runs the step again two seconds later
(`commandStepPollMs`). A command that failed, was rejected, expired or was cancelled throws
`DRIVER_ERROR` with its reason.

## States [#states]

<Mermaid
  chart="stateDiagram-v2
  [*] --> pending
  pending --> running: start
  pending --> cancelled: cancel
  running --> paused: pause
  paused --> running: resume
  running --> succeeded: complete
  running --> failed: fail
  running --> cancelled: cancel
  paused --> cancelled: cancel
  failed --> running: retry
  failed --> cancelled: cancel
  succeeded --> [*]
  cancelled --> [*]"
/>

A `failed` workflow is not terminal: `workflows.retry` runs it again. Steps have their own table:

| Step state                          | Meaning                                                              | Next                                               |
| ----------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------- |
| `pending`                           | not started                                                          | `running`, `skipped` (a false `when`), `cancelled` |
| `running`                           | a control-plane step is executing, or an agent step is about to send | `waiting`, `succeeded`, `failed`, `cancelled`      |
| `waiting`                           | parked on its command, or on the agent rejoining                     | `succeeded`, `failed`, `cancelled`                 |
| `failed`                            | failed; back to `pending` with `attempt + 1` while retries remain    | `pending`                                          |
| `succeeded`, `skipped`, `cancelled` | done                                                                 | none                                               |

## Step options [#step-options]

These fields of a profile step (`ProfileStep`) control how it runs:

<TypeTable
  type="{
  use: { type: 'string', description: 'A registered step id, or a driver operation <concern>.<operation>. Dotted, at most 128 characters.', required: true },
  on: { type: &#x22;'control-plane' | 'site' | 'node' | 'ramdisk'&#x22;, description: &#x22;Where it runs. Required for driver operations; for a registered step it must match its runsOn or relays.&#x22; },
  with: { type: 'Record<string, Json>', description: &#x22;The step's input, validated against the step's input manifest when the profile is expanded.&#x22; },
  timeout: { type: 'Duration', description: &#x22;How long one attempt may take: '90s', '30m', '1h30m' or milliseconds. Defaults to the step definition's timeout (10 minutes for driver operations).&#x22; },
  retries: { type: 'number', description: 'Extra attempts after a failure, 0 to 100.', default: &#x22;the step definition's, else 0&#x22; },
  await: { type: &#x22;'rejoin'&#x22;, description: 'Complete when the agent comes back with a new boot or role. Only on steps that run on an agent.' },
  onFailure: { type: &#x22;'fail' | 'rescue' | 'continue' | ProfileStep&#x22;, description: 'What a failure does once retries are spent. A step here runs, then the workflow fails; it cannot have its own onFailure step.', default: &#x22;'fail'&#x22; },
  fallback: { type: 'string', description: 'The step to use when use does not resolve.' },
  when: { type: 'string', description: 'A label expression on the target node; false skips the step. At most 256 characters.' },
}"
/>

`when` is deliberately small: `key` (the label is present and not `false`), `!key`, `key=value` and
`key!=value`, joined by `&&`, for example `nvswitch && gpu.model=H100` or `!mig.capable`. An
expression that parses to nothing is false.

## Running and waiting [#running-and-waiting]

The `workflow` reconciler (sharded by node; its key is the node id, or the workflow id for workflows
without a node) advances each workflow one transition per transaction, with the workflow row locked.
The API verbs lock the same row, so they serialize with the reconciler.

* **Control-plane steps** are marked `running` in one transaction, run outside any transaction, and
  are settled in another that first checks the step is still `running` with the same attempt. A
  crash in between re-runs the step under the same idempotency key, `wf:<id>:<stepId>`. A step that
  cannot finish yet throws `stepNotReady(ms)` and runs again after `ms` (5 seconds by default), until
  its timeout. A step that waits for something outside the workflow, such as an operator's decision, throws
  `stepWaiting(until)` instead: its timeout does not run until `until`, and counts in full from the
  moment the step goes on (`metal.ensureRamdisk` waits for `metal.confirmMac` this way).
* **Agent steps** send exactly one command, keyed `wf:<id>:<stepId>` and `#<attempt>` on retries,
  with the step's deadline as the command's `not_after`. Site steps go to the node's site agent (see
  [where site steps go](/docs/guides/bare-metal/sites)); a driver operation's command carries
  `targetNodeId` naming the machine.
* **Waiting.** A waiting step is looked at again every 10 seconds (`workflowPollMs`) and never later
  than its deadline, so progress never depends on a wake-up being delivered. When a workflow
  finishes, the engine queues its owner.

### Awaiting a rejoin [#awaiting-a-rejoin]

Some commands end the process that runs them: `boot.kexec` replaces the ramdisk with the installed
OS, and `node.reboot` and `daemon.upgrade` restart the node. Their kinds are self-terminating, and a
step that sends one sets `await: 'rejoin'`:

* When the command is sent, the step records the node's current `boot_id` and session role.
* It succeeds when the agent is connected again with a different `boot_id` or role, whatever the
  command itself reported. Its result is `{ bootId, role }`.
* Only its timeout fails it (`REJOIN_TIMEOUT`), so an `onFailure` step, such as a BMC power reset,
  fires only when the machine did not come back in time.

```ts
{ use: 'boot.kexec', on: 'ramdisk', await: 'rejoin', timeout: '15m',
  onFailure: { use: 'power.setPower', on: 'site', with: { action: 'reset' } } }
```

## Failure, retry and cancel [#failure-retry-and-cancel]

When an attempt fails and retries remain, the step goes back to `pending` with `attempt + 1`
(`workflow.step.retrying`), and its command gets a new idempotency key. Once retries are spent:

* `continue` moves on to the next step.
* `fail` (the default) and `rescue` mark the workflow failing: the rest of the main phase is
  cancelled, the cleanup phase runs, and the workflow ends `failed`. With `rescue`, the workflow error
  carries `rescue: true`.
* A step in `onFailure` runs first, then the workflow fails the same way.
* A failing cleanup or on-failure step never stops the rest of the cleanup. A step definition's own
  cleanup runs only if its step started.

The error a failed step records says why:

| Code                                                                         | Cause                                                                    |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `TIMEOUT`                                                                    | no result before the step's deadline; an in-flight command is cancelled  |
| `REJOIN_TIMEOUT`                                                             | an `await: 'rejoin'` step's agent did not come back in time              |
| `NO_SITE_AGENT`                                                              | no connected site agent served the node's site before the deadline       |
| `COMMAND_FAILED`, `COMMAND_REJECTED`, `COMMAND_EXPIRED`, `COMMAND_CANCELLED` | the step's command ended that way, with its reason                       |
| `DRIVER_ERROR`                                                               | a relaying step's command failed, or the agent's result failed a check   |
| `INVALID_CONFIG`                                                             | the step is not registered in this worker, or a port it needs is missing |
| `INVALID_INPUT`                                                              | the step's input no longer passes its manifest                           |

The workflow's own error is `<stepId> (<action>): <message>` with the step's code.

`better-compute doctor` reports two provision findings (`provisionFindings`): `workflows-stuck` (warning) for
active workflows with a step more than 5 minutes past its deadline, since the engine fails an overdue step as soon
as it looks and so no worker is running their `workflow` reconciler; and `workflows-failed` (info) for workflows
that failed in the last day. Metal adds `metal-failed` and `metal-stalled`; see
[observability](/docs/operations/observability#doctor).

## The workflows API [#the-workflows-api]

The `workflows` group reads a node's workflows and lets an operator abort, retry, pause or resume one.
All six methods are decided on the node (`compute-node`): `get` and `list` as `compute:workflows:read`,
`abort`, `retry`, `pause` and `resume` as `compute:workflows:manage`.

```ts
const { items } = await compute.api.workflows.list(credential, {
  tenantId: 'root',
  nodeId: 'node_01k…',
  limit: 20, // at most 100; newest first, with a cursor
});
const wf = await compute.api.workflows.get(credential, { tenantId: 'root', nodeId: 'node_01k…', workflowId: items[0].id });
// wf.state, wf.stepId (the step in flight), wf.error, and every step with its
// phase, action, runsOn, state, attempt, commandId, deadline, result and error
```

* [`workflows.abort`](/docs/reference/api/workflows#abort) (with an optional `reason`) cancels pending
  steps and the in-flight command, runs the cleanup phase, and ends the workflow `cancelled`. A
  pending or failed workflow is cancelled at once, with no cleanup owed. Aborting twice does nothing;
  aborting a succeeded workflow is `INVALID_STATE`.
* [`workflows.retry`](/docs/reference/api/workflows#retry) runs a failed workflow again from
  `fromStep`, by default the first main step that did not succeed. That step and every later main step
  go back to `pending` with the next attempt number, and the on-failure and cleanup steps are re-armed.
  It is `CONFLICT` when the node has another active workflow by now.
* [`workflows.pause`](/docs/reference/api/workflows#pause) and
  [`workflows.resume`](/docs/reference/api/workflows#resume) (v0.2, `compute:workflows:manage`) take
  `{ tenantId, nodeId, workflowId }` and return the workflow. Pause applies to a `running` workflow: the step
  in flight finishes and no further step starts until `resume`, which puts it back to `running` and wakes it.
  A workflow in its cleanup phase cannot pause, and pausing or resuming from any other state is
  `INVALID_STATE`. Hosts call the same engine functions, `pauseWorkflow` and `resumeWorkflow`.

> **Metal's own workflows.** 
  Metal reacts to the end of every workflow it owns. Aborting one with `workflows.abort` fails the
  node, and by the time a metal workflow has failed, the node is already `failed` and metal no longer
  watches that workflow. Steer metal nodes with `metal.abort` and `metal.recover`, which start a fresh
  workflow.

## Events [#events]

The engine publishes into the workflow's tenant, with subject `workflow/<id>` and data holding
`workflowId`, `nodeId`, `profileId` and `profileVersion`:

* `workflow.created`, `.started`, `.paused`, `.resumed`, `.retried`, `.succeeded`, `.failed`,
  `.cancelled`;
* `workflow.step.started`, `.succeeded`, `.failed`, `.retrying`, `.skipped`.

When a workflow whose profile has `hostConfig` succeeds, the engine also invalidates the node's
desired state, so the new host config reaches the daemon as NodeSpec. See
[desired state and commands](/docs/guides/concepts/desired-state-and-commands).

## What the tests guarantee [#what-the-tests-guarantee]

* A worker killed at any point of a deploy resumes from the persisted workflow, and no destructive
  command is sent twice (`tests/metal-chaos.test.ts`).
* A site agent that disappears mid-deploy holds its steps until it is back, then they finish without
  duplicates.
* A failed command is retried under a new idempotency key, and the workflow fails after cleanup when
  retries run out; `onFailure: 'continue'` and false `when` expressions behave as above
  (`tests/provision-engine.test.ts`).

To write your own steps and profiles, see [steps and profiles](/docs/extending/steps-and-profiles).


# Fractional GPUs (/docs/guides/capacity/fractional-gpus)

> Share GPUs as MIG instances, time-slice slots or MPS slots, each booked as its own atom, and what of it runs today versus in v0.5.



A whole H100 for a notebook or a small inference server wastes most of it. A pool can instead split
its GPUs, and each piece becomes an atom of its own: booked, placed, metered
and kept exclusive exactly like a whole GPU. The control-plane side of this has landed ahead of its
roadmap phase, v0.5; the node side has not (the last section of this page lists what runs today).

## Sharing modes and atoms [#sharing-modes-and-atoms]

A pool's `sharing` mode, fixed when the pool is created, decides how its GPUs are split:

| Pool `sharing` | Atom class | Atom id                 | Atoms per GPU                         |
| -------------- | ---------- | ----------------------- | ------------------------------------- |
| `exclusive`    | `gpu`      | the GPU UUID            | 1                                     |
| `mig`          | `gpu.mig`  | `<gpuUuid>/gi<g>/ci<c>` | one per MIG instance the GPU reports  |
| `timeslice`    | `gpu.slot` | `<gpuUuid>/s<k>`        | `policy.sharing.replicas`, default 4  |
| `mps`          | `gpu.slot` | `<gpuUuid>/s<k>`        | `policy.sharing.replicas`, default 10 |

Inventory derives the atoms from what the node reports (see
[inventory and health](/docs/guides/nodes/inventory-and-health)):

* A GPU in MIG mode becomes an unbookable `gpu` parent plus one `gpu.mig` atom per
  MIG instance. This happens whatever the pool's mode, but only `mig` pools accept
  `gpu.mig` requests, so a MIG-mode GPU in another pool cannot be reserved.
* In a `timeslice` or `mps` pool, a GPU not in MIG mode becomes an unbookable parent plus `replicas`
  slot atoms (1 to 48, the most clients MPS serves per GPU; a708aa6). In `mps` pools each slot also carries `mpsUnits`, its share of the GPU's 100
  MPS units (`floor(100 / replicas)`).
* A GPU whose MIG mode is changing (the pending mode differs from the current one) has every atom
  unbookable and marked `partitionPending`, so nothing new lands on it until the change completes.

Atoms carry the attributes selectors match on. MIG atoms have `gpuProduct`, `gpuModel`, `migProfile`
(such as `3g.40gb`), `giId`, `ciId`, `migUuid` and `numaNode`. Slot atoms have `gpuProduct`,
`gpuModel`, `sharing`, `slot`, `slots`, `vramGiB`, `numaNode`, and `mpsUnits` in MPS pools.

## Reserve a fraction [#reserve-a-fraction]

Ask for the atom class and narrow it with a selector:

```ts
await compute.api.reservations.create(credential, {
  tenantId: 'acme',
  poolId: migPool,
  devices: [
    {
      class: 'gpu.mig',
      count: 1,
      selector: { matchExpressions: [{ key: 'migProfile', op: 'In', values: ['3g.40gb'] }] },
    },
  ],
  duration: '4h',
  idempotencyKey: 'notebook-ana-0924',
});
```

Admission checks every device request against the pool's mode, with `INVALID_INPUT`:

* `gpu.mig` only in `mig` pools: `devices[0]: gpu.mig atoms exist only in mig pools`;
* `gpu.slot` only in `timeslice` or `mps` pools: `devices[0]: gpu.slot atoms exist only in timeslice or mps pools`;
* a request's own `sharing`, when given, must be `exclusive` or the pool's mode:
  `devices[0]: the pool shares GPUs as timeslice, not mps`.

Whole `gpu` requests stay valid in every pool and take GPUs that are not split. Atoms of one GPU count
as `NV`-connected to each other for the topology scorer. The `better-compute reserve` command asks for
whole GPUs only; reserve fractions through the API, the typed client or
`better-compute api reservations.create`.

## Quota, policy and billing [#quota-policy-and-billing]

* **Quota** is kept per atom class: a MIG instance charges `gpu.mig`, a slot charges `gpu.slot`.
  A pool owner's default quota covers only `gpu`, so set `gpu.mig` or `gpu.slot` rows with
  `ownerQuota`, [`pools.offer`](/docs/reference/api/pools#offer) or
  [`quotas.set`](/docs/reference/api/quotas#set), or every fractional request is `QUOTA_EXCEEDED`.
* **IAM policies** see fractions as `gpuSlices` on the reservation, apart from whole GPUs in
  `gpuCount`, so a limit on `gpuCount` does not count slices.
* **Billing** uses the pool's GPU meter at each atom's share of a GPU hour: a `timeslice` slot is
  `1 / slots`, an MPS slot `mpsUnits / 100`, and a MIG instance its compute slices over 7
  (`3g.40gb` is 3/7). Two of four time-slice slots and one `3g.40gb` instance for an hour bill
  0.5 + 3/7, about 0.928571 `gpu.h100.hours`.
* **Rollups** keep `gpu.mig` and `gpu.slot` rows next to `gpu`. `capacity.summary`'s per-model totals
  and `capacity.timeline` count whole GPUs only; read the class rows with
  [`capacity.byPool`](/docs/reference/api/capacity#bypool).

## Single-tenant sharing [#single-tenant-sharing]

Time slicing and MPS do not isolate one workload from another the way MIG does. `timeslice` and `mps`
pools are therefore single-tenant: the offering validator refuses to offer them to anyone but their
owner, with `INVALID_STATE`. `mig` pools can be offered like `exclusive` ones. See
[pools](/docs/guides/capacity/pools).

## Changing a layout: the atom rule [#changing-a-layout-the-atom-rule]

A flat `EXCLUDE` constraint cannot see a MIG instance and its parent overlapping, so repartitioning a
GPU under a live booking would slip past it. The atom rule closes that gap: a mode or layout change
books **every atom of the GPU, parent included**, for the change window. It commits only when nothing
of the GPU is booked in that window, and keeps anything new off the GPU until the window is released.

`better-compute/capacity` exports the primitive:

```ts
import { bookPartitionChange, releasePartitionChange } from 'better-compute/capacity';

const change = await bookPartitionChange(tx, { gpuId: 'GPU-…', during: { start, end } });
if (change.status === 'conflict') {
  // something of the GPU is booked; the whole family is free from change.freeFrom
} else {
  // apply the layout, then:
  await releasePartitionChange(tx, change.bookingId);
}
```

The window's bookings use the allocation id `partition:<gpuId>:<start>` and have no allocation row.
An unknown GPU is `NOT_FOUND`. For `mig` pools, `policy.migLayout` names the layout the `mig-layout`
config unit applies, such as `all-1g.10gb` or `mixed:3g.40gb+2x2g.20gb`; a profile's host config can take
it with `from: 'pool'` (see [profiles and images](/docs/guides/bare-metal/profiles-and-images)).

### The MIG layout gate [#the-mig-layout-gate]

Changing `policy.migLayout` on a `mig` pool must not repartition MIG instances under their holders, so the capacity
module gates the change per node, with the atom rule:

1. [`pools.update`](/docs/reference/api/pools#update) with a new `migLayout` freezes every node of the pool at its
   current layout, with the new one pending, in `capacity_mig_layouts` (migration `capacity/0004_mig_layouts`); a newer
   change replaces a pending one. Provision's `mig-layout` unit reads the node's frozen layout
   (`provision_gated_mig_layout`), falling back to the pool's for a node never gated.
2. The `capacity.mig` reconciler (sharded by node) books every GPU family of the node, parent and atoms, from the end
   of its last live booking onward, so idle time before that still books but nothing lands past it
   (`node.mig.scheduled`, with the time).
3. At that time it grants the new layout and invalidates the node's spec (`node.mig.granted`); the daemon
   repartitions the idle GPUs.
4. Once the node has converged a newer spec (a higher generation, observed), it releases the partition bookings
   (`node.mig.applied`) and the new atoms take work.

At the grant, the gate asks whether the node's host config takes its layout from the pool (`carriesPoolLayout`):
provision's last succeeded workflow with a host config has `mig-layout { from: 'pool' }`, which is what
`hostSpecFor` resolves. If not (or without provision), nothing will repartition the node for this change, and the
holds are released at the grant. If so, they are held until a newer generation is observed, however long that
takes. No timer releases a hold: with a NodeSpec backlog, work could otherwise land on the old atoms and a late
repartition would destroy them. A node that never converges keeps its GPUs held, and `better-compute doctor` reports
it as `capacity.mig.stuck` after 24 hours (see [observability](/docs/operations/observability#doctor)).

A node that joins later reads the pool's layout directly: it is not schedulable before its first converged spec.

## What runs today [#what-runs-today]

Shipped in the control plane:

* atoms derived from inventory for all three modes, with `partitionPending` during a MIG mode change;
* the sharing checks at admission, and placement, booking and exclusivity of `gpu.mig` and `gpu.slot`
  atoms (covered by `tests/capacity-fractional.test.ts`);
* the atom-rule primitive, fractional metering and the single-tenant offering rule;
* the MIG layout gate, which wraps a pool's layout change in `bookPartitionChange` (covered by
  `tests/capacity-mig-gate.test.ts`);
* the MIG torture test in `tests/capacity-torture.test.ts`: placers race on one MIG pool while simulated
  repartitions go through the gate, and no parent and child, or atoms of different layouts, are ever live at once.

On the node, `bcd` (built and tested end to end, not released) holds device assignments through its devices
converger: CDI specs, env files and node-signed `AllocationBound` and `AllocationReleased` reports. Since 46a61db it
holds `gpu.mig` atoms (resolved to the MIG instance's `MIG-…` UUID from the inventory; `device-missing` until the
instance exists) and `gpu.slot` atoms (their GPU), refuses a whole GPU while part of it is held and a part while the
whole is held, and gives a slot hold `CUDA_MPS_PIPE_DIRECTORY` when MPS runs (see
[device assignments](/docs/daemon/delivery#device-assignments)).

bcd's host converger (6c2343f) applies the layout: the [`mig-layout` unit](/docs/daemon/host-config#mig-layout)
partitions every idle MIG-capable GPU (`all-disabled`, `all-<profile>`, `mixed:<n>x<profile>+…`), leaves a GPU that
a live hold uses alone, and asks for a reboot when a mode change needs a GPU reset that fails. For `mps` and
`timeslice` pools, provision renders the [`gpu-sharing` unit](/docs/daemon/host-config#gpu-sharing) (1f0f683), with
which bcd runs the MPS control daemon as `bcd-mps.service`. All of it is tested against fakes; no GPU has been
repartitioned on a real node.

Still to come in v0.5:

* a MIG repartition and an MPS pool on real hardware;
* the remaining v0.5 exit tests and the real-hardware nightly.

PLAN.md plans to reset GPUs on release and leaves one question open: clearing memory between tenants
relies on the driver's guarantees, and a zeroing pass may be added. See the
[roadmap](/docs/reference/roadmap).


# Gang and calendar reservations (/docs/guides/capacity/gang-and-calendar)

> Book several nodes at once on one fabric, book a window that starts later, and let BookingRepair move a future booking off hardware that went bad.



Two workloads need more than "some GPUs, now". Distributed training needs several machines at once,
all on the same high-speed fabric, or none at all. Planned work, such as a training run next Monday or
a nightly batch, needs its capacity promised ahead of time. A gang covers the
first and a calendar reservation the second; they combine
freely. Both landed ahead of their roadmap phase, v0.5.

## Gang reservations [#gang-reservations]

Add `gang` to a reservation request. `devices` then describes **one node**, and the gang books `size`
nodes that each take all of it:

```ts title="pretrain.ts"
const run = await compute.api.reservations.create(credential, {
  tenantId: 'acme',
  poolId,
  devices: [{ class: 'gpu', count: 8, topology: 'nvlink-all' }], // per node
  gang: { size: 4, topologyKey: 'fabric.domain' }, // 4 nodes, one NVLink fabric
  duration: '3d',
  idempotencyKey: 'pretrain-run-7',
});
```

<TypeTable
  type="{
  size: { type: 'number', description: 'Nodes to book, 2 to 64.', required: true },
  topologyKey: {
    type: 'string',
    description: 'A node label, such as fabric.domain or rack. Every member shares one value of it; nodes without the label never join. Without a key, any nodes of the pool form a gang.',
  },
}"
/>

`fabric.domain` is an inventory label: the NVLink fabric cluster and clique of the node's GPUs, set
only when all of them agree. Operator labels such as `rack` work the same way.

### How a gang is placed [#how-a-gang-is-placed]

1. Every node of the pool is planned for the per-node request with the normal
   [filters, atom pick and scorers](/docs/guides/capacity/placement).
2. The nodes that fit are grouped by their `topologyKey` value. Each group with at least `size` fitting
   nodes contributes a candidate gang made of its `size` best nodes.
3. Candidate gangs are ranked by the sum of their members' scores, then by group name, and one is
   picked at random among the best `topN`.
4. All members are committed inside one outer savepoint, one allocation per node. If any member loses
   a race, the **whole gang** rolls back, that node is excluded, and the gang is planned again, up to
   `maxAttempts` times.

Gangs never use the SQL shortlist and always place inline, even with the pool-lease placer. When no
group is large enough the reservation is `pending`, with a reason such as:

```text
no gang of 4 nodes fits in one fabric.domain (at most 3 nodes fit together)
```

### What a gang counts as [#what-a-gang-counts-as]

* **Quota** counts the whole gang: `size` times the per-node devices, charged all or nothing.
* **IAM** sees the whole gang too. The draft's `gpuCount` is the per-node whole GPUs times `size`, and
  `gangSize` carries `size`, so a policy capped at 8 GPUs refuses a gang of two 8-GPU nodes.
* **The lifecycle** moves all members together. The reservation becomes `active` only when every
  member node has reported its devices bound; extend moves every member's end or none; release and
  cancel free every member.
* **`reservations.get`** shows one member's `binding` (the first bound allocation by id). Each member
  node carries its own assignment in its desired state.

[`capacity.preview`](/docs/reference/api/capacity#preview) accepts `gang` and answers with the members of
the best gang, or the reason none fits. A regular gang that does not fit may preempt spot work on
several nodes in one notice ([spot and preemption](/docs/guides/capacity/spot-and-preemption)), and a
`binding: 'node'` lease can be a gang of whole machines.

## Calendar reservations [#calendar-reservations]

A reservation with a `start` in the future is a calendar reservation. It is placed at admission for
the window from `start` to `start + duration`: the atoms are booked now, so nobody else can take them
for that window.

```bash
better-compute reserve --tenant acme --pool pool_01k… --gpus 8 --for 12h --start 2026-09-28T08:00:00Z --key weekly-eval
better-compute reserve --tenant acme --pool pool_01k… --gpus 2 --for 1h --start +2h
```

* Bookings are intervals, so the same atoms can serve other reservations that end before `start`.
  The pool's rollup counts the booking as `reserved` until it begins.
* The reservation stays `placed` until its start, then binds like any other. Billing starts only
  when it is `active`.
* Its quota charge is held from admission, including the time before the start.
* [`reservations.cancel`](/docs/reference/api/reservations#cancel) withdraws it before the start and
  refunds the quota. `reservations.extend` works on a placed reservation too.
* If nothing fits the window at admission, the reservation is `pending` and retried every 30 seconds
  for the same start. Once the start has passed, a successful retry books from the moment it is
  placed, for the full duration.

## BookingRepair [#bookingrepair]

Between placement and start, hardware can go bad: a node is cordoned for maintenance, a GPU is
quarantined after an XID error, a node is unreachable. A calendar reservation that simply waited for
its start would then fail to bind. **BookingRepair** checks the booking before that happens.

At 15 minutes before the start, and again at the start, the lifecycle checks every allocation of a
`placed` reservation: its node must still be schedulable and every booked atom healthy and bookable.
A node lease is judged by whether its node is still leasable, and only before its window begins,
because from then on the node is metal's to deploy. If an allocation fails the check:

1. The pool is locked first, keeping capacity's lock order.
2. The reservation's allocations are released, and it is placed again for **the same window**,
   avoiding the bad nodes. The quota charge is kept.
3. On success the reservation stays `placed` on its new nodes, and `reservation.repaired` reports
   the nodes it left (`from`) and the new ones (`nodeIds`).
4. If nothing fits, the reservation goes `pending` with `Unschedulable` and reason
   `BookingRepairFailed`, and retries like any pending reservation.

Repairs always place inline. A reservation that starts immediately is checked at its start, just
before it binds.

## The timeline API [#the-timeline-api]

[`capacity.timeline`](/docs/reference/api/capacity#timeline) draws booked GPUs over time for a calendar
view, decided as `compute:capacity:read` on the pool.

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant asking; the pool must be visible to it.', required: true },
  poolId: { type: 'string', description: 'The pool.', required: true },
  from: { type: 'number', description: 'Epoch milliseconds.', default: 'the start of the current hour' },
  to: { type: 'number', description: 'Epoch milliseconds, after from.', default: 'from plus 24 hours' },
  stepMinutes: { type: 'number', description: 'Bucket width, 5 to 1440 minutes.', default: '60' },
}"
/>

A timeline spans at most 7 days and 336 buckets; anything longer is `INVALID_INPUT`.

```json title="capacity.timeline, from now to 4 hours ahead"
{
  "poolId": "pool_01k…",
  "allocatable": 8,
  "buckets": [
    { "start": 1790000000000, "end": 1790003600000, "booked": 3, "free": 5 },
    { "start": 1790003600000, "end": 1790007200000, "booked": 3, "free": 5 },
    { "start": 1790007200000, "end": 1790010800000, "booked": 2, "free": 6 },
    { "start": 1790010800000, "end": 1790014400000, "booked": 0, "free": 8 }
  ]
}
```

`booked` counts the distinct whole GPUs (`gpu` atoms) of the pool with any booking that overlaps the
bucket, so a GPU busy for five minutes of an hour counts for the hour. `free` is
`max(0, allocatable − booked)`, where `allocatable` is the pool's current rollup value, not a forecast
of future health. MIG instances and slots are not counted.

  - [Reservations](/docs/guides/capacity/reservations): The lifecycle, bindings, extend and release.

  - [Placement](/docs/guides/capacity/placement): The per-node planning a gang is built from.


# Capacity (/docs/guides/capacity)

> How Better Compute counts GPUs, books them as atoms over time, keeps every booking exclusive, and reports free capacity per pool.



Capacity is the part of Better Compute that answers two questions: how many GPUs a
pool can hand out over a time window, and which exact devices a
reservation gets. It lives in `@better-compute/capacity` (also
`better-compute/capacity`), which the server registers as an always-on built-in module right after
inventory; you do not register it yourself.

Everything here rests on one idea: a claim on hardware is a row with a time range, and the database
refuses two overlapping rows for the same device. Placement, quotas and rollups are layered on top
of that rule, so no scheduler bug or concurrent request can hand one GPU to two tenants.

## Atoms and bookings [#atoms-and-bookings]

Inventory turns what a node reports into devices. Each has a class
(`gpu`, `gpu.mig`, `gpu.slot`, `nic.vf`, `disk` or `node`), typed attributes such as `gpuProduct`,
`gpuModel`, `vramGiB` and `numaNode`, and two flags: `healthy` and `bookable`. A bookable device is an
atom, and only atoms are ever booked.

* A whole GPU is one `gpu` atom, identified by its GPU UUID.
* A GPU in MIG mode is an unbookable `gpu` parent plus one `gpu.mig` atom per instance
  (`<gpuUuid>/gi<g>/ci<c>`).
* A GPU in a `timeslice` or `mps` pool is an unbookable parent plus `gpu.slot` atoms
  (`<gpuUuid>/s<k>`). See [fractional GPUs](/docs/guides/capacity/fractional-gpus).

When a reservation is placed, capacity writes one allocation per node
and one booking per atom:

<Mermaid
  chart="flowchart LR
  R[&#x22;Reservation rsv_…&#x22;] --> A1[&#x22;Allocation alloc_… on node A&#x22;]
  R --> A2[&#x22;Allocation alloc_… on node B (gangs)&#x22;]
  A1 --> B1[&#x22;Booking: GPU-1, during&#x22;]
  A1 --> B2[&#x22;Booking: GPU-2, during&#x22;]
  A2 --> B3[&#x22;Booking: GPU-9, during&#x22;]"
/>

A booking is `(device_id, allocation_id, during)`, where `during` is a `tstzrange`. The allocation
also records scalar claims (`vcpu`, `memoryGiB`, `diskGiB`) that VPS instances take from
`node_resources`. Since v1.1's first pieces, GPU reservations also book host CPUs, memory and 1 GiB huge pages
with their devices ([host resources](/docs/guides/capacity/reservations#host-resources)), which the node turns into
a cgroup slice for the workload, and a reservation can hand its GPUs to a node's kubelet
([Kubernetes workers](/docs/guides/capacity/kubernetes)). What is still missing is on
[the roadmap](/docs/reference/roadmap#cpu-memory-and-kubernetes-v11).

## Capacity math [#capacity-math]

Every number capacity reports is derived from bookings and node state, following PLAN.md section 10.2:

```text
allocatable = healthy, bookable atoms on schedulable nodes   (scalars: floor(allocatable × ratio))
allocated   = bookings whose during contains compute_now()   (scalars: node_resources.allocated)
reserved    = bookings that start after compute_now()
available   = max(0, allocatable − allocated − reserved)
```

Time is always the database's `compute_now()`, never the API process's clock, so every process agrees
on what "now" is (see [consistency](/docs/guides/concepts/consistency)).

A node's atoms count as allocatable only when the node is schedulable: role `host`, not in
maintenance, not cordoned, provision state
`adopted` or `deployed`, `connected`, no `NoSchedule` reason, at least one converged spec, and trust
`platform` (or `tenant-owned` in its owner's pool). Everything else lands in a bucket that explains why
it is not allocatable:

| Bucket         | What lands there                                                                                                                 |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `provisioning` | Provision state other than `adopted` or `deployed`, or no converged spec yet                                                     |
| `maintenance`  | Nodes in maintenance, and cordoned nodes                                                                                         |
| `unreachable`  | Connectivity other than `connected`                                                                                              |
| `unhealthy`    | Nodes with a `NoSchedule` reason, and unhealthy atoms on otherwise schedulable nodes                                             |
| excluded       | Not counted at all: nodes whose role is not `host`, `tenant-controlled` nodes, and `tenant-owned` nodes in another tenant's pool |

GPUs are never overcommitted. Only scalar resources carry an overcommit `ratio`.

`available` is deliberately conservative: an atom booked now and again later counts in both
`allocated` and `reserved`. For an exact answer about a specific window, ask the placer with
[`capacity.preview`](/docs/reference/api/capacity#preview) instead of subtracting rollup numbers.

## Why double-booking cannot happen [#why-double-booking-cannot-happen]

Six independent guards hold. The torture tests in `tests/capacity-concurrency.test.ts` attack the
first five with 1,000 interleaved and 1,000 concurrent operations: reservations, releases, cordons,
taints and `healthy` flips.

1. **Device-time exclusivity.** `device_bookings` carries
   `EXCLUDE USING gist (device_id WITH =, during WITH &&)`. At most one live booking exists per atom
   per instant, whatever the placer or API does. This is the
   EXCLUDE constraint.
2. **The atom rule.** A partitioned GPU's parent is not bookable, and a MIG or slot layout change books
   every atom of the GPU, parent included, for the change window. A flat `EXCLUDE` cannot see
   parent and child overlapping; the atom rule removes the case.
3. **Guarded counters.** Scalars are claimed with
   `UPDATE node_resources SET allocated = allocated + $n WHERE … AND allocated + $n <= floor(allocatable * ratio)`,
   with a `CHECK` constraint as a backstop.
4. **Linearizable eligibility.** The placer locks the chosen node `FOR SHARE OF n` under the
   schedulability predicate, and the chosen atoms `FOR SHARE` where they are healthy and bookable.
   Cordon, taint, liveness and quarantine writes update those same rows, so whichever transaction
   commits second sees the first.
5. **Quota.** `quota_usage` is charged with a guarded `used + n <= max` update, and a
   `CHECK (used BETWEEN 0 AND max)` backs it up.
6. **Defence in depth on the node.** The node refuses an assignment that overlaps one it already
   holds. The TypeScript FakeDaemon and the devices converger of `bcd` both answer such an assignment
   with `AllocationFailed`.

Every guarded statement returns fewer rows instead of raising, and `INSERT … ON CONFLICT DO NOTHING`
(without a conflict target) honours the exclusion constraint. A lost race therefore rolls back to a
savepoint and the placer replans, without aborting the request's transaction. The details are on
[placement](/docs/guides/capacity/placement).

## Rollups [#rollups]

Reading capacity never aggregates devices. `pool_capacity_rollup` holds one row per pool and resource
(`gpu`, `gpu.mig`, `gpu.slot`, and scalars such as `vcpu`), and two mechanisms keep it current:

* **Deltas in the booking transaction.** Every placement adds its atoms to `allocated` (the booking
  has started) or `reserved` (it starts later), and every release subtracts them, in the same
  transaction that writes the bookings.
* **A periodic rebuild.** The `capacity.rollup` reconciler recomputes every pool from nodes, devices,
  bookings and `node_resources` every 5 minutes (`rollupEvery` in the capacity options). The rebuild
  is what moves a booking from `reserved` to `allocated` once it starts, and what picks up bucket
  changes such as a cordon or a node going unreachable.

Between rebuilds the rollup can lag reality by up to one period. Treat it as a dashboard number; the
placer never reads it.

## The capacity API [#the-capacity-api]

Four read methods serve capacity, all decided as `compute:capacity:read`:

| Method                                                       | Decided on | Returns                                                                                   |
| ------------------------------------------------------------ | ---------- | ----------------------------------------------------------------------------------------- |
| [`capacity.summary`](/docs/reference/api/capacity#summary)   | the tenant | Every pool the tenant owns or is offered, with its rollup rows, plus totals per GPU model |
| [`capacity.byPool`](/docs/reference/api/capacity#bypool)     | the pool   | One pool's rollup rows (`pools.capacity` answers the same)                                |
| [`capacity.preview`](/docs/reference/api/capacity#preview)   | the pool   | Whether a reservation request would fit now, and where, without booking anything          |
| [`capacity.timeline`](/docs/reference/api/capacity#timeline) | the pool   | GPUs booked per time bucket, for calendar views                                           |

```ts title="capacity.ts"
const summary = await compute.api.capacity.summary(credential, { tenantId: 'acme' });
// summary.byGpuModel.H100 → { total, allocatable, allocated, reserved, available }

const rows = await compute.api.capacity.byPool(credential, { tenantId: 'acme', poolId });
```

`summary.byGpuModel` adds up only the whole-GPU (`gpu`) rows of each pool; MIG and slot rows are in
`summary.pools[].capacity`. A pool without a GPU model is summed under `other`. Each rollup row looks
like this:

```json title="capacity.byPool (one row)"
{
  "poolId": "pool_01k…",
  "resource": "gpu",
  "total": 64,
  "allocatable": 56,
  "allocated": 24,
  "reserved": 8,
  "available": 24,
  "provisioning": 0,
  "unhealthy": 1,
  "maintenance": 7,
  "unreachable": 0,
  "updatedAt": 1790000000000
}
```

`total` is the sum of `allocatable` and the four buckets. From the CLI, `better-compute capacity`
prints the summary, `--pool` one pool's rows, and `--preview` with `--gpus` and `--for` a placement
preview (see the [CLI reference](/docs/reference/cli)):

```bash
better-compute capacity --tenant acme
better-compute capacity --tenant acme --preview --pool pool_01k… --gpus 8 --for 1d --selector "vramGiB >= 80"
```

Preview and timeline are described with the features they serve: [placement](/docs/guides/capacity/placement)
and [gang and calendar reservations](/docs/guides/capacity/gang-and-calendar).

## In this section [#in-this-section]

  - [Pools](/docs/guides/capacity/pools): Classes, sharing modes, policy, offerings with quota, and archiving.

  - [Placement](/docs/guides/capacity/placement): The placer contract, filters, scorers, the shortlist and the commit.

  - [Reservations](/docs/guides/capacity/reservations): Drafts, the admission sequence, states, bindings, leases and the API.

  - [Quotas and usage](/docs/guides/capacity/quotas-and-usage): Per-pool limits, hourly usage windows and Better IAM billing.

  - [Gang and calendar reservations](/docs/guides/capacity/gang-and-calendar): Multi-node bookings on one fabric, future start times and BookingRepair.

  - [Spot and preemption](/docs/guides/capacity/spot-and-preemption): Preemptible reservations, notices and what happens to their allocations.

  - [Fractional GPUs](/docs/guides/capacity/fractional-gpus): MIG instances, time-slice and MPS slots as atoms.

  - [Domain model](/docs/guides/concepts/domain-model): Where pools, nodes, devices and reservations sit among the other records.


# Kubernetes workers (/docs/guides/capacity/kubernetes)

> Join a pool's machines to a Kubernetes cluster and hand a reservation's GPUs to their kubelets with binding kubernetes, through bcd's device plugin or DRA driver.



PLAN.md §3.8 lets compute hand a cluster machines and GPUs and take them back, without scheduling a pod or running
a control plane. A pool names a Kubernetes cluster; its machines join that cluster as workers; a reservation with
`binding: 'kubernetes'` puts its GPUs on the node's kubelet, and bcd offers only those. The cluster's scheduler then
places pods on them, and when the reservation ends the GPUs are withdrawn and pods still holding them are deleted.
Pods get only the reserved GPUs when bcd is the only source of GPUs on the node; see [GPU isolation](#gpu-isolation).

> **Built end to end in code, run only against fakes.** 
  Every piece below is committed: the pool policy and the rendered `k8s-join` unit, the reservation binding, the
  node's join, the devices converger's Kubernetes holds, the device plugins and the DRA driver. The capacity tests cover
  the reservation side on PGlite, and `tests/e2e/kubernetes-bindings.test.ts` runs the binding through the dev stack
  and its FakeDaemons (8a084ca). The join unit is tested against a fake host, and the device plugins and DRA driver
  against a fake kubelet over real unix sockets. No test has joined a real cluster or started a pod on a GPU this way.

## Two ways into a cluster [#two-ways-into-a-cluster]

|                               | A Kubernetes pool (this page)                                                                             | bcd on Kubernetes                                                                     |
| ----------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Who owns the machines         | The control plane: bcd is installed on the host and applies its host configuration                        | The cluster: bcd runs inside it as a [DaemonSet](/docs/daemon/kubernetes)             |
| How they get into the cluster | The `k8s-join` config unit runs `kubeadm join`                                                            | They are cluster nodes already                                                        |
| GPUs for pods                 | bcd's device plugins or DRA driver offer a `binding: 'kubernetes'` reservation's GPUs, chosen by `gpuApi` | The same, in hosted mode (1e5f96b), chosen by `[kubernetes] gpu_api` in `config.toml` |

<div className="fd-steps">
  <div className="fd-step">
    ## The pool [#the-pool]

    Give the pool a `policy.kubernetes`. Provision reads it when it builds each node's spec.

    ```ts title="create-pool.ts"
    await compute.api.pools.create(rootCredential, {
      tenantId: 'root',
      siteId: 'site_01k…',
      class: 'metal-host',
      gpuModel: 'H100',
      policy: {
        kubernetes: {
          apiServer: 'https://k8s.example.com:6443',
          caCertSha256: 'sha256:<64 hex digits>',
          joinTokenRef: 'secret://root/sec_01k…', // from secrets.put
          labels: { 'example.com/pool': 'h100' },
          taints: ['example.com/gpu=reserved:NoSchedule'],
          kubernetesVersion: '1.34.1',
          gpuApi: 'device-plugin',
          platformOperated: true, // required for a join on a platform site (e4d14cd)
        },
      },
      idempotencyKey: 'pool-k8s-h100',
    });
    ```

    <TypeTable
      type="{
  apiServer: { type: 'string', description: 'The API server, https://host[:port] only. With caCertSha256 and joinTokenRef: all three or none.' },
  caCertSha256: { type: 'string', description: &#x22;The SHA-256 of the cluster CA's public key: kubeadm's discovery-token-ca-cert-hash, with or without sha256:.&#x22; },
  joinTokenRef: { type: 'string', description: 'A secret:// reference to a kubeadm bootstrap token (abcdef.0123456789abcdef). The token never enters a spec.' },
  labels: { type: 'Record<string, string>', description: &#x22;Labels for the machines' Node objects; the kubernetes.io and k8s.io prefixes are refused.&#x22; },
  taints: { type: 'string[]', description: 'key[=value]:NoSchedule | PreferNoSchedule | NoExecute.' },
  kubernetesVersion: { type: 'string', description: 'A release such as 1.34.1: the pkgs.k8s.io repository bcd installs kubeadm, the kubelet and kubectl from when they are missing.' },
  gpu: { type: 'boolean', description: 'Hand reserved GPUs to the kubelet.', default: 'true' },
  gpuApi: { type: &#x22;'device-plugin' | 'dra'&#x22;, description: &#x22;How: bcd's device plugins, or its DRA driver gpu.bcd.dev. Passed to the k8s-join unit; see Device plugin or DRA below.&#x22;, default: &#x22;'device-plugin'&#x22; },
  platformOperated: { type: 'boolean', description: 'The platform operates this cluster. Required with apiServer on a pool whose site is not custom (e4d14cd). For a pool whose bcd runs as a DaemonSet inside the cluster: nodes joining with a hosted token keep platform trust, and only when the platform owns the pool. See /docs/daemon/kubernetes#trust.' },
}"
    />

    `apiServer`, `caCertSha256` and `joinTokenRef` go together (7db110a): all three say the machines join that cluster
    through the `k8s-join` unit; none says bcd already runs inside the cluster as a [DaemonSet](/docs/daemon/kubernetes);
    a partial set is `INVALID_INPUT`. With the three, provision turns the policy into the `k8s-join` config unit on every
    machine of the pool, adopted ones included (see [host configuration](/docs/daemon/host-config#k8s-join)).

    **A platform site's pool joins only a cluster the platform operates** (e4d14cd). A cluster's admins get root on its
    workers through privileged pods, while machines on a platform site keep their platform identity and receive brokered
    secrets. So `pools.create` and `pools.update` refuse the three join settings on a pool whose site is not `custom`
    unless `platformOperated` is `true` (`INVALID_INPUT`, "a platform site's pool joins only a cluster the platform
    operates; confirm it with policy.kubernetes.platformOperated: true"). A tenant's `custom` site needs no confirmation,
    since its nodes are tenant-owned anyway, and a pool without the join settings (bcd as a DaemonSet) is not affected.
    bcd's half of the rule is the `[kubernetes] allowed_clusters` list below.

    Since 37dcd02 the pools API checks the policy when `pools.create` or `pools.update` writes it: provision registers its
    host-unit checks as a `pool-policy` extension, so a bad `policy.kubernetes` is `INVALID_INPUT` at the API (`policy:
    ...` for a failure that is not already a compute error) instead of breaking every node's spec later.

    **The bootstrap token.** Store it with [`secrets.put`](/docs/reference/api/secrets) in the tenant that owns the
    machines or the pool, and name it by its `secret://<tenant>/<id>` reference. The secret broker releases it to a
    machine only while that machine's current spec names exactly this reference in `k8s-join.tokenRef`, and only when
    the reference belongs to the node's owner or the pool's owner (binding rule 3; see
    [sealed secrets](/docs/operations/security#sealed-secrets)). Since e4b0c01 provision refuses a `vault://` or `env://`
    reference here, since the broker reads the tenant from a `secret://` reference and would never release it.
  </div>

  <div className="fd-step">
    ## The machines [#the-machines]

    Each machine runs `bcd node` the usual way. The first pass of its host converger installs kubeadm, the kubelet and
    containerd where missing, prepares the kernel and containerd (CDI on, the systemd cgroup driver), fetches the token,
    and runs `kubeadm join`, registering the Node with the pool's labels and taints. From then on it keeps the kubelet
    running and the Node's labels in step, adding `bcd.dev/node-id`, `bcd.dev/site`, `bcd.dev/gpu.count`,
    `bcd.dev/gpu.product` and `bcd.dev/gpu.driver`. It never moves a node that is joined to another cluster.

    Taints are set only at registration (6ef198b): Kubernetes' NodeRestriction forbids a kubelet to change its own
    taints. When you change the pool's `taints` later, joined nodes report the condition `KubernetesTaintsDrift`, whose
    message is the `kubectl taint node … --overwrite …` command a cluster admin runs to apply them (see
    [host configuration](/docs/daemon/host-config#k8s-join)).

    Joining a cluster gives it root on the machine, so root decides which clusters a machine may join, in `config.toml`:

    ```toml title="/etc/bcd/config.toml"
    [kubernetes]
    allowed_clusters = "https://k8s.example.com:6443#sha256:<64 hex digits>"
    ```

    Since 9f860ad the list is required whether `bcd node` runs as root or under
    [`bcd helper`](/docs/daemon#privilege-separation): with none, every join is refused, and `bcd doctor` fails a joined
    node without one. The details, files and failure cases are on
    [host configuration](/docs/daemon/host-config#k8s-join).
  </div>

  <div className="fd-step">
    ## The reservation [#the-reservation]

    Reserve with `binding: 'kubernetes'`:

    ```ts title="reserve.ts"
    await compute.api.reservations.create(credential, {
      tenantId: 'acme',
      poolId: 'pool_01k…',
      devices: [{ class: 'gpu', count: 2 }],
      duration: '8h',
      binding: 'kubernetes',
      resources: { vcpu: 32, memoryGiB: 256 },
      idempotencyKey: 'train-k8s-2026-09-26',
    });
    ```

    From the CLI (ea36a94), preview first, then reserve:

    ```bash
    better-compute capacity --tenant acme --preview --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes
    better-compute reserve --tenant acme --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes
    ```

    `@better-compute/console-kit`'s reservation form takes the same host CPU, memory and Kubernetes fields (e833433), and
    the console app's reservation page shows them (4533265): its **Give the GPUs to the pool's Kubernetes cluster**
    checkbox appears only for a pool the tenant owns that has `policy.kubernetes`.

    * The pool must have `policy.kubernetes`, or the request is `INVALID_INPUT`
      (`Pool … does not join a Kubernetes cluster (policy.kubernetes)`).
    * **Only the pool's owner books it** (3825ec7). The device plugins and the DRA driver hand bound GPUs to any pod of
      the cluster that asks and cannot tell tenants apart, so a tenant the pool is merely offered to could pay for GPUs
      that another tenant's pods then take. A pool's cluster is therefore its owner's: `binding: 'kubernetes'` from any
      other tenant is `INVALID_INPUT` (`Pool …'s Kubernetes cluster is its owner's: only the owner books binding
      kubernetes`), checked when the draft is written, before the IAM decision. Tenants the pool is offered to book
      holds on it instead (or instances, where the pool runs them).
    * Everything else is a normal reservation: the same decision, quota, placement and lifecycle. `resources` books host
      CPU, memory and 1 GiB pages with the GPUs, or the pool's per-GPU share when left out (see
      [host resources](/docs/guides/capacity/reservations#host-resources)). `capacity.preview` books the same host
      resources, so a preview that fits is a create that fits (17f600b).
    * It places only on nodes whose bcd advertises `binding.kubernetes` (and `devices.limits` when it books host
      resources), since an older bcd would drop the binding; see [older daemons](#older-daemons).
    * An agent can ask for it too: the [MCP](/docs/guides/clients/mcp) tools `reserve_gpus` and `preview_reservation`
      take `kubernetes: true` (binding `kubernetes`), with optional `vcpu` and `memoryGiB`.
    * At the start, the node's spec gains a device assignment with `BINDING_KUBERNETES`. bcd binds it like a hold: it
      checks the devices, writes `/etc/cdi/bcd-<allocation>.json` and reports a node-signed `AllocationBound`, which makes
      the reservation `active`. The CDI spec carries the annotation `bcd.dev/binding: kubernetes` and, unlike a hold's,
      no `NVIDIA_VISIBLE_DEVICES`, since a pod may get only part of the reservation.
  </div>
</div>

## The device plugin [#the-device-plugin]

`bcd node` runs kubelet device plugins (`deviceplugin/v1beta1`) while the kubelet's socket exists on a machine bcd
joined with `gpu` on, or, in [hosted mode](/docs/daemon/kubernetes#gpus-for-pods), on any node of the DaemonSet.
Every 5 seconds, and at once after a bind or a release, it reads the Kubernetes holds from `/etc/cdi` and advertises
exactly their devices, one resource per device class, each on its own socket in `/var/lib/kubelet/device-plugins`:

| Resource           | Devices                  | Socket              |
| ------------------ | ------------------------ | ------------------- |
| `bcd.dev/gpu`      | Whole GPUs               | `bcd-gpu.sock`      |
| `bcd.dev/gpu.mig`  | MIG instances            | `bcd-gpu.mig.sock`  |
| `bcd.dev/gpu.slot` | Time-slice and MPS slots | `bcd-gpu.slot.sock` |

* **Devices.** Since 9f860ad each device's kubelet id is `<atom>@<allocation>`, the atom's CDI device name (the atom
  id with `/` as `_`) tied to the allocation that holds it, so the same GPU re-bound to another allocation is another
  device, and a pod that got it under the old allocation holds an id nobody advertises and is reaped. Its NUMA node comes from the
  inventory, and it is advertised unhealthy while its GPU has a pending or failed row remap, uncorrectable ECC
  errors since the driver loaded, or an XID condition. Since 6f5a99f an XID, or a reset that clears it, triggers a
  pass at once, so a GPU that goes bad is never offered as healthy. Pods are not deleted for health, since the device
  is still bound: only a release reaps. On the control plane's side an XID that quarantines the GPU makes the
  reservation `Degraded` (`AtomUnhealthy`, naming the GPU) within a pass and releases nothing (04f6bda);
  `tests/e2e/kubernetes-bindings.test.ts` checks it with an XID 48.
* **Allocate.** For each container the plugin answers the CDI names `bcd.dev/gpu=<device>` and, where NVIDIA's CDI
  spec (in `/etc/cdi` or `/var/run/cdi`) names the GPU or MIG instance, `nvidia.com/gpu=<uuid>` too, so containerd
  injects the device nodes and driver libraries. It sets `NVIDIA_VISIBLE_DEVICES` to the GPU or MIG UUIDs and
  `BCD_ALLOCATION_IDS` to the allocations they belong to (also as the annotation `bcd.dev/allocation-ids`). A device
  that is no longer bound is refused, so the kubelet cannot start a container on a released GPU. With the NVIDIA GPU
  operator, let it generate its CDI specs with `--device-name-strategy=uuid`, so the `nvidia.com/gpu=<uuid>` names
  exist.
* **Preferred allocation.** Asked for a set of devices, the plugin prefers the ones it must include, then devices of
  the same allocation on the same NUMA node, then the allocation with the most free devices.
* **Restarts.** When the kubelet restarts (its socket changes) or the plugin's socket disappears, the plugin
  registers again. When the node is no longer a worker with `gpu` on, the plugins stop.

`bcd status` prints how many devices each resource offers the kubelet.

A pod asks for the resource the way it asks for any extended resource:

```yaml title="train.yaml"
apiVersion: v1
kind: Pod
metadata:
  name: train
spec:
  tolerations:
    - key: example.com/gpu
      operator: Equal
      value: reserved
      effect: NoSchedule
  containers:
    - name: main
      image: registry.example.com/train:latest
      resources:
        limits:
          bcd.dev/gpu: 2
```

Compute decides which GPUs a node offers, not which pod takes them: any pod the cluster's scheduler places there may
claim advertised devices. That is why only the pool's owner books `binding: 'kubernetes'` (3825ec7): the cluster and
every GPU reserved into it belong to one tenant. Use the Node labels, taints and your cluster's own policies to keep a
reservation's GPUs for the workloads it was made for; `BCD_ALLOCATION_IDS` tells a container which allocation it got.

## GPU isolation [#gpu-isolation]

bcd's plugins and DRA driver offer exactly the bound GPUs, but pods are limited to them only when nothing else on the
node hands out GPUs (d6f8549). Another device plugin, or a container runtime that gives a pod any GPU named in
`NVIDIA_VISIBLE_DEVICES`, lets any pod take every GPU. With the NVIDIA GPU operator, set `devicePlugin.enabled=false`,
set the container toolkit's `accept-nvidia-visible-devices-envvar-when-unprivileged = false` (CDI only), and generate
CDI specs with `--device-name-strategy=uuid`. This applies to joined workers and to
[bcd on Kubernetes](/docs/daemon/kubernetes#gpu-isolation) alike; the generated DaemonSet carries the requirements
in its `bcd.dev/requires` annotation.

Since 9f860ad bcd enforces this, and since f88bcdb (security review 2) it follows every path it knows of. bcd offers
no GPUs at all when the kubelet can also allocate an `nvidia.com/*` resource (the GPU operator's device plugin, read
from pod-resources' allocatable resources); when the node's ResourceSlices (read with the kubelet's credentials at
most once a minute) show another GPU DRA driver offering devices, such as `gpu.nvidia.com`; or when a container
toolkit configuration accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods (the key `true`, or absent, which is the
toolkit's default). The toolkit configuration that counts is the one the container runtime runs: bcd reads
containerd's `BinaryName` (plain, k3s, RKE2 and `conf.d` drop-ins) and CRI-O's `runtime_path` for
`nvidia-container-runtime` (not the `.cdi` variant) and checks that binary's configuration, the GPU operator's
`<installDir>/toolkit/.config/nvidia-container-runtime/config.toml` or `/etc/nvidia-container-runtime/config.toml`.
A runtime that runs the toolkit with that file missing or unreadable is exposure, since the toolkit then takes its
default; the two known files also count whenever they are present. As the cluster's DaemonSet, a node where no
container runtime configuration is readable is exposure as well (see
[bcd on Kubernetes](/docs/daemon/kubernetes#gpu-isolation)). In every case the device plugins stop, the DRA slice is
deleted, and bcd reports the condition `ForeignGpuExposure` (reason `OtherGpuPath`) with a message naming what to
close. On nodes bcd joins itself, the [`container-toolkit` unit](/docs/daemon/host-config#container-toolkit) sets the
toolkit's keys so this check passes. The gateway records the
condition as a node condition (b91dcff), and since ebd9806 it is NoSchedule while true: a GPU booked on that node would
not be isolated from pods that take it by the other path, so the node takes no new work until the path is closed. On a
tenant-controlled node the condition stays advisory and changes no NoSchedule reason, as all of bcd's conditions do.
A change of the condition also wakes the node's started reservations, and while it is true every active reservation
with an allocation on the node is `Degraded` with reason `ForeignGpuExposure` ("GPUs exposed outside bcd on
node\_…", 4570dfe), whatever its binding, since its GPUs are no longer the tenant's alone. A lost binding still takes
precedence, and the exposure takes precedence over `AtomUnhealthy`. Nothing is released: the reservation keeps its
GPUs and the condition clears when bcd stops reporting the exposure.

`bcd doctor` runs the same checks (35534e5, f88bcdb): on a node bcd joined with `gpu` on, its `gpu-isolation` check
fails, with the same message, while a toolkit configuration lets unprivileged pods pick GPUs, and, while the kubelet
runs, when another GPU DRA driver offers devices on the node; on a DaemonSet node it fails on the toolkit and on an
unreadable runtime configuration. So you see why no GPU is offered before you go looking.

## Device plugin or DRA [#device-plugin-or-dra]

Since 7d24836 bcd also has a Dynamic Resource Allocation driver, `gpu.bcd.dev`, for clusters that serve
`resource.k8s.io`. A node runs the device plugins or the DRA driver, never both, since one GPU offered twice could go
to two pods. The choice is `gpuApi`:

| Where                     | Setting                                                                                                                    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| A joined worker           | The `k8s-join` unit's `gpuApi`, rendered from `policy.kubernetes.gpuApi` (e4b0c01) and recorded in the join state          |
| A hosted node (DaemonSet) | `[kubernetes] gpu_api` in `config.toml`, written by `bcdDaemonSetManifests({ gpuApi })` or `kubernetes manifest --gpu-api` |

`device-plugin` is the default. With `dra`, the same rule holds as for the device plugins: only what compute bound
with `BINDING_KUBERNETES` is offered, and [GPU isolation](#gpu-isolation) applies the same way.

* **Registration.** bcd serves the kubelet's plugin registration at
  `/var/lib/kubelet/plugins_registry/gpu.bcd.dev-reg.sock` (type `DRAPlugin`, versions `v1.DRAPlugin` and
  `v1beta1.DRAPlugin`) and the DRA service at `/var/lib/kubelet/plugins/gpu.bcd.dev/dra.sock`.
* **The ResourceSlice.** One slice per node, `<node>-gpu.bcd.dev` (driver `gpu.bcd.dev`, pool the Node's name), in
  the newest `resource.k8s.io` version the API server serves (`v1`, `v1beta2`, or `v1beta1` with its `basic` wrapper).
  It lists only the healthy devices of the node's Kubernetes holds, each named after its atom, `gpu-<uuid>`, or
  `gpu-<uuid>-gi<g>-ci<c>` and `gpu-<uuid>-s<k>` for MIG instances and slots, and since 9f860ad ending in 8 hex
  characters of its allocation, so a GPU re-bound to another allocation is another device; with the attributes `uuid`, `class`
  (`gpu`, `gpu.mig`, `gpu.slot`), `allocation`, `visibleDevice` (the GPU or MIG UUID) and `numaNode`. bcd applies it
  server-side (field manager `bcd`) with the kubelet's credentials when the devices change, and every 10 minutes
  otherwise, so a deleted slice comes back; with no device left it deletes the slice.
* **Prepare.** `NodePrepareResources` reads each claim with the kubelet's credentials, checks that its UID is the one
  the kubelet asked for and that it is allocated, then reads the holds again and answers each `gpu.bcd.dev` device's
  CDI ids (`bcd.dev/gpu=<device>` and, where NVIDIA's spec names it, `nvidia.com/gpu=<uuid>`). A device that is not
  bound to Kubernetes on this node at that moment is refused, whatever the slice said, and so is an unhealthy one,
  even for a claim allocated before it went bad (6f5a99f); the slice drops a GPU the moment it goes bad. Since
  3f73449 two more results are refused: one with `adminAccess`, which would reach a device already allocated to
  another claim ("admin access to gpu.bcd.dev devices is refused"), and a device already prepared for another claim,
  until that claim is unprepared ("is prepared for claim …"). Since 8bcbe92 the device-to-claim record is kept on
  disk, in `/var/lib/kubelet/plugins/gpu.bcd.dev/prepared.json` (0600, written through a temporary file and a
  rename), and read back when bcd starts, so it survives a restart. The check and the record happen under one lock,
  so two concurrent prepares cannot both take a device, and the record is on disk before the kubelet hears of the
  prepare; a record that cannot be written fails the claim.
  `NodeUnprepareResources` changes nothing on the host (the CDI specs are the holds'); it only removes the claim's
  devices from the record. A failed write of that is returned to the kubelet, which retries, and until then the
  devices stay prepared.
* **The reaper.** A pod whose claim still holds one of this node's `gpu.bcd.dev` devices that compute released is
  deleted after the same 30 seconds as with the device plugins (fa8611e). Other drivers' claims are left alone.

A pod then asks through a ResourceClaim for driver `gpu.bcd.dev`, selecting on the attributes. The DRA driver needs
`kubectl` on the node and the kubelet's kubeconfig, and on a DaemonSet the generated manifests' `plugins_registry` and
`plugins/gpu.bcd.dev` mounts, which they add only with `gpuApi: 'dra'` (see
[host mounts](/docs/daemon/kubernetes#host-mounts)).

## Release [#release]

When the reservation ends, the node's spec loses the assignment. bcd removes the CDI spec, and the device plugins or
the DRA slice withdraw the devices in the same moment. Since cd5fc82 it reports `AllocationReleased` only once no pod
holds a device of the allocation any more:

* The pending release is written to `<data_dir>/releasing.json` before the CDI spec goes, so a crash never loses it.
  An unreadable file fails closed (a18f423, b01130f); see [an unreadable releases file](#an-unreadable-releases-file).
* Until the report, the devices stay claimed on the node: no hold, VM, `gpu.reset`, `diag.dcgm` or host config unit
  gets them. The control plane waits for the report too (e7d922c): a released, bound Kubernetes reservation keeps its
  GPUs and host share booked, and billed, until the node reports `AllocationReleased` or the node is lost, though
  never billed past the allocation's own end (metering stops at the earlier of its release and its end). The reservation shows
  as released at once, and the spec withdraws the devices, but no other allocation can take them while pods still
  hold them. Since 0127ed7 `reservations.get` and `list` show such a released reservation with the computed condition
  `DevicesHeld` (reason `PodsHoldDevices`, `since` the release, the message naming the nodes whose pods still hold
  the devices), so the tenant can see why it is still booked and billed; it disappears once the release ends. The
  secret broker gives such a releasing allocation nothing.
  `tests/sim/kubernetes-host-resources-1k.test.ts` checks this (see
  [the control-plane simulator](/docs/operations/testing#the-control-plane-simulator)).
* **A release stuck past the booking's end** (462e28f). Pods can hold the GPUs longer than the reservation was
  booked for (bcd then reports `KubernetesReleaseStuck`, below). While the node is reachable and no other allocation
  booked those devices next, the hosts pass carries the release's device bookings `stuckReleaseStepMs` (5 minutes)
  further each time they come within half a step of their end, locking the devices as the placer does, so a GPU pods
  still hold is never offered again. `finishStaleReleases` closes a release without the node's report only when the
  node is lost, or once its bookings ended because another allocation had booked those devices next (that one may
  then fail on the node until the pods let go). Only the device bookings are carried, never the allocation's own
  end, so billing stays capped there: the carried bookings keep the devices, not the meter, running. Since e7a56d7
  (security review 3) the carry stops `maxStuckCarryMs` (24 hours) past the allocation's own end: the node has been
  `KubernetesReleaseStuck`, and NoSchedule, for that long, and `finishStaleReleases` then closes the release
  without the node's report.
* **A bound report does not end a release** (e7a56d7). An `AllocationBound` that arrives while the release is
  pending (a node re-sending it, or a restarted bcd converging again before it received the withdrawing spec) keeps
  the release's pending mark, as does an [attested binding](/docs/guides/capacity/reservations#attested-bindings)
  that replaces the binding. Before, such a report erased the mark: the later `AllocationReleased` was then ignored,
  and the node's host share stayed claimed while the GPUs stayed booked to their original end. As a further guard,
  `finishStaleReleases` also releases any unreleased allocation whose reservation is terminal (`denied`,
  `released`, `failed`, `cancelled` or `expired`) and that is not waiting for a deferred release.
* "No pod holds it" comes from the kubelet's pod-resources API (a device plugin id `<atom>@<allocation>`, or a
  `gpu.bcd.dev` claim of this node's pool named for the allocation; 14111c1). Since 8bcbe92 (security review 2) two
  more sources can keep the devices in use, and every check fails closed:
  * **A prepared DRA claim.** A `gpu.bcd.dev` device named for the allocation that the DRA driver's `prepared.json`
    still lists is in use until the kubelet unprepares its claim, whatever pod-resources says (pod-resources reports
    DRA devices only with the `KubeletPodResourcesDynamicResources` feature gate). An unreadable record counts every
    device as prepared.
  * **A pod process.** A process in a `kubepods` cgroup with a released whole GPU's `/dev/nvidia<minor>` open (the
    minor from `/proc/driver/nvidia/gpus/*/information`) holds it, which covers what pod-resources does not account
    for: static pods, a lost kubelet checkpoint. It covers whole GPUs only, since MIG instances and slots share their
    GPU's device node, and it needs the host's `/proc`: on a DaemonSet node without `hostPID` it is off.
  * **The kubelet.** Any node that is or was a worker counts: a hosted node, one with a `k8s-join` record (even a
    released or unreadable one), or one with the kubelet's kubeconfig, `/var/lib/kubelet/config.yaml` or pod-resources
    socket on disk. There, a pod-resources socket that is missing or failing means the devices count as held. Only a
    node that never was a worker answers "not in use" without asking.
* bcd checks again when the reaper sees such a pod gone, on every converge pass, and at the latest 30 seconds later.
  A release pending for more than 5 minutes raises the condition `KubernetesReleaseStuck` (reason `PodsHoldDevices`),
  naming the allocation and its devices. Since ebd9806 the gateway records it as a node condition that is NoSchedule
  while true (advisory on a tenant-controlled node): the GPUs may still be in use, so an operator looks at the node
  before it takes more work.
* An allocation bound again before its release was reported is dropped from the pending list without a report.

Pods still
holding a released `bcd.dev/*` device (the kubelet's PodResources API names them), or a released `gpu.bcd.dev` device
through a claim, are deleted once they have held it for 30 seconds, with `kubectl delete pod --wait=false --grace-period=10` and the kubelet's credentials; a pod that still holds the device a minute later is deleted with
`--grace-period=0 --force` (9f860ad). DaemonSet pods are not spared. Since 8bcbe92 the reaper keeps running after
the node stops offering GPUs, when compute released the machine, `gpu` is off, or the kubelet is restarting: on a
node that is or was a worker it deletes any pod holding a `bcd.dev` device compute does not bind there, and with
nothing offered, that is every such device. bcd never deletes a pod after a failed read of the holds. While bcd holds
devices for Kubernetes and the kubelet's pod-resources socket is missing or failing, so released devices cannot be
reclaimed, it reports the condition `PodResourcesUnavailable`. Since ebd9806 that condition is NoSchedule for the whole
node while true (advisory on a tenant-controlled node): bcd cannot tell which pods hold which devices, and placement has no NoSchedule for one binding only,
so the node takes no work of any kind until the socket answers again.

**Taking the machine back.** When a node's spec no longer names `k8s-join` (the pool dropped Kubernetes, or the
machine left the pool), bcd cordons its Node, deletes its running and pending workload pods with each pod's own grace
period capped at 30 seconds (`--grace-period=30`, 8bcbe92), records the release, which stops the device plugins or
DRA driver, and reports `KubernetesReleased` (9f860ad, 028b81c). It leaves static mirror pods, finished pods and pods
of other Nodes alone, and DaemonSet pods too unless they ask for a `bcd.dev/` resource or a resource claim; the
reaper, which keeps running, forces what still holds a device. A cluster that forbids a node
to cordon itself still gets drained. bcd does not reset the kubelet: the machine stays a cordoned member until an
operator removes it, and the unit's return uncordons it.

### An unreadable releases file [#an-unreadable-releases-file]

If `<data_dir>/releasing.json` (by default `/var/lib/bcd/releasing.json`) exists but cannot be read or parsed, bcd
no longer knows which releases were pending, and a pod of one of them may still hold its GPUs. Until a18f423 the file
was read as empty, so those GPUs lost their claim. Since then (security review) bcd fails closed:

* It renames the file to `releasing.json.unreadable-<unix seconds>` (kept, never deleted) and writes the marker
  `<data_dir>/releasing.unreadable` with the time and the error.
* While the marker exists, every GPU of the node counts as claimed by "(pending Kubernetes releases that could not be
  read)": no hold, VM, `gpu.reset` or `diag.dcgm` gets one. The node's own live holds keep theirs as usual.
* The node reports the condition `KubernetesReleasesUnreadable` (reason `ReleasesFileUnreadable`), whose message names
  the marker's path.

The marker clears itself on a converge pass once the kube manager's `HeldOutside` (38f90fb, b01130f) answers that no
pod may hold a bcd GPU of an allocation outside the node's live set: the allocations the spec wants, the node's holds
and its pending releases. It checks the same sources as a release, each failing closed:

* device plugin ids `<atom>@<allocation>` in pod-resources, and `gpu.bcd.dev` devices, prepared in the DRA driver's
  record or claimed in pod-resources, whose name carries another allocation's tag;
* pod processes with a GPU open, asked only about the whole GPUs no live allocation holds (in whole or in part, as a
  MIG instance or slot), since a process cannot be told apart by allocation;
* a node that is or was a worker whose kubelet cannot be asked, or whose DRA record is unreadable, counts as held.
  A node that never was a worker answers "not held" at once.

When the node's holds cannot be read, nothing is decided that pass. An operator can also clear the claim by removing
the marker (by default `/var/lib/bcd/releasing.unreadable`) once they know no pod holds a GPU of a lost release.

In the control plane the condition is a node condition, NoSchedule while true (4884a65; advisory on a
`tenant-controlled` node, like the other Kubernetes conditions): the gateway writes it to `node_conditions`, publishes
`node.condition.changed` on each change, and placement offers the node for no work until the claim clears. Before
4884a65 the gateway dropped it, so placement kept booking the node's GPUs and each new hold failed on the node with
`AllocationFailed` (reason `overlap`). See [node conditions](/docs/guides/nodes/inventory-and-health).

## Host resources and pods [#host-resources-and-pods]

A `kubernetes` binding's pods live in the kubelet's own cgroups (`kubepods.slice`), where the kubelet enforces their
requests and limits. So since e01d7ff bcd creates no `bcd-alloc-<allocation>.slice` for such a hold and names no cgroup
parent in its CDI spec or env. The booked `resources` still count: bcd's host admission counts a Kubernetes hold's
memory and huge pages beside the VMs and other holds, and refuses a hold that does not fit with `resources-exhausted`,
which the control plane re-places (see [device assignments](/docs/daemon/delivery#device-assignments)). Size the pods'
requests to the reservation yourself; compute does not write them.

## Older daemons [#older-daemons]

A bcd from before 409f783 does not know `BINDING_KUBERNETES` or `DeviceAssignment.limits` and drops both as unknown
fields: the GPUs of a Kubernetes reservation would never reach the kubelet (they would fail as an unsupported binding
only after placement), and a host share would run unbounded. A GPU host's bcd now advertises two capabilities in its
`Hello`, which the gateway writes as system labels (see [capabilities](/docs/daemon/sessions#hello)):

| Capability           | System label                    | Capacity requires it for                                                                                             |
| -------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `binding.kubernetes` | `bc.dev/cap.binding.kubernetes` | Any reservation with `binding: 'kubernetes'`                                                                         |
| `devices.limits`     | `bc.dev/cap.devices.limits`     | Any reservation that books host resources, which a Kubernetes reservation does by default (the pool's per-GPU share) |

Placement enforces them on create, on re-placing a pending reservation, on repairing a booking and in
`capacity.preview`, so in a pool whose nodes run mixed bcd versions a Kubernetes reservation lands only on upgraded
nodes, and waits `pending` when none fits. An operator-set label of the same name does not count. A host without GPUs
advertises neither.

## What is missing [#what-is-missing]

* **No real cluster has run this.** Neither the device plugins nor the DRA driver has served a real kubelet.
* **Pod requests are not derived from the reservation.** The kubelet enforces what the pods ask for, not what the
  reservation booked.

  - [Host configuration](/docs/daemon/host-config#k8s-join): The k8s-join unit in detail.

  - [Reservations](/docs/guides/capacity/reservations): Requests, resources and bindings.

  - [On Kubernetes](/docs/daemon/kubernetes): bcd as a DaemonSet in an existing cluster.


# Placement (/docs/guides/capacity/placement)

> How the placer picks a node and its atoms for a request, the filters and scorers it runs, and how a guarded commit makes lost races harmless.



The placer decides where a request lands: which node of the pool, which
atoms on it, and which scalar amounts it claims. It runs for reservations, for VPS instances placed
through the capacity port, for gangs, for BookingRepair and for spot preemption. Placement never
charges quota and never talks to IAM; both happen before it is called.

## The placer contract [#the-placer-contract]

Every placer implements one interface from `@better-compute/core`:

```ts title="@better-compute/core"
interface Placer {
  readonly id: string;
  place(tx: SqlTx, request: PlacementRequest, context: PlacementContext): Promise<PlacementOutcome>;
}

type PlacementOutcome =
  | { status: 'placed'; allocations: Allocation[] } // bookings, counters and allocations written in tx
  | { status: 'pending'; reason: string } // nothing fit after replanning
  | { status: 'queued' }; // a batching placer took the request
```

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant the allocations belong to.', required: true },
  poolId: { type: 'string', description: 'The pool to place into.', required: true },
  reservationId: { type: 'string', description: 'The owner of the allocations. Exactly one of reservationId and instanceId.' },
  instanceId: { type: 'string', description: 'The owner when a VPS instance is placed.' },
  devices: { type: 'DeviceRequest[]', description: 'Device classes, counts, selectors and topology. At most 64 devices in all.', required: true },
  topology: { type: &#x22;'any' | 'nvlink-all'&#x22;, description: 'The default topology for every device request.', default: &#x22;'any'&#x22; },
  gang: { type: '{ size: number; topologyKey?: string }', description: 'Book size nodes, each taking devices (v0.5).' },
  resources: { type: 'Record<string, number>', description: 'Scalar claims such as vcpu or memoryGiB, non-negative integers.' },
  during: { type: '{ start: number; end: number }', description: 'The booked interval in epoch milliseconds; must be non-empty.', required: true },
  excludeNodes: { type: 'string[]', description: 'Nodes to leave out, as after a lost race.' },
  nodeLabels: { type: 'Record<string, string>', description: &#x22;Labels every candidate must carry with exactly these values, or with any value for '*', such as a runtime a daemon advertised (bc.dev/driver.kata: '*'). Absent: no label constraint.&#x22; },
}"
/>

A request must claim at least one device or resource; anything malformed is `INVALID_INPUT` before any
lock is taken. `PlacementContext` carries the database `clock`, a `random` function (seeded in the
simulator, so placements replay), the scheduler `plugins`, the scorer `weights`, `topN`, and the
admission context when the call comes from an admitted request.

## Two placers [#two-placers]

|                                                        | `inline` (default)                            | `pool-lease` (v0.4, for production on Postgres)                                                                                               |
| ------------------------------------------------------ | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `reservations.create` answers                          | `placed` or `pending`                         | `pending` with a `Queued` condition, then `placed` once the batch runs; with `wait: { timeoutMs }` the answer holds until then (at most 30 s) |
| Where placement runs                                   | In the request's second transaction           | In the `capacity.placer` reconciler that owns the pool's shard                                                                                |
| Pool lock                                              | Taken only for the commit, a few milliseconds | Never at admission; once per batch of at most 64, after the batch's rows and the pool index are loaded                                        |
| Throughput in `bench/placements` (PGlite, 1,000 nodes) | about 35 placements/s                         | about 118 placements/s                                                                                                                        |

```ts title="compute.ts"
import { betterCompute } from 'better-compute';

export const compute = betterCompute({
  // …database, secret, baseURL, iam
  capacity: { placer: 'pool-lease', maxBatch: 64, maxAttempts: 3 },
  scheduler: { weights: { spread: 0.5 }, topN: 5 },
});
```

Both placers commit with guarded statements, so neither can double-book. The pool-lease placer plans
queued reservations oldest first against an in-memory index of the pool, skipping rows another
transaction holds; bookings made elsewhere meanwhile are caught by the guarded commit, and the request
is planned again without that node. Instances, gangs, BookingRepair and spot preemption
always place inline, because they need an answer at once. The inline placer stays the library default
because it is synchronous and easier to test. See [scaling](/docs/operations/scaling) for sizing.

## The pipeline [#the-pipeline]

`planPlacement` is the pure half of placement: no SQL, no clock. The inline placer runs it and then
commits the chosen plan:

<Mermaid
  chart="flowchart TD
  V[&#x22;Validate the request&#x22;] --> P[&#x22;Read the pool (archived: pending)&#x22;]
  P --> AD[&#x22;Scheduler plugins: admit()&#x22;]
  AD --> C[&#x22;Candidates: SQL shortlist, or every node of the pool&#x22;]
  C --> F[&#x22;Filters: eligibility, excluded, labels, scalars, plugin filters, device fit&#x22;]
  F --> K[&#x22;Atom pick: plugin preferAllocation, else NVLink and NUMA aware&#x22;]
  K --> S[&#x22;Scorers: binpack, topology, fragmentation, spread, plugins&#x22;]
  S --> T[&#x22;Random pick among the top N&#x22;]
  T --> L[&#x22;Lock the pool FOR UPDATE, then guarded commit in a savepoint&#x22;]
  L -->|&#x22;lost race&#x22;| X[&#x22;Exclude that node, replan (up to maxAttempts)&#x22;]
  X --> C
  L -->|&#x22;committed&#x22;| OK[&#x22;placed&#x22;]"
/>

Plans are made on plain reads. The pool row lock is taken only right before the first commit, and the
commit re-validates the node, the atoms and the scalars, so the critical section is the commit alone.

## Eligibility filters [#eligibility-filters]

A node that fails a filter is rejected with a filter name and a reason. These are the exact reasons,
checked in this order:

| Filter               | Reason                                                                    | Capacity bucket |
| -------------------- | ------------------------------------------------------------------------- | --------------- |
| `excluded`           | `excluded by the request`                                                 |                 |
| `eligible`           | `in another pool`                                                         |                 |
| `eligible`           | `role <role>`                                                             | excluded        |
| `eligible`           | `tenant-controlled nodes add no capacity`                                 | excluded        |
| `eligible`           | `tenant-owned node outside its owner's pool`                              | excluded        |
| `eligible`           | `in maintenance`                                                          | maintenance     |
| `eligible`           | `cordoned`                                                                | maintenance     |
| `eligible`           | `provision state <state>`                                                 | provisioning    |
| `eligible`           | `awaiting its first converged spec`                                       | provisioning    |
| `eligible`           | `connectivity <connectivity>`                                             | unreachable     |
| `eligible`           | `NoSchedule: <reasons>`                                                   | unhealthy       |
| `labels`             | `needs label <key>=<value>` (the first `nodeLabels` entry the node lacks) |                 |
| `fits`               | `no <resource>`, or `<resource>: <n> requested, <m> free`                 |                 |
| a plugin filter's id | the plugin's reason                                                       |                 |
| `fits`               | `<n> of <count> <class> free` (plus `matching the selector`)              |                 |
| `topology`           | `no <count> <class> fully connected by NVLink`                            |                 |

`NoSchedule` reasons come from `nodes.no_schedule_reasons`, which conditions, taints and liveness
maintain in the same transaction as their cause (see [inventory and health](/docs/guides/nodes/inventory-and-health)).
The one reason a request can tolerate is a dedication to its own tenant: a node tainted `dedicated=acme` places
`acme`'s work and no one else's (see [operating nodes](/docs/guides/nodes/operating-nodes)). Whole-node leases,
which ignore every other NoSchedule reason, honour dedications too (`leasableSqlFor`), in their candidate query, at
commit and in BookingRepair. The schedulability check exists twice, as `ineligibility(node, pool, tenantId?)` in
TypeScript and as the SQL predicate the commit locks with (`schedulableSqlTolerating`), and a test keeps the two in
agreement. A third form, `mayBeSchedulableSql` (schedulable for some tenant: no reason, or one dedication alone),
drives the node controller's wake edge, so a node that becomes schedulable only for its dedicated tenant still
wakes that tenant's pending reservations.

Scalar room is `floor(allocatable × ratio) − allocated` from the node's `node_resources` row. TypeScript's floor matches the SQL
guards' numeric one (`scalarCapacity()`, a44b886, adds an epsilon that absorbs float rounding), so `100 × 0.29` is
29 in the pipeline, the pool index and the rollup, not 28, and the last unit places; the guarded commit still
decides. A queued reservation is not placed into a pool archived after it
was queued: the batch placer reads the locked pool row, and such a reservation stays pending with `the pool is
archived`, as inline placement answers.

`nodeLabels` (v0.2) is checked on every path: the pipeline filter above, the SQL shortlist (a `node_labels` check
added only when the request names labels, so other requests pay nothing) and the pool-lease index, which memoizes
its rankings per label set. The labels a request usually needs are the ones the gateway writes at every `Hello`:
each driver the daemon advertises becomes the system label `bc.dev/driver.<id>=<version>`, and each kind it serves
`bc.dev/driver-kind.<id>.<kind>=1` (14ceb1e; see [labels](/docs/guides/nodes/inventory-and-health#labels)). Plans read labels without locks, and the commit
reads the locked nodes' labels again (bb6da07, in `commitPlan` and `commitPlansBulk`), so a label that went away in
between, such as a capability a downgraded daemon no longer advertises, makes that commit a lost race, and the placer
replans without the node. A value of `'*'` (`anyLabelValue`) asks only for the label to exist, whatever its value,
on every path. The capacity API does not take `nodeLabels` from callers; a module sets it on the requests it builds
through the capacity port. The vps module does (PLAN.md's walkthrough 4): every hosted instance asks for
`bc.dev/driver.<runtime>: '*'` (`runtimeNodeLabels`), so it lands only on hosts whose daemon advertised that runtime,
in any version. Since 0fca413 that includes bcd's built-ins, `libvirt-kvm` and `cloud-hypervisor`, which a host build
runs (and advertises) only when `virsh` or the `cloud-hypervisor` binary is installed, and `cloud-hypervisor` under
`bcd helper` only since 7fe74a1; before, the built-ins asked for no label and an instance could be placed on a host
without its runtime. A host that does not advertise the runtime is passed over, and with none left the instance is
`Unschedulable` with a message naming the label.

Capacity sets `nodeLabels` on its own reservations too (409f783, `requiredCapabilities`): a reservation that books host
resources asks for `bc.dev/cap.devices.limits: '*'`, and one with `binding: 'kubernetes'` for
`bc.dev/cap.binding.kubernetes: '*'`, the capabilities a GPU host's bcd advertises in `Hello` when it enforces
`DeviceAssignment.limits` and serves Kubernetes bindings. A `node` binding asks for neither. It applies on create,
re-place, repair and `capacity.preview`, where a node without the label is rejected with filter `labels`. Capacity
reads a `bc.dev/` label only when its source is `system`, so an operator cannot fake a capability. See
[capabilities](/docs/daemon/sessions#hello).

## Device fit and selectors [#device-fit-and-selectors]

For each device request, the candidates are atoms of the requested class that are free for the whole
`during` window, healthy, bookable, and match the request's selector. A selector is a list of
`matchExpressions`, all of which must hold. Each expression reads a device attribute first, then a
node label with the same key.

```json title="A selector"
{
  "matchExpressions": [
    { "key": "gpuProduct", "op": "In", "values": ["H100-SXM5-80GB"] },
    { "key": "vramGiB", "op": "Gte", "values": [80] },
    { "key": "fabric.domain", "op": "Exists" }
  ]
}
```

* Operators: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Gte`, `Lt`, `Lte`.
* `In` and `NotIn` compare as strings; the four comparisons are numeric and take exactly one value.
  A value that is not a number never matches a comparison.
* At most 32 expressions per selector and 64 values per expression. Keys start with a letter and may
  contain letters, digits, `.`, `_`, `/` and `-`.

The CLI and `parseSelector` (exported by `better-compute/capacity`) accept a shorthand, and
`formatSelector` prints one back:

```bash
better-compute reserve --tenant acme --pool pool_01k… --gpus 8 --for 3d \
  --selector "gpuProduct in (H100-SXM5-80GB), vramGiB >= 80, mig.capable = true, fabric.domain"
```

`key in (a, b)` and `key notin (a, b)` are sets, `=` and `!=` are one-value `In` and `NotIn`, a bare
`key` is `Exists`, and `!key` is `DoesNotExist`. A bare key matches whatever the value is, so write
`mig.capable = true` rather than `mig.capable` when the value matters. Device attributes include `gpuProduct`, `gpuModel`,
`vramGiB`, `numaNode`, `architecture`, `computeCapability` and `migCapable`; node labels include
`gpu.product`, `gpu.model`, `cuda.driver.major`, `mig.capable` and `fabric.domain`.

## Atom pick and NVLink [#atom-pick-and-nvlink]

When a node has more matching atoms than the request needs, the placer picks which ones:

1. A scheduler plugin's `preferAllocation` picker for the device class wins. It must return exactly
   `count` distinct free candidates, or the node is rejected.
2. Otherwise the default pick prefers NVLink-complete sets, then the fewest NUMA nodes. It grows a set
   from every seed atom, adding the atom whose worst link to the set is best, same NUMA node as the
   seed breaking ties, and keeps the set with the best worst link, then the smallest NUMA spread.

Links are ranked `NV` > `PIX` > `PXB` > `PHB` > `NODE` > `SYS`, as `nvidia-smi topo` prints them; MIG
instances and slots of one GPU count as `NV` to each other. With `topology: 'nvlink-all'` (the CLI's
`--nvlink`), a pick of more than one GPU whose worst link is not `NV` is rejected.

The placer reads links from the capacity table `capacity_device_links`, filled by
`syncDeviceLinks(tx, nodeId, gpusSection)` from the inventory topology matrix. The built-in inventory
controller does not call it in this release, so unless your host does, every pair of GPUs has an
unknown link: the topology scorer treats such a pick neutrally, and an `nvlink-all` request for more
than one GPU cannot be placed.

## Scorers [#scorers]

Every node that passes the filters gets a score between 0 and 1 from each scorer, and its total is the
weighted sum:

| Scorer          | Weight | Score                                                                                                       |
| --------------- | ------ | ----------------------------------------------------------------------------------------------------------- |
| `binpack`       | 1      | `1 − freeAfter / total` over atoms of the requested classes: fuller nodes first, so 8-GPU boards stay whole |
| `topology`      | 1      | 1 for one GPU; `1 − worst / 5` by link rank; 0.5 when a link is unknown                                     |
| `fragmentation` | 0.25   | 1 when the node ends up full, 0.5 when the free atoms left are a power of two, else 0                       |
| `spread`        | 0      | `1 − binpack`, for deployments that prefer spreading                                                        |

Plugin scorers add their own ids; their scores are clamped to 0..1 and weighted by the scorer's
`weight` (1 by default). `scheduler.weights` overrides any weight by id, built-in or plugin. Ties
break on node id, so a plan is reproducible. Finally the placer picks at random among the best `topN`
(default 5), so concurrent placements into one pool rarely collide on the same node.

```ts title="rack-affinity.ts"
import { defineSchedulerPlugin } from 'better-compute';

export const rackAffinity = defineSchedulerPlugin({
  id: 'rack-affinity',
  filters: [
    {
      id: 'no-quarantine-rack',
      filter: (request, node) => (node.labels['rack'] === 'r13' ? { reason: 'rack r13 is quarantined' } : true),
    },
  ],
  scorers: [{ id: 'rack-a', weight: 0.5, score: (request, node) => (node.labels['rack']?.startsWith('a') ? 1 : 0) }],
});
// betterCompute({ scheduler: { plugins: [rackAffinity] } })
```

[Scheduler plugins](/docs/extending/scheduler-plugins) covers the `admit` hook and pickers.

## The shortlist [#the-shortlist]

Loading every node, device and booking of a large pool for each request is expensive. When a request
has no selectors, no gang and no plugin scorers, the inline placer first asks SQL for every
schedulable node with enough free atoms and scalar room, ranks them by the part of the score that
depends only on counts, and plans only the best 48 (at least 8 × `topN`). It uses that plan only when
no node outside the shortlist could reach the top N; otherwise it plans the whole pool. The result is
the same ranking either way, which `tests/capacity-shortlist.test.ts` checks.

## Commit and lock order [#commit-and-lock-order]

The chosen plan is written inside `SAVEPOINT capacity_place_<attempt>`, with statements that return
fewer rows instead of raising:

```sql title="The guarded commit"
SELECT n.id FROM nodes n JOIN pools p ON p.id = n.pool_id
 WHERE n.id = $1 AND n.pool_id = $2 AND <schedulable> FOR SHARE OF n;       -- node still schedulable
SELECT id, class FROM devices
 WHERE id = ANY($atoms) AND node_id = $1 AND healthy AND bookable FOR SHARE; -- atoms still healthy
INSERT INTO device_bookings (device_id, allocation_id, during)
  SELECT unnest($atoms), $alloc, tstzrange($start, $end)
  ON CONFLICT DO NOTHING RETURNING device_id;                                -- fewer rows: booked meanwhile
UPDATE node_resources SET allocated = allocated + $n
 WHERE node_id = $1 AND resource = $r AND allocated + $n <= floor(allocatable * ratio)
 RETURNING allocated;                                                        -- no row: ran out
INSERT INTO allocations (…) VALUES (…);                                      -- then the rollup delta
```

A short answer rolls back to the savepoint with one of these reasons, excludes the node, and replans:
`<node> is no longer schedulable`, `an atom on <node> turned unhealthy or unbookable`,
`an atom on <node> was booked concurrently`, or `<resource> on <node> ran out`. After `maxAttempts`
(default 3) lost races the answer is `pending` with
`Unschedulable after 3 conflicting attempts: …`.

Every transaction in capacity that locks several of these tables takes them in one order:
`quota_usage` → `pools` → `nodes` → `devices` → `node_resources` → `pool_capacity_rollup`. Quota is
charged before placement and refunded before a release, so reserve and release never wait on each
other in a cycle. Code that cordons, taints or releases nodes follows the same order.

## Reading a pending reason [#reading-a-pending-reason]

When nothing fits, the reason summarizes the rejections, most common first, at most five groups:

```text
no node fits (3 nodes): 2 × fits: 1 of 4 gpu free; 1 × cordoned
```

An empty pool answers `the pool has no nodes`. A reservation stores the reason in its `Unschedulable`
condition (reason `NoFit`), so [`reservations.get`](/docs/reference/api/reservations#get) shows it.

## Previewing a placement [#previewing-a-placement]

[`capacity.preview`](/docs/reference/api/capacity#preview) runs the same pipeline without the commit.
It takes the same input as `reservations.create`, including a syntactically valid `idempotencyKey`
that is never stored, and is decided as `compute:capacity:read` on the pool, so it checks neither quota
nor the reservation policy.

```ts
const preview = await compute.api.capacity.preview(credential, {
  tenantId: 'acme',
  poolId,
  devices: [{ class: 'gpu', count: 8, topology: 'nvlink-all' }],
  duration: '1d',
  idempotencyKey: 'preview',
});
// { fits: true, candidates: [{ nodeId: 'node_…', atoms: ['GPU-…', …], score: 2.25 }] }
// or { fits: false, candidates: [], reason: 'no node fits (…)', preemptible?: { reservations, startsAt } }
```

`candidates` lists up to 10 nodes, best first. A gang preview shows the members of the best gang, and
a `binding: 'node'` preview the leasable nodes it would take. `preemptible` appears for a regular
request that would fit by preempting spot work; see [spot and preemption](/docs/guides/capacity/spot-and-preemption).


# Pools (/docs/guides/capacity/pools)

> Pools group the nodes of one site under one GPU model, sharing mode and policy, and offerings make a pool visible to other tenants with a quota.



A pool is the unit you schedule into and sell from. It lives in one
site, holds machines with **one GPU model**, shares its GPUs in one way, and
carries the policy its nodes and reservations follow. Every reservation names exactly one pool, and
every quota is per pool.

A pool has an owner tenant. For platform capacity that is the root tenant; a customer that brings its
own hardware owns the pools on its `custom` site. Other tenants see a pool only when its owner
**offers** it to them, with a quota.

## What a pool is made of [#what-a-pool-is-made-of]

<TypeTable
  type="{
  id: { type: 'string', description: 'pool_ followed by a ULID, or an id derived from an idempotency key.' },
  ownerTenantId: { type: 'string', description: 'The tenant that created the pool and may offer, update and archive it.' },
  siteId: { type: 'string', description: 'The site the pool lives in. It never changes.' },
  class: {
    type: &#x22;'metal-host' | 'metal-dedicated' | 'vps-host' | 'provider'&#x22;,
    description: 'What the machines are for. It sets the role joining machines get and never changes.',
  },
  gpuModel: { type: 'string', description: 'The GPU family, such as H100 (1 to 32 letters, digits or hyphens). Policies and meters use it.' },
  sharing: {
    type: &#x22;'exclusive' | 'mig' | 'timeslice' | 'mps'&#x22;,
    description: 'How GPUs are handed out. It never changes after creation.',
    default: &#x22;'exclusive'&#x22;,
  },
  agent: {
    type: &#x22;'bcd' | 'none'&#x22;,
    description: 'bcd when machines run the daemon; none when they are reached only through a provider API.',
    default: &#x22;'bcd' ('none' for provider pools)&#x22;,
  },
  providerAccountId: { type: 'string', description: 'provider pools only: the provider account (pacct_) that backs the pool.' },
  policy: { type: 'PoolPolicy', description: 'Acceptance checks, overcommit, timers and sharing parameters; see below.', default: '{}' },
  version: { type: 'number', description: 'Raised by every update.' },
}"
/>

### Classes [#classes]

The class decides the role a machine gets when it joins the pool (see
[joining](/docs/guides/nodes/joining)), and only `host` nodes add schedulable capacity:

| Class             | Node role   | Used for                                                                                                        |
| ----------------- | ----------- | --------------------------------------------------------------------------------------------------------------- |
| `metal-host`      | `host`      | Bare-metal GPU servers running `bcd`, reserved as holds                                                         |
| `vps-host`        | `host`      | Hypervisors that run [VPS instances](/docs/guides/vps/instances)                                                |
| `provider`        | `host`      | Machines of a [provider account](/docs/guides/vps/providers), with `agent: 'none'`                              |
| `metal-dedicated` | `dedicated` | Machines [metal](/docs/guides/bare-metal/deploying) deploys whole; `dedicated` nodes add no reservable capacity |

### One GPU model [#one-gpu-model]

A pool holds one `gpuModel`, the family name inventory derives from the NVML name (`H100` for
`H100-SXM5-80GB`). Policies, IAM attributes and meters (`gpu.h100.hours`) all key on it, so a
pool that mixed models would bill and authorize wrongly. Selectors can still narrow within the model by
`gpuProduct`, `vramGiB` or any other attribute ([placement](/docs/guides/capacity/placement)).

### Sharing modes [#sharing-modes]

* `exclusive` (the default): each GPU is one atom, booked whole.
* `mig`: GPUs in MIG mode expose their instances as `gpu.mig` atoms.
* `timeslice` and `mps`: each GPU is split into `gpu.slot` atoms. These modes give no isolation
  between workloads, so these pools are single-tenant: they can be offered only to their owner.

[Fractional GPUs](/docs/guides/capacity/fractional-gpus) covers the atoms and what a request must ask
for in each mode.

## Pool policy [#pool-policy]

`policy` is a JSON object. Unknown fields are stored as given; these are the fields the code reads:

<TypeTable
  type="{
  acceptance: {
    type: '{ kind: string; with?: object }[]',
    description: 'Checks a node must pass on adoption before it becomes schedulable. v0.1 runs diag.nvml.',
  },
  overcommit: {
    type: 'Record<string, number>',
    description: 'Overcommit ratios for scalar resources such as vcpu and memoryGiB. GPUs are never overcommitted.',
  },
  lostAfter: {
    type: 'Duration',
    description: 'How long an unreachable node waits before it is declared lost. Applied only to provider pools, never to metal.',
  },
  disruptionBudget: {
    type: 'number',
    description: 'The share of the pool that remediation and rollouts may disrupt at once, from 0 to 1.',
    default: '0.05',
  },
  maxPending: {
    type: 'Duration',
    description: 'How long a reservation may stay pending before it fails with MaxPendingExceeded.',
    default: &#x22;'24h'&#x22;,
  },
  hardwareProfile: { type: 'string', description: 'The metal hardware profile whose driver bindings nodes inherit.' },
  joinAttestors: {
    type: 'string[]',
    description: &#x22;(v0.5, 1516728) The attestors that may admit a machine into the pool, by id: for example ['tpm'] to require the TPM, or ['tpm', 'x509pop']. A token join is 'join-token' and must be listed to be allowed. JoinService refuses a join or reclaim admitted by any other attestor with ACCESS_DENIED (reason pool-attestor). pools.create and pools.update refuse a list that is not 1 to 16 ids of attestors the control plane runs. Unset: any. See /docs/guides/nodes/attestation#requiring-an-attestor.&#x22;,
  },
  trustProvisioningNetwork: {
    type: 'boolean',
    description: '(v1.0, 8de6e2f) Bare metal: bind a PXE MAC that discovery proposes at once, without metal.confirmMac. Only for a provisioning network no untrusted host can reach: any host on it can claim a machine\'s UUID. The ramdisk\'s UUID and NICs are then checked against the machine. See /docs/guides/bare-metal/sites#discovering-new-machines.',
    default: 'false',
  },
  sharing: {
    type: '{ replicas?: number }',
    description: 'timeslice and mps pools: slot atoms per GPU (1 to 48, the most MPS serves; pools.create and pools.update refuse more, and inventory makes at most 48, a708aa6). Defaults are 4 for timeslice and 10 for mps. The same number reaches each host in the gpu-sharing config unit (/docs/daemon/host-config#gpu-sharing).',
  },
  hostReserve: {
    type: '{ vcpu?: number; memoryGiB?: number }',
    description: &#x22;(v1.1) What each host keeps for itself out of its bookable CPUs and memory, for pools other than vps-host. Default 2 CPUs and 8 GiB; huge page pools come out of memory on top of it.&#x22;,
  },
  perGpu: {
    type: '{ vcpu?: number; memoryGiB?: number }',
    description: &#x22;(v1.1) The host CPUs and memory a reservation books per whole GPU when it asks for no resources. Default: the smallest share any GPU host of the pool can give. See /docs/guides/capacity/reservations#host-resources.&#x22;,
  },
  hugepages: {
    type: &#x22;{ size: '2Mi' | '1Gi'; count: number; perNuma?: boolean }&#x22;,
    description: &#x22;(v1.1) Huge pages every host reserves, through the hugepages config unit (per NUMA node with perNuma). See /docs/daemon/host-config#hugepages.&#x22;,
  },
  kernelModules: {
    type: '{ load?: string[]; options?: Record<string, string> }',
    description: '(v1.1) Kernel modules every host loads, through the kernel-modules config unit. See /docs/daemon/host-config#kernel-modules.',
  },
  cpu: {
    type: &#x22;{ governor?: 'performance' | 'schedutil' | 'ondemand' | 'powersave'; isolate?: string }&#x22;,
    description: '(v1.1) The cpu config unit on every host: the cpufreq governor, and a cpulist of CPUs isolated from the kernel scheduler (never CPU 0; effective after a reboot), which pinned guests take first. See /docs/daemon/host-config#cpu.',
  },
  iommu: {
    type: '{ passthrough?: boolean }',
    description: '(v1.1) The iommu config unit: the IOMMU on for GPU passthrough to VMs (effective after a reboot); set it on vps-host pools with GPU flavors. See /docs/daemon/host-config#iommu.',
  },
  gpu: {
    type: '{ powerLimitWatts?: number; ecc?: boolean }',
    description: &#x22;(v1.1) The gpu-settings config unit: each GPU's power cap (1 to 10,000 W) and ECC mode. See /docs/daemon/host-config#gpu-settings.&#x22;,
  },
  kubernetes: {
    type: &#x22;{ apiServer?, caCertSha256?, joinTokenRef?, labels?, taints?, kubernetesVersion?, gpu?, gpuApi?: 'device-plugin' | 'dra', platformOperated? }&#x22;,
    description: &#x22;(v1.1) The pool's hosts work for a Kubernetes cluster, and reservations may use binding 'kubernetes'. apiServer, caCertSha256 and joinTokenRef together make every host join through the k8s-join unit; none of them marks a pool whose bcd runs as a DaemonSet in the cluster. gpuApi picks bcd's device plugins or its DRA driver. On a site that is not custom, the join set needs platformOperated: true (e4d14cd). platformOperated counts only on a pool the platform owns. See /docs/guides/capacity/kubernetes.&#x22;,
  },
  telemetry: {
    type: '{ otlpEndpoint?, intervalSeconds?, headersRef?, resourceAttributes? }',
    description: &#x22;(v1.1) Where the pool's hosts export bcd's metrics, over the deployment's nodeTelemetryModule default. See /docs/operations/observability#node-metrics-from-bcd.&#x22;,
  },
  migLayout: { type: 'string', description: 'mig pools: the layout the mig-layout config unit applies, such as all-1g.10gb. A change reaches each node only once its GPUs are free (the MIG layout gate, /docs/guides/capacity/fractional-gpus#the-mig-layout-gate).' },
  tenantRoot: {
    type: 'boolean',
    description: &#x22;Dedicated-root pools (v0.5): reservations may use binding 'node' to lease whole machines.&#x22;,
  },
  singleTenant: {
    type: 'boolean',
    description: &#x22;Offered to at most one tenant besides the owner, never with includeDescendants (a second offer is INVALID_STATE). VPS container runtimes, which share the host kernel, require it. pools.update refuses to turn it on while more than one other tenant, or an includeDescendants offer, holds the pool.&#x22;,
  },
  replacePreempted: {
    type: 'boolean',
    description: &#x22;(P3) Provider pools: when the provider reclaims an instance's Spot machine, recreate the instance in place on a new machine under the same id instead of leaving it lost. Replacements back off and stop after three in a row that never reach running. pools.create and pools.update refuse a value that is not a boolean (INVALID_INPUT). See /docs/guides/vps/instances#spot-preemption.&#x22;,
    default: 'false',
  },
  workloadIdentity: {
    type: 'boolean',
    description: &#x22;VPS pools (v0.5): hosted instances may get workload identity tokens from their host's metadata service. Off by default. See /docs/guides/vps/instances#workload-identity.&#x22;,
  },
  admission: {
    type: &#x22;{ budgetCheck?: 'none' | 'meter' }&#x22;,
    description: &#x22;'meter': reservations.create, reservations.extend and instances.create call IAM billing.check after the decision and refuse a spent budget with SPEND_LIMIT_REACHED. An offer's onBudgetExceeded stop-new or drain turns the same check on.&#x22;,
  },
}"
/>

The placer reads the overcommit ratio stored on each `node_resources` row, not the policy. For `vps-host` pools the
vps module's `vps.hosts` work writes `vcpu`, `memoryGiB` (less the huge page pools) and `diskGiB` from each host's
inventory with the pool's ratio (default 1, clamped to 0.1 to 100). For every other pool, since 8a7f8eb, capacity's
`capacity.hosts` reconciler (every 30 seconds) writes `vcpu` (logical CPUs less `hostReserve.vcpu`, with the ratio)
and `memoryGiB` (installed memory less the huge page pools and `hostReserve.memoryGiB`, with the ratio), and for every
pool `hugepages2Mi` and `hugepages1Gi` (the configured pages, never overcommitted). `tenant-controlled` nodes get no
rows, a row never shrinks below what is allocated, and unchanged rows are not rewritten. GPU reservations book these
rows with their devices (see [host resources](/docs/guides/capacity/reservations#host-resources)).

The host policy fields (`hugepages`, `kernelModules`, `cpu`, `iommu`, `gpu`, `kubernetes`, and a `sharing` mode of
`mps` or `timeslice`) become config units in every host's NodeSpec, through provision (see
[host configuration](/docs/daemon/host-config#where-the-units-come-from)). Since 37dcd02 `pools.create` and
`pools.update` (when it changes the policy) run the modules' `pool-policy` validators before writing, and provision
registers one that checks these fields as nodes would get them, so a bad value is `INVALID_INPUT` at the API; a
validator's failure that is not a compute error reads `policy: ...`. The server adds its own validator for
`joinAttestors` (1516728), which knows the attestors JoinService runs: `join-token` and every registered one.
`admission.budgetCheck: 'meter'` adds a Better IAM
`billing.check` to every reservation and instance admission in the pool, on top of the IAM policy described in
[quotas and usage](/docs/guides/capacity/quotas-and-usage).

## Create a pool [#create-a-pool]

[`pools.create`](/docs/reference/api/pools#create) is authorized as `compute:pools:create` on the
tenant, and the site must belong to that tenant. Platform pools are therefore created by the root
tenant on its `shared` or `dedicated` sites, and a customer creates pools on its own `custom` site.

```ts title="create-pool.ts"
const pool = await compute.api.pools.create(rootCredential, {
  tenantId: 'root',
  siteId: 'site_01k…',
  class: 'metal-host',
  gpuModel: 'H100',
  sharing: 'exclusive',
  policy: {
    acceptance: [{ kind: 'diag.nvml' }],
    maxPending: '6h',
  },
  idempotencyKey: 'pool-fra1-h100',
});
```

* `ownerQuota` sets the owner's own quota rows. It defaults to `{ gpu: 2147483647 }`, effectively
  unlimited whole GPUs. Owners of `mig`, `timeslice` or `mps` pools must add `gpu.mig` or `gpu.slot`
  rows themselves, because a missing row grants 0.
* With `idempotencyKey`, the pool id is derived from the tenant, the caller and the key, as for
  reservation drafts. A retry returns the same pool; the same key with a different site, class,
  sharing mode, agent or provider account is `CONFLICT`. [Fleet config](/docs/guides/fleet-config)
  uses this to plan pools it has not created yet.
* `class: 'provider'` needs `providerAccountId` and `agent: 'none'`; other classes refuse
  `providerAccountId`. Both are `INVALID_INPUT`.
* A site the tenant does not own is `NOT_FOUND`.

The pool starts with an empty rollup and emits `pool.created`. Machines join it with a join token (see
[joining](/docs/guides/nodes/joining)).

## Offerings [#offerings]

An offering is a `pool_offers` row. It makes the pool exist in another
tenant's view and creates that tenant's quota rows in the same transaction.

```ts title="offer.ts"
await compute.api.pools.offer(rootCredential, {
  tenantId: 'root', // the pool's owner, acting
  poolId,
  toTenantId: 'acme',
  quota: { gpu: 32 },
  priority: 10,
});
```

* [`pools.offer`](/docs/reference/api/pools#offer) is `compute:pools:share` on the pool, and only the
  owner can call it (anyone else gets `NOT_FOUND`). Offering again updates the row and the quota.
* `quota` maps resources to limits: `gpu`, `gpu.mig`, `gpu.slot`, or a scalar such as `vcpu`. A
  resource without a quota row grants 0.
* `includeDescendants` and `priority` (-1000 to 1000, default 0) are stored with the offer and
  round-trip through fleet config. In this release the capacity module matches the offered tenant
  exactly and the placer does not read `priority`.
* The **offering validator** refuses, with `INVALID_STATE`, to offer a `timeslice` or `mps` pool, or a
  pool that holds `tenant-owned` nodes, to anyone but its owner. A `policy.singleTenant` pool can be
  offered to one tenant besides its owner, and never with `includeDescendants`. An archived pool cannot
  be offered either.

The offered tenant then sees the pool in [`pools.list`](/docs/reference/api/pools#list) and
[`pools.get`](/docs/reference/api/pools#get), each pool with `owned`, `archived` and the tenant's own
quota rows:

```json title="pools.list, as acme sees it"
{
  "items": [
    {
      "id": "pool_01k…",
      "ownerTenantId": "root",
      "siteId": "site_01k…",
      "class": "metal-host",
      "gpuModel": "H100",
      "sharing": "exclusive",
      "agent": "bcd",
      "policy": { "acceptance": [{ "kind": "diag.nvml" }], "maxPending": "6h" },
      "version": 1,
      "owned": false,
      "archived": false,
      "quotas": [{ "tenantId": "acme", "poolId": "pool_01k…", "resource": "gpu", "max": 32, "used": 4, "available": 28 }]
    }
  ]
}
```

[`pools.unoffer`](/docs/reference/api/pools#unoffer) removes the offer and answers
`{ removed: boolean }`. It lowers the tenant's quotas on the pool to what is in use, so live
reservations keep their bookings and charge while nothing new fits. Pending reservations of that
tenant fail with `PoolGone` at their next retry. A pool cannot be unoffered from its owner.

Offering also writes the pool's row in the IAM authz projection
for the tenant, with the attributes `gpuModel`, `class`, `sharing`, `region` and `owned`, so tenant
policies can decide on them (see [actions and resources](/docs/guides/iam/actions-and-resources)).

## Update and archive [#update-and-archive]

[`pools.update`](/docs/reference/api/pools#update) (`compute:pools:manage`, owner only) takes a new
`policy`, a new `gpuModel`, or both. The policy is replaced as a whole, not merged, so send every field
you want to keep. The update refreshes the pool's authz row for the owner and every offered tenant and
emits `pool.updated`. Class, site, sharing mode and agent cannot change.

[`pools.archive`](/docs/reference/api/pools#archive) (`compute:pools:manage`, owner only) retires a
pool without disturbing what runs on it:

* every offer is removed, each offered tenant's quotas drop to what is in use, and their authz rows
  are removed;
* the pool takes no new work: new reservations are `NOT_FOUND`, and the placer answers `pending` with
  `the pool is archived`;
* placed and active reservations keep their bookings until they end, while pending ones fail with
  `PoolGone` at their next retry;
* only the owner still sees the pool, with `archived: true`.

Archiving is idempotent and emits `pool.archived` with the list of unoffered tenants.

When Better IAM deletes a tenant, compute's tenancy teardown (`capacityTenantDeleted`) archives the pools
it owns the same way (`pool.archived` with `reason: 'tenant deleted'`, quotas of the offered tenants
zeroed), withdraws every offer made to it (`pool.unoffered` to the owner) and ends its reservations. Other
tenants' live bookings in an archived pool run to their end. See
[tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants).

## The pools API [#the-pools-api]

| Method                                                 | Action                  | Decided on | Phase |
| ------------------------------------------------------ | ----------------------- | ---------- | ----- |
| [`pools.list`](/docs/reference/api/pools#list)         | `compute:pools:read`    | tenant     | v0.1  |
| [`pools.get`](/docs/reference/api/pools#get)           | `compute:pools:read`    | pool       | v0.1  |
| [`pools.create`](/docs/reference/api/pools#create)     | `compute:pools:create`  | tenant     | v0.1  |
| [`pools.capacity`](/docs/reference/api/pools#capacity) | `compute:capacity:read` | pool       | v0.1  |
| [`pools.offer`](/docs/reference/api/pools#offer)       | `compute:pools:share`   | pool       | v0.1  |
| [`pools.unoffer`](/docs/reference/api/pools#unoffer)   | `compute:pools:share`   | pool       | v0.1  |
| [`pools.update`](/docs/reference/api/pools#update)     | `compute:pools:manage`  | pool       | v0.2  |
| [`pools.archive`](/docs/reference/api/pools#archive)   | `compute:pools:manage`  | pool       | v0.2  |

All eight are served today. Events: `pool.created`, `pool.offered`, `pool.unoffered`, `pool.updated`,
`pool.archived`, and `quota.updated` from `quotas.set`. `better-compute pools list --tenant acme`
lists pools from the [CLI](/docs/reference/cli).


# Quotas and usage (/docs/guides/capacity/quotas-and-usage)

> Per-pool quotas that cap what a tenant holds at once, hourly usage windows metered from active reservations, and how both reach Better IAM billing.



Better Compute keeps two separate books. A quota caps how much a tenant may
hold in one pool at the same time; it is counted in compute's database and enforced at admission.
Usage records what a tenant actually consumed, hour by hour, and hands it to Better IAM billing,
where prices and budgets live. A quota never stops a running reservation, and billing never places
one.

## Quotas [#quotas]

Quotas live in `quota_usage`, one row per tenant, pool and resource, with a `max` and a `used`
counter guarded by `CHECK (used BETWEEN 0 AND max)`. **A missing row grants 0**: a tenant can reserve
in a pool only for the resources its offering gave it.

A request counts against quota per resource:

* each device class by count: `gpu`, `gpu.mig`, `gpu.slot`, `nic.vf`, `disk`, `node`;
* each scalar by amount: `vcpu`, `memoryGiB`, `diskGiB` (VPS instances);
* a gang counts `size` times the per-node request.

Charging happens in the second admission transaction, before placement, all or nothing inside a
savepoint, with a guarded update per resource:

```sql
UPDATE quota_usage SET used = used + $n
 WHERE tenant_id = $1 AND pool_id = $2 AND resource = $3 AND used + $n <= max;
-- no row updated: QUOTA_EXCEEDED, and every counter stays as it was
```

The refusal is [`QUOTA_EXCEEDED`](/docs/reference/errors) (HTTP 409) with the numbers in `details`:

```json
{
  "error": {
    "code": "QUOTA_EXCEEDED",
    "message": "Quota exceeded for gpu in pool pool_01k…: 9 requested, 8 of 8 left",
    "details": { "resource": "gpu", "poolId": "pool_01k…", "requested": 9, "max": 8, "used": 0 }
  }
}
```

**A reservation holds its charge until it is terminal** (decision D9). `pending`, `placed`, `binding`,
`active` and `releasing` all count, so a tenant cannot queue unlimited pending demand. Re-placements
after a bind timeout, BookingRepair and a spot re-queue keep the charge. Release, cancel, failure and
expiry refund it, and a refund never drives a counter below 0, so a double refund cannot corrupt it.

### Where quota rows come from [#where-quota-rows-come-from]

| Operation                                                                                                  | Effect on quota rows                                                          |
| ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`pools.create`](/docs/reference/api/pools#create)                                                         | Creates the owner's rows from `ownerQuota`, by default `gpu` at 2,147,483,647 |
| [`pools.offer`](/docs/reference/api/pools#offer)                                                           | Creates or resizes the offered tenant's rows from `quota`                     |
| [`quotas.set`](/docs/reference/api/quotas#set)                                                             | Sets one row for any tenant and pool (fleet operators)                        |
| [`pools.unoffer`](/docs/reference/api/pools#unoffer), [`pools.archive`](/docs/reference/api/pools#archive) | Lower the tenant's rows to what is in use                                     |

Setting a maximum never lowers it below `used`: the stored `max` becomes the larger of the two. To shrink
a quota under live reservations, set it again once they have released.

### Read and set quotas [#read-and-set-quotas]

[`quotas.get`](/docs/reference/api/quotas#get) (`compute:quotas:read` on the tenant) lists the
tenant's rows, optionally for one pool. [`quotas.set`](/docs/reference/api/quotas#set)
(`compute:quotas:manage` on `compute-fleet/global`, which resolves only in the root tenant) sets any
tenant's limit on any pool and emits `quota.updated`.

```ts title="quotas.ts"
const { items } = await compute.api.quotas.get(credential, { tenantId: 'acme', poolId });
// [{ tenantId: 'acme', poolId, resource: 'gpu', max: 32, used: 4, available: 28 }]

await compute.api.quotas.set(rootCredential, { tenantId: 'acme', poolId, resource: 'gpu', max: 64 });
```

Resource names start with a letter and use letters, digits, `.`, `_` and `-` (at most 64 characters);
`max` is a non-negative integer. An unknown pool is `NOT_FOUND`. VPS instances charge the same rows
through the capacity port, which charges, places, and gives the charge back when nothing fits (see
[instances](/docs/guides/vps/instances)).

## Usage windows [#usage-windows]

The `capacity.usage` reconciler runs every 5 minutes (`usageEvery` in the
capacity options) and writes usage windows: one row per allocation, hour
and meter in `usage_windows`.

* **The basis is reserved time** (decision D14): from the moment an allocation is bound, which is when
  the reservation becomes `active`, until it is released or its booking ends. A reservation that never
  goes active bills nothing; time spent `placed` or `binding` is not billed.
* **Windows are hour-aligned** and written only when final: the hour has closed, or billing ended inside
  it. A window never changes once written. Its quantity is the meter's rate times the fraction of the
  hour used, rounded to six decimals.
* **Every window has an idempotency key**, `cmp:<allocationId>:<windowStartMs>`, with `:<meter>` added
  for an allocation's second and later meters. Rows are inserted with `ON CONFLICT DO NOTHING`, so every
  pass is safe to repeat, including after a worker restart.
* **Each pass looks back 48 hours** and handles at most 5,000 allocations.

| Allocation                     | Meters                                                                                                                                                                                                                                                |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A `hold` reservation           | `gpu.<model>.hours`, one unit per GPU; the model is the pool's, lower-cased (`gpu.h100.hours`)                                                                                                                                                        |
| A spot reservation             | `gpu.<model>.spot.hours`                                                                                                                                                                                                                              |
| MIG instances and slots        | The same GPU meter, at the atom's share of a GPU; see [fractional GPUs](/docs/guides/capacity/fractional-gpus)                                                                                                                                        |
| A `binding: 'node'` lease      | `node.metal.hours`, plus the GPU meter for the GPUs healthy at each window's start                                                                                                                                                                    |
| A VPS instance                 | `vps.vcpu.hours`, `vps.memory.gib.hours`, and `gpu.<model>.hours` for GPU flavors, written by the `vps.usage` job                                                                                                                                     |
| A warm sandbox instance (v0.5) | `vps.sandbox.warm`, its flavor's vCPUs per hour, attributed to the tenant with no identity until a sandbox claims it; from the claim on, the lines of a VPS instance for the sandbox's owner. See [warm pools](/docs/guides/vps/sandboxes#warm-pools) |

Tags have bounded cardinality: `region`, `pool`, `gpuModel`, `project` and `agent`, never a
reservation, allocation or node id. The window is attributed to the reservation's owner identity: the
person for work an agent does in a delegated session (tagged `agent`), or the agent itself for agent
keys.

`meterAllocation` is exported by `better-compute/capacity`, so a module that books its own allocations
can meter them with the same keys and rules.

## Into Better IAM billing [#into-better-iam-billing]

The IAM bridge's `iam.usage` reconciler runs hourly, before Better IAM's daily period close, and sends
closed, unrecorded windows through `IamPort.recordUsage`. Recording is poison-isolating: a failing
batch is bisected down to the event, so one bad window never blocks the others.

* An identity deleted before the flush is re-recorded under its successor if the inbox saw one, or
  without an identity (tenant-attributed, tagged `attribution=orphaned`).
* A window that still fails on its own, such as a key reused on another meter, an unknown meter or a
  closed billing period, is dead-lettered into `usage_dead_letters`.
* A transient failure, such as IAM being unreachable, fails the whole pass, which retries later with
  the same keys.
* Each window stores `recordedAt` and its `outcome`: `recorded`, `duplicate`, `reattributed` or
  `dead-lettered`.

The meters must exist in Better IAM billing. `better-compute iam setup --meters` creates the whole catalog:
`gpu.<family>.hours` and `gpu.<family>.spot.hours` for every known GPU family, the provider-VM and reserved GPU
meters for each model in `--gpu-models` (default `H100`, `H200`, `B200`, `A100`, `L4`, `L40S`, plus `other`), `node.metal.hours`,
`vps.vcpu.hours`, `vps.memory.gib.hours`, `vps.sandbox.warm`, `vps.volume.gib.hours` and the other billable units.
Windows for a meter that does not exist are dead-lettered. Prices, invoices and spend reports are
Better IAM's; see [billing and usage](/docs/guides/iam/billing-and-usage).

## Read usage [#read-usage]

[`usage.list`](/docs/reference/api/usage#list) (`compute:usage:read`) returns a tenant's windows, or one
project's when `projectId` is given (then decided on the project). It filters by `meter`, `from` and
`to` (epoch milliseconds) and pages with `limit` (at most 500, default 100) and `cursor`.

```ts
const page = await compute.api.usage.list(credential, {
  tenantId: 'acme',
  meter: 'gpu.h100.hours',
  from: Date.UTC(2026, 8, 1),
});
```

```json title="One usage window"
{
  "idempotencyKey": "cmp:alloc_01k…:1790002800000",
  "allocationId": "alloc_01k…",
  "tenantId": "acme",
  "identityId": "…",
  "meter": "gpu.h100.hours",
  "quantity": 4,
  "windowStart": 1790002800000,
  "windowEnd": 1790006400000,
  "tags": { "region": "eu-central-1", "pool": "pool_01k…", "gpuModel": "H100" },
  "recordedAt": 1790010000000,
  "outcome": "recorded"
}
```

[`usage.export`](/docs/reference/api/usage#export) (v0.2) returns the windows of a range of at most 93 days in one
response, as CSV or JSON lines, with the total quantity per meter; it stops at 50,000 windows and says so with
`truncated`. See [exporting usage](/docs/guides/iam/billing-and-usage#export-usage).

## Spend and budgets [#spend-and-budgets]

Quotas cap GPUs, not money. Budgets are enforced in the admission decision itself: the example tenant
policy `gpu-user` that `better-compute iam setup` prints includes

```json title="gpu-user, StopWhenOverBudget"
{
  "sid": "StopWhenOverBudget",
  "effect": "deny",
  "actions": [
    "compute:reservations:create",
    "compute:reservations:extend",
    "compute:instances:create",
    "compute:sandboxes:create",
    "compute:sandboxes:update"
  ],
  "resources": ["*"],
  "conditions": { "Bool": { "principal.spendExceeded": true } }
}
```

Every create and extend is decided by Better IAM, so once a tenant's spend passes its budget no new
reservation, extension or instance is allowed. Reservations and instances that are already running continue.
The two sandbox actions cover [sandboxes](/docs/guides/vps/sandboxes): `sandboxes.create`, and `sandboxes.extend`
and `sandboxes.snapshot` under `sandboxes:update`. On top of the decision,
`reservations.create`, `reservations.extend`, `instances.create` and `sandboxes.create` run a per-pool budget check
after it (`budgetAdmissionCheck`): when the pool's `policy.admission.budgetCheck` is `'meter'`, or the tenant's
offering has `onBudgetExceeded: 'stop-new'` or `'drain'`, they call Better IAM's `billing.check` and refuse a
spent budget with [`SPEND_LIMIT_REACHED`](/docs/reference/errors#spend_limit_reached) (402). On an exceeded
budget alert, `serve`'s default IAM handler also publishes `quota.budget-exceeded` for `notify`, `stop-new` and
`drain`, and `drain` cancels the tenant's unstarted reservations in that pool. See
[budgets](/docs/guides/iam/billing-and-usage#budgets) and [policies](/docs/guides/iam/policies).


# Reservations (/docs/guides/capacity/reservations)

> Reserve GPUs from a pool with an idempotent request that Better IAM decides on, follow it from draft to released, and extend, renew or release it.



A reservation is a tenant's claim on devices of one pool for a time
window: four NVLink-connected H100s for eight hours from now, or two nodes for three days starting
Monday. It is created as a request-as-resource: the request is first written as a
draft, Better IAM decides on that draft with everything it asks for as
attributes, and only an allowed draft is placed. So a policy such as "at most 8 H100s for at most 72
hours in these regions" is one decision on the reservation itself, not a check you write in code.

## Reserve GPUs [#reserve-gpus]

  **Server:**

    ```ts title="reserve.ts"
    import { compute } from './compute';

    const reservation = await compute.api.reservations.create(
      { headers: request.headers }, // the caller's credential
      {
        tenantId: 'acme',
        poolId: 'pool_01k…',
        devices: [{ class: 'gpu', count: 4, topology: 'nvlink-all' }],
        duration: '8h',
        idempotencyKey: 'train-2026-09-24',
      },
    );
    // reservation.state: 'placed' (or 'pending' with an Unschedulable condition)
    ```
  
  **Browser client:**

    ```ts title="reserve.ts"
    import { createComputeClient } from 'better-compute/client';
    import type { compute } from './compute';

    const client = createComputeClient<typeof compute>({
      baseURL: 'https://compute.example.com',
      token: () => session.getToken(),
    });
    await client.reservations.create({
      tenantId: 'acme',
      poolId: 'pool_01k…',
      devices: [{ class: 'gpu', count: 4, topology: 'nvlink-all' }],
      duration: '8h',
      idempotencyKey: 'train-2026-09-24',
    });
    ```
  
  **HTTP:**

    ```bash
    curl -X POST https://compute.example.com/api/compute/reservations/create \
      -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" \
      -H "Content-Type: application/json" \
      -H "X-Better-Compute: 1" \
      -d '{"tenantId":"acme","poolId":"pool_01k…","devices":[{"class":"gpu","count":4,"topology":"nvlink-all"}],"duration":"8h","idempotencyKey":"train-2026-09-24"}'
    # → { "data": { "id": "rsv_…", "state": "placed", … } }
    ```
  
  **CLI:**

    ```bash
    better-compute reserve --tenant acme --pool pool_01k… --gpus 4 --for 8h --nvlink --key train-2026-09-24
    better-compute reserve --tenant acme --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256
    better-compute capacity --tenant acme --preview --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256
    better-compute reservations get rsv_… --tenant acme --query binding
    ```

    `reserve` sends a new idempotency key on every run unless you pass `--key`, and prints the key so you
    can retry. `--cpus` and `--memory` (GiB) book [host resources](#host-resources) with the GPUs, and `--kubernetes` asks
    for `binding: 'kubernetes'` (ea36a94); `capacity --preview` takes the same three. See the [CLI reference](/docs/reference/cli).
  
<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant reserving. The caller must belong to it (or be root).', required: true },
  poolId: { type: 'string', description: 'A pool the tenant owns or is offered, not archived.', required: true },
  devices: {
    type: 'DeviceRequest[]',
    description: '1 to 8 entries of { class, count (1 to 64), selector?, topology?, sharing? }; at most 64 devices in all.',
    required: true,
  },
  duration: { type: 'string | number', description: 'Like 30m, 8h, 7d or 1h30m, or milliseconds; from 1 minute to 365 days.', required: true },
  idempotencyKey: { type: 'string', description: '1 to 128 letters, digits, dots, underscores, colons or hyphens. Scoped to the tenant and the caller.', required: true },
  start: { type: 'number', description: 'Epoch milliseconds; a later start makes a calendar reservation.', default: 'now' },
  topology: { type: &#x22;'any' | 'nvlink-all'&#x22;, description: 'nvlink-all requires every GPU of a device request to be NVLink-connected.', default: &#x22;'any'&#x22; },
  binding: { type: &#x22;'hold' | 'node' | 'kubernetes'&#x22;, description: &#x22;hold hands the devices to software on the node; node leases whole machines (v0.5); kubernetes hands them to the node's kubelet, in a pool with policy.kubernetes (v1.1).&#x22;, default: &#x22;'hold'&#x22; },
  node: { type: '{ profileId: string; profileVersion?: number }', description: &#x22;binding 'node' only: the metal profile to deploy.&#x22; },
  resources: { type: '{ vcpu?: number; memoryGiB?: number; hugepages1Gi?: number }', description: 'Host CPUs (0 to 4096), memory in GiB (0 to 65536) and 1 GiB pages (0 to 4096) per node, booked with the devices (v1.1). Left out: the pool\'s per-GPU share. Refused with binding node. See host resources below.' },
  leaseSeconds: { type: 'number', description: '30 to 86400. Makes the reservation renewable: it is released unless renewed within this period.' },
  projectId: { type: 'string', description: 'A project (prj_) to account the reservation to.' },
  spot: { type: 'boolean', description: 'Preemptible, billed at the spot meter (v0.5).', default: 'false' },
  gang: { type: '{ size: number; topologyKey?: string }', description: '2 to 64 nodes, each taking devices, all or nothing (v0.5).' },
  wait: { type: '{ timeoutMs: number }', description: '0 to 30000. Holds the answer while the reservation is queued for the pool-lease placer; see waiting for placement below.' },
}"
/>

Unknown fields are `INVALID_INPUT`. `binding: 'instance'` is `UNSUPPORTED`: VPS instances are created
with `instances.create` and book through the capacity port (see [instances](/docs/guides/vps/instances)).

## The reservation object [#the-reservation-object]

<TypeTable
  type="{
  id: { type: 'string', description: 'rsv_ plus 52 characters, derived from the tenant, the caller and the idempotency key.' },
  tenantId: { type: 'string', description: 'The owning tenant.' },
  poolId: { type: 'string', description: 'The pool it is booked in.' },
  projectId: { type: 'string', description: 'The project, when the request named one.' },
  state: { type: 'ReservationState', description: 'See states and conditions below.' },
  request: { type: 'ReservationSpec', description: 'What was asked for, with duration stored in milliseconds (for example 28800000ms).' },
  during: { type: '{ start: number; end: number }', description: 'The booked window, once placed. Absent while pending.' },
  leaseExpiresAt: { type: 'number', description: 'Renewable reservations: when the lease lapses unless renewed.' },
  ownerIdentityId: { type: 'string', description: 'The identity that created it: the person, or the agent for agent keys.' },
  agentId: { type: 'string', description: 'The agent, for delegated sessions.' },
  delegationId: { type: 'string', description: 'The delegation, for delegated sessions.' },
  idempotencyKey: { type: 'string', description: 'The key it was created with.' },
  conditions: { type: 'Condition[]', description: '{ type, status, reason?, message?, since } entries.' },
  binding: { type: '{ nodeId; cdiDevices; envFile }', description: 'hold bindings, once a node reported the devices bound.' },
  operation: { type: '{ id; state; error? }', description: &#x22;The async operation behind the reservation: running, pending, succeeded or failed. See waiting for placement.&#x22; },
  version: { type: 'number', description: 'Raised by every change.' },
}"
/>

```json title="An active reservation"
{
  "id": "rsv_…",
  "tenantId": "acme",
  "poolId": "pool_01k…",
  "state": "active",
  "request": {
    "poolId": "pool_01k…",
    "devices": [{ "class": "gpu", "count": 4, "topology": "nvlink-all" }],
    "duration": "28800000ms",
    "binding": "hold"
  },
  "during": { "start": 1790000000000, "end": 1790028800000 },
  "ownerIdentityId": "…",
  "idempotencyKey": "train-2026-09-24",
  "conditions": [],
  "binding": {
    "nodeId": "node_01k…",
    "cdiDevices": ["bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…"],
    "envFile": "/run/bcd/allocations/alloc_….env"
  },
  "operation": { "id": "rsv_…", "state": "succeeded" },
  "version": 4
}
```

## Drafts and idempotency [#drafts-and-idempotency]

The draft id is deterministic: `rsv_` followed by the base32 SHA-256 of the tenant, the caller's
identity and the idempotency key. Retrying with the same key reaches the same row, and the same key
from another identity gets a row of its own, so nobody can squat on or replay someone else's request.

The stored request hash covers the validated input without `wait` and `idempotencyKey`. Durations
are normalized to milliseconds and `binding` defaults to `hold`, so `'2h'` and `7200000` are the same
request, with or without `binding: 'hold'`.
A second call with the same key:

* with the same request, on a reservation past the draft stage, returns the stored reservation with
  no second IAM decision;
* with the same request, on a draft (for example after `QUOTA_EXCEEDED`), decides the same draft again;
* with a different request, is `CONFLICT` with `details.reservationId`;
* on a refused key, stays `ACCESS_DENIED`.

Drafts are internal: `reservations.get` answers `NOT_FOUND` for them and `reservations.list` skips
them.

## The admission sequence [#the-admission-sequence]

Every `create` follows the normative sequence of PLAN.md section 3.4, and `extend` follows it against
the existing reservation. IAM is never called inside a compute transaction.

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant C as Caller
  participant A as compute api
  participant D as Postgres
  participant I as Better IAM
  C->>A: reservations.create(credential, input)
  A->>I: authenticate (cached 30 s)
  A->>A: tenant pre-check
  A->>D: tx 1: idempotency lookup, draft, authz row
  A->>I: require compute:reservations:create on the draft
  alt refused
    A->>D: draft becomes denied (or awaiting-confirmation)
    A-->>C: ACCESS_DENIED or CONFIRMATION_REQUIRED
  else allowed
    A->>D: tx 2: charge quota, place, audit row, outbox
    A-->>C: placed, pending or queued
  end"
/>

      ### Authenticate and pre-check [#authenticate-and-pre-check]

    The credential is authenticated (cached for 30 seconds). If the caller's tenant is neither the input
    tenant nor the root tenant, the answer is `ACCESS_DENIED` before compute's database is read, the same
    answer whether or not anything exists.
  
      ### Transaction 1: the draft [#transaction-1-the-draft]

    The idempotency row is looked up and locked. The pool must be owned by or offered to the tenant and
    not archived, or the answer is `NOT_FOUND`. Device classes must fit the pool's sharing mode, and
    `binding: 'node'` needs a `tenantRoot` pool, `binding: 'kubernetes'` a pool with `policy.kubernetes` that the
    tenant owns (3825ec7; a tenant the pool is only offered to is `INVALID_INPUT`). A request
    without `resources` gets the pool's per-GPU share of host CPU and memory here, stored in the request (see
    [host resources](#host-resources)). Then the draft is inserted with its row in the IAM
    authz projection, parented to the pool and owned by the caller:

    | Attribute                                   | Value                                                  |
    | ------------------------------------------- | ------------------------------------------------------ |
    | `gpuModel`                                  | The pool's GPU model, or `other`                       |
    | `gpuCount`                                  | Whole GPUs requested, times the gang size              |
    | `gpuSlices`                                 | MIG instances and slots requested, times the gang size |
    | `gangSize`                                  | 1, or the gang size                                    |
    | `durationHours`                             | The duration in hours, to three decimals               |
    | `region`                                    | The region of the pool's site                          |
    | `binding`, `spot`, `renewable`, `projectId` | As requested; `renewable` is true with `leaseSeconds`  |
  
      ### Decide exactly once [#decide-exactly-once]

    `iam.require` decides `compute:reservations:create` on `compute-reservation/<draft id>`. The resolver
    hands IAM the attributes above and the pool's relations, so pool access and request limits are one
    decision; see [policies](/docs/guides/iam/policies). A refusal marks the draft `denied`, unless the
    caller is an agent in a delegated session and IAM opened a confirmation request (below).

    Still outside any transaction, the budget check (`budgetAdmissionCheck`) runs when the pool's policy
    sets `admission.budgetCheck: 'meter'` or the tenant's offer has `onBudgetExceeded: 'stop-new'` or
    `'drain'`: it calls Better IAM's `billing.check` for the caller, and a spent budget answers
    [`SPEND_LIMIT_REACHED`](/docs/reference/errors#spend_limit_reached). The draft stays, so the same key is
    decided again once the budget's window resets. See [budgets](/docs/guides/iam/billing-and-usage#budgets).
  
      ### Transaction 2: quota and placement [#transaction-2-quota-and-placement]

    Quota is charged all-or-nothing first. `QUOTA_EXCEEDED` ends the request and leaves the draft, so a
    retry with the same key is decided again. Then the [placer](/docs/guides/capacity/placement) runs. A
    regular request that does not fit may [preempt spot work](/docs/guides/capacity/spot-and-preemption).
    The outcome is `placed`, `pending` with an `Unschedulable` condition (retried every 30 seconds), or,
    with the pool-lease placer, `pending` with a `Queued` condition. The same transaction schedules the
    lifecycle, writes the event to the outbox and records the audit row with the admission context.
  
Two details differ from the PLAN.md text. Quota is charged by the reservation service before
placement, never by the placer, because a pending reservation keeps its charge (decision D9).

## Waiting for placement [#waiting-for-placement]

`create` is async-with-wait (PLAN.md section 9.1): every reservation carries an `operation`, and a
request with `wait: { timeoutMs }` holds its answer until that operation stops running. The same client
code therefore works with both [placers](/docs/guides/capacity/placement#two-placers).

| `operation.state` | Reservation                                                                                                             |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `running`         | A draft, or `pending` with a `Queued` condition: waiting for the pool-lease batch. The only state a `wait` waits on     |
| `pending`         | `pending` without `Queued` (`Unschedulable`, retried every 30 seconds), or `awaiting-confirmation`                      |
| `succeeded`       | `placed`, `binding`, `active`, `releasing` or `released`                                                                |
| `failed`          | `failed` (`CAPACITY_UNAVAILABLE`), `denied` (`ACCESS_DENIED`), `cancelled` or `expired` (`INVALID_STATE`), with `error` |

After transaction 2 commits (or a repeated key returns the stored reservation), the server re-reads the
reservation in read-only transactions, backing off from 50 ms to 1 s, until the operation is no longer
`running`, `timeoutMs` passes (capped at 30 seconds), or the request is aborted. The answer is the last
read either way: a timeout is not an error, it returns the reservation still `Queued`. With the inline
placer a create is placed or `Unschedulable` at once, so `wait` returns immediately. `wait` is excluded
from the request hash, and `extend` does not wait.

```ts
const reservation = await client.reservations.create({ ...input, wait: { timeoutMs: 20_000 } });
if (reservation.operation?.state === 'running') {
  // Still queued after 20 s: read it again later with reservations.get.
}
```

The `better-compute reserve` command sends `wait` (its `--wait`, 20 seconds by default) and, if the
answer is still `Queued`, keeps polling `reservations.get` until the same deadline. The MCP
`reserve_gpus` tool sends `wait` from its `waitSeconds`.

## States and conditions [#states-and-conditions]

<Mermaid
  chart="stateDiagram-v2
  state &#x22;awaiting-confirmation&#x22; as awaiting
  [*] --> draft
  draft --> placed: allowed and placed
  draft --> pending: allowed, nothing fits
  draft --> denied: refused
  draft --> awaiting: delegated refusal
  awaiting --> draft: same key retried
  awaiting --> expired: deadline passed
  pending --> placed: a retry fits
  pending --> failed: maxPending or pool gone
  pending --> cancelled: cancel or release
  placed --> binding: start reached
  placed --> pending: repair failed or spot requeue
  placed --> cancelled: cancel
  binding --> active: every allocation bound
  binding --> pending: bind timeout
  binding --> failed: bind retries exhausted
  placed --> releasing: release
  binding --> releasing: release
  active --> releasing: end, release or lease lapse
  releasing --> released"
/>

`denied`, `released`, `failed`, `cancelled` and `expired` are terminal. The `capacity.reservations`
reconciler drives each reservation, woken by the work queue at the moments
that matter: the start, 15 minutes before it, the bind deadline, the end and the lease deadline.
While a reservation is active, the reconciler only sets and clears conditions from its nodes and atoms, and
never releases it (PLAN.md section 10.5): node connectivity and health changes wake it, and it rechecks every
15 minutes without them.

| Condition              | Reason                                  | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ---------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Unschedulable`        | `NoFit`                                 | Nothing fits; the message is the placer's summary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `Unschedulable`        | `MaxPendingExceeded`                    | Pending longer than the pool's `maxPending` (default 24 h); the reservation failed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `Unschedulable`        | `PoolGone`                              | The pool is no longer offered to the tenant, or archived; the reservation failed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `Unschedulable`        | `BookingRepairFailed`                   | The booking went bad before the start and nothing else fits                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `Unschedulable`        | `SpotPreempted`                         | A spot reservation that had not started was preempted and re-queued                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `Queued`               | `PoolPlacer`                            | Waiting for the pool-lease placer's next batch                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `BindTimeout`          | `NodeDidNotBind`, `NodeRejectedBinding` | The node did not bind in time or refused; the message counts attempts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `Degraded`             | `BindingLost`                           | A node reported it lost devices it had bound                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `Degraded`             | `ForeignGpuExposure`                    | A node of an active reservation reports the node condition `ForeignGpuExposure`: something besides bcd can hand its GPUs to pods, so they are not the tenant's alone (4570dfe). The message lists the nodes. `BindingLost` wins over it, and it wins over `AtomUnhealthy`; nothing is released. See [GPU isolation](/docs/guides/capacity/kubernetes#gpu-isolation)                                                                                                                                                                                                            |
| `Degraded`             | `AtomUnhealthy`                         | A booked atom of an active reservation is unhealthy; the message lists them. `BindingLost` wins when both apply                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `NodeUnreachable`      | `Unreachable`                           | A node of an active reservation is not connected; the message lists the nodes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `Preempted`            | `SpotPreempted`                         | A running spot reservation was cut back to its notice deadline                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `ConfirmationRequired` | the confirmation id                     | Waiting for a person; the message is the deadline in epoch milliseconds                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `UnhealthyAtoms`       | `NotBilled`                             | A node lease holds unhealthy GPUs that it is not billed for                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `DevicesHeld`          | `PodsHoldDevices`                       | A released `binding: 'kubernetes'` reservation whose allocation is still releasing: pods on the named nodes still hold its GPUs, which stay booked until the node reports them free (0127ed7), billed never past the allocation's own end; a release stuck past that end is carried in 5-minute steps, unbilled, for at most 24 hours (e7a56d7). `since` is when the release began. Computed by `reservations.get` and `list`, not stored; it disappears once `AllocationReleased` or the hosts pass ends the release. See [release](/docs/guides/capacity/kubernetes#release) |

Lifecycle changes emit events with subject `reservation/<id>` and `state` and `poolId` in their data:
`reservation.placed` (with `nodeIds`, `start`, `end`), `reservation.pending`, `reservation.queued`,
`reservation.binding`, `reservation.active`, `reservation.extended`, `reservation.released` (with
`reason`: `release`, `expired`, `lease-lapsed` or `preempted`), `reservation.cancelled`,
`reservation.failed`, `reservation.awaiting-confirmation`, `reservation.expired`,
`reservation.repaired`, `reservation.preempting`, `reservation.preempted`, `reservation.reassigned` (an
offboarded owner's reservation handed to the successor, with `reservationId`, `from`, `to` and `reason`; it
has its own row in the event catalog and `webhooks.eventTypes`; see
[IAM events](/docs/guides/iam#iam-events-compute-reacts-to)) and
`reservation.binding-rejected` (with `allocationId`, `nodeId` and `reason`; see
[attested bindings](#attested-bindings)). Subscribe with
[webhooks](/docs/guides/events/webhooks) or [watch](/docs/guides/events/watch).

A node that becomes unreachable only raises conditions on the node; the reservation keeps its
booking (principle P6).

## Bindings [#bindings]

The binding says how the devices reach a workload.

**`hold`** (v0.1, the default) hands the devices to software already running on the node, such as
Slurm, Kubernetes or containerd with CDI. At the start the reservation moves to `binding` and the
node's desired state gains a `devices` assignment per allocation (`allocationId`, `deviceIds`, and
`until`, the booking's end). The node writes a CDI spec and an env file and
reports `AllocationBound` with the CDI device names; once every allocation is bound the reservation is
`active`, and `reservations.get` returns `binding`. Failure handling:

* no report within 5 minutes, or an `AllocationFailed` report: the bookings are released and the
  reservation is placed again, at most twice; the third failure fails it;
* an `AllocationReleased` report for a bound allocation: `Degraded` with `BindingLost` until the node
  reports it bound again.

The gateway accepts a report only about an allocation placed on the reporting node's own session; a report naming
another node's allocation is ignored like an unknown one. A report that arrives while the reservation is still
`placed` is judged as soon as the start is reached, not after the bind timeout.

**`kubernetes`** (v1.1, a21c1c1) is a hold whose devices the node hands to its kubelet: the assignment carries
`BINDING_KUBERNETES`, bcd binds it like a hold and marks its CDI spec for its device plugin, which advertises exactly
those devices as `bcd.dev/gpu`, `bcd.dev/gpu.mig` or `bcd.dev/gpu.slot` until the reservation ends. It needs a pool
with `policy.kubernetes` (`INVALID_INPUT` otherwise), and only the pool's owner may book it (3825ec7): the kubelet
gives bound GPUs to any pod of the cluster, so a tenant the pool is merely offered to gets `INVALID_INPUT` and books
holds instead. It places only on nodes whose bcd advertises the `binding.kubernetes` capability (system label
`bc.dev/cap.binding.kubernetes`, 409f783): an older bcd drops `BINDING_KUBERNETES`, and its GPUs would never reach the
kubelet. See [Kubernetes workers](/docs/guides/capacity/kubernetes#older-daemons).

### Host resources [#host-resources]

A reservation can book a host's CPUs, memory and 1 GiB huge pages with its GPUs (PLAN.md §3.8, a21c1c1):

* **What is bookable.** The `capacity.hosts` reconciler (every 30 seconds, `hostSyncEvery`) keeps each host's
  `vcpu`, `memoryGiB`, `hugepages2Mi` and `hugepages1Gi` rows in step with its applied inventory (8a7f8eb), and since
  b7e8bf5 inventory wakes it (and vps's `vps.hosts`) for a host as soon as it applies that host's snapshot, so a new
  or re-inventoried host is bookable at once rather than at the next pass; see
  [pools](/docs/guides/capacity/pools#pool-policy). A host without rows books none.
* **`resources`.** `{ vcpu, memoryGiB, hugepages1Gi }` per node (per gang member for a gang), booked on the devices'
  node in the same guarded statement as the devices, so a node that cannot give them is not a fit.
* **Without `resources`**, a request books the pool's per-GPU share of CPU and memory, times the GPUs' worth it asks
  for: `policy.perGpu.vcpu` and `policy.perGpu.memoryGiB` where set, otherwise the smallest share any GPU host of the
  pool can give (`allocatable × ratio ÷ physical GPUs`, counting every physical GPU whether bookable or not), so the
  share fits on every host; the product is rounded down. Since 004579e fractional GPUs count too (`gpuEquivalent`): a
  MIG instance is 1/7 of a GPU, and a time-slice or MPS slot 1/`policy.sharing.replicas` (default 4, or 10 for `mps`),
  so MIG and slot work also carries, and is limited to, its share of the host. It is resolved once, when the draft is
  written, and stored in the request. Pools whose hosts have no rows book none, so they behave as before.
* **The share is also the floor** (41826a2). `resources` below the pool's per-GPU share, or `resources: {}`, are
  raised to it, so booked GPUs never run a workload without CPU and memory limits while the same CPU and memory are
  sold again.
* **Quota.** The share rides on the GPUs and needs no quota beyond theirs; the reservation records it as `hostShare`
  when it is created, and since 004579e always, as `{}` when the share is empty (a pool without one), so anything
  booked above an empty share is charged too. Anything booked above it is charged to the tenant's `vcpu` and `memoryGiB` quota (1 GiB huge
  pages count as memory), and a missing quota row grants 0, so the excess is `QUOTA_EXCEEDED` unless the offer names
  these resources (41826a2). One GPU's quota can then not take a whole host's CPUs and memory. Reservations made before
  `hostShare` existed charge their devices only. IAM sees `vcpu` and `memoryGiB` (summed over a gang) as attributes of
  the reservation.
* **Preview.** `capacity.preview` books the same host resources as `reservations.create` would, the request's or the
  pool's per-GPU share, so a preview that fits is a create that fits (17f600b); a `node` binding books none.
* **Only on a bcd that enforces them** (409f783). An older bcd drops `DeviceAssignment.limits` as an unknown field and
  would run the workload unbounded, so a reservation that books any host resource (a non-zero `vcpu`, `memoryGiB` or
  `hugepages1Gi`, including the per-GPU share) places only on nodes with the system label
  `bc.dev/cap.devices.limits`, which the gateway writes from the `devices.limits` capability a GPU host's bcd
  advertises in `Hello` (see [capabilities](/docs/daemon/sessions#hello)). An operator label of the same name does not
  count: capacity reads `bc.dev/` labels only when their source is `system`. The requirement applies to create, to
  re-placing a pending reservation, to repairing a booking and to `capacity.preview` (a node without the label is
  rejected with filter `labels`, `needs label bc.dev/cap.devices.limits=*`); the placer also re-reads the labels under
  lock when it commits, so a daemon downgraded between plan and commit is caught. When no advertising node fits, the
  reservation waits `pending` like any other. A plain GPU hold that books no host resources (a pool whose hosts have no
  resource rows) needs no capability, so it still places on older daemons within the N-2 skew window, as does a
  `node` binding.
* **From the CLI and the console.** `better-compute reserve` and `better-compute capacity --preview` take `--cpus`
  (0 to 4096) and `--memory` (GiB, 0 to 65,536), sent as `resources`, and `--kubernetes` for `binding: 'kubernetes'`
  (ea36a94). `@better-compute/console-kit`'s reservation form takes the same three as `vcpu`, `memoryGiB` and
  `kubernetes`, and previews with them (e833433). The console app's reservation page shows them since 4533265:
  optional **Host CPUs** (0 to 4096) and **Host memory, GiB** (0 to 65,536), left empty for the pool's per-GPU share,
  and the Kubernetes checkbox only for a pool the tenant owns that has `policy.kubernetes`.
* **On the node** (c5a419c, 260e33c), the booked resources become the assignment's limits: `cpu_millis` from
  `vcpu`, `memory_bytes` from `memoryGiB`, `hugepages_1gi`, and a cpuset on the NUMA nodes of the hold's devices when
  those nodes have enough CPUs (`cpus`) or memory (`mems`). Since b6b804e a cpuset never includes the host's reserved
  CPUs, the lowest non-isolated, `policy.hostReserve.vcpu` (default 2), and the reserve counts against the fit. The node's cpulists (its NUMA nodes'
  CPUs and isolated CPUs) come from its inventory, so capacity reads them within bounds (004579e): CPU ids up to 8191,
  at most 32 KiB and 4096 comma-separated parts, a longer list read as empty, so an oversized list cannot stall the
  control plane. bcd
  admits the hold against the host's memory beside its VMs and other holds, and one that does not fit fails with
  `resources-exhausted`, which re-places it (31555bf). It then creates the cgroup slice
  `bcd-alloc-<allocation>.slice` with those limits, names it in the CDI spec (`bcd.dev/cgroup-parent`) and the env
  (`BCD_CGROUP_PARENT`), and stops it once it is empty after the release, retrying every pass (e01d7ff).
* **Where the workload runs.** The workload manager on the node (containerd, Slurm, a wrapper) places the workload in
  that slice; see [device assignments](/docs/daemon/delivery#device-assignments). Once the reservation is active,
  [`reservations.get`](/docs/reference/api/reservations#get) shows the slice as `binding.cgroupParent`
  (`HoldBinding.cgroupParent`, 6aa412b), which bcd reports in `AllocationBound.cgroup_parent` (c81f230, 19dbf60); the
  control plane keeps it only when it is exactly `bcd-alloc-<allocation>.slice` for that same allocation (004579e), so
  a node cannot point a reservation at another allocation's slice. It is empty for a hold without limits and for a
  `kubernetes` binding: that binding gets no slice, since its pods live in the kubelet's cgroups, where the kubelet
  enforces their requests and limits; its booked limits still count in bcd's host admission (see
  [Kubernetes workers](/docs/guides/capacity/kubernetes#host-resources-and-pods)).

### Attested bindings [#attested-bindings]

A gateway relays `AllocationBound`, so on its own a compromised gateway could claim a binding the node never made.
Attested bindings (v0.2, principle P14) close that gap: the node signs the report with its own key, the gateway stores
the signed bytes as received (`allocations.bound_report`, `bound_sig_kid`, `bound_sig`, `core/0017`), and the
reservation worker verifies them before the binding counts.

With `pki.commandSigning` configured, `betterCompute()` turns this on by default: capacity's `attestedBindings` is
`pkiAllocationBindingCheck()` from `@better-compute/server`. You can also pass your own
`capacity: { attestedBindings }` check. The pki check accepts a report only when:

* it is signed by a certificate of that node (current, or recently expired within the offline grace, never
  revoked);
* the certificate's role runs workloads (`node` or `node-tenant` by default; `roles` changes it);
* the signed body reports exactly this allocation bound.

A good report replaces the binding with the signed CDI devices and env file, marked `attested`, whatever the gateway
said. With the option on, only attested allocations count as bound. A report that fails the check never counts, and
each rejection publishes `reservation.binding-rejected` with `allocationId`, `nodeId` and `reason` (`unsigned
report`, a signature failure, `the signed report is not this allocation bound`, …):

| State when rejected | What happens                                                                                                |
| ------------------- | ----------------------------------------------------------------------------------------------------------- |
| `binding`           | `bound_at` is cleared, so nothing is billed; the reservation waits for a genuine report or its bind timeout |
| `active`            | The binding is marked lost: `Degraded` with `BindingLost`, and billing keeps the real binding               |

Node leases (`binding: 'node'`) are exempt: metal's worker binds them after its deploy workflow verified the machine.
The end-to-end test `tests/e2e/attested-bindings.test.ts` runs this through a real gateway on the dev stack: an
unsigned report, one signed by a foreign key, and the node's genuine signature over another allocation never
activate the reservation, and a forged re-report of an active binding marks it lost. Both daemons sign these
reports with their node key: the FakeDaemon, and the Go daemon `bcd`, whose devices converger sends every bind and
release as a node-signed `AttestedReport` (purpose `attested-report`).

Both implement the node side the same way (CDI spec `/etc/cdi/bcd-<alloc>.json`, env file
`/run/bcd/allocations/<alloc>.env`, device names `bcd.dev/gpu=<id>`); the CDI spec is the hold itself, and neither
drops a hold on its own. The FakeDaemon drives the simulator and `better-compute dev --sim N`; `bcd` is built and
tested end to end against the gateway but not released. See
[desired state and commands](/docs/guides/concepts/desired-state-and-commands) and [the daemon](/docs/daemon).

**`node`** (v0.5) leases whole machines in a pool whose policy sets `tenantRoot: true`. It needs
`node: { profileId, profileVersion? }` and cannot be spot. It books every bookable atom of `ready`,
platform-trusted nodes; metal deploys the profile from the start and marks the allocation bound, with
2 hours to do so and no second placement. See [dedicated root](/docs/guides/bare-metal/dedicated-root).

## Start, lease, extend and release [#start-lease-extend-and-release]

**Start.** Without `start` the window begins at placement. A `start` in the future books the window
now; see [calendar reservations](/docs/guides/capacity/gang-and-calendar). A pending reservation keeps
its requested start while it is still ahead; once it has passed, the window begins when a retry
places it, with the full duration.

**Renewable leases.** With `leaseSeconds`, the lease runs out `leaseSeconds` after placement.
[`reservations.renew`](/docs/reference/api/reservations#renew) (`compute:reservations:extend`) moves it
to now plus `leaseSeconds` without touching `during`, in states `placed`, `binding` and `active`. A
lapsed lease releases the reservation with reason `lease-lapsed`. Renewing a reservation created
without `leaseSeconds` is `INVALID_STATE`. Leases suit agents that may die without cleaning up.

**Extend.** [`reservations.extend`](/docs/reference/api/reservations#extend) takes exactly one of
`until` (epoch milliseconds) or `by` (a duration), plus an `idempotencyKey`. It is admitted like a
create: transaction 1 writes the pending change into the authz row, with `durationHours` set to the
total after the change, so a cap such as `BoundedExtend` also bounds extensions. On allow, every
allocation's end moves later, all or nothing, and the `EXCLUDE` constraint decides:

```ts
const result = await compute.api.reservations.extend(credential, {
  tenantId: 'acme',
  reservationId,
  by: '4h',
  idempotencyKey: 'train-2026-09-24-ext-1',
});
// { reservation, extended: true }
// or { reservation, extended: false, availableUntil: 1790040000000 } when another booking starts first
```

An `until` that the reservation already reaches answers `extended: false` without a decision, so
prefer `until` when you may retry: a repeated `by` extends again from the new end. A refusal clears
the pending change; so does a handler that throws after an allow (the reservation changed between the
transactions), through the admission's `abandoned` hook in a transaction of its own, so a refused
lifetime is never merged into later decisions on the reservation (a2288ec). A retry with the same key
decides again.

One extend at a time stages the pending change (f36248d, review 7), since the authorization projection keeps one
pending value per resource: two concurrent extends could otherwise interleave, one decided on the other's shorter
duration and then applying its own longer one past `BoundedExtend`, and one's refusal could clear the other's
change. `capacity_extend_holds` (migration `capacity/0005_extend_holds`: the reservation, the hold and when it was
taken; deleted with the reservation) gives the slot to one request. The hold is a nonce the server mints for each
extend (`newId('hold')`, kept on the parsed input, which callers cannot set), not the caller's `x-request-id`, which
two concurrent extends could share (d6f7ddc); its age is measured with `compute_now()`, since replicas' clocks may
disagree. While another extend holds the slot, an extend is `CONFLICT` with `details.reason: 'extend-in-flight'`
and `retryAfterSeconds: 60`; a hold older than 60 seconds is taken over. Transaction 2 applies the change only while
its hold still owns the slot, and a refusal, an abandon or a budget refusal clears the pending change only when the
hold is its own. The completing transaction
takes the pool lock first (bb6da07), so an extend of a gang and a batch commit cannot deadlock on each other's
bookings. Sandboxes do the same with their own hold (see [sandboxes](/docs/guides/vps/sandboxes)).

**Release and cancel.** [`reservations.release`](/docs/reference/api/reservations#release) ends a
reservation at `compute_now()`: the current booking is truncated, future ones are deleted, quota is
refunded, and billing stops. Releasing a pending reservation cancels it, and releasing a terminal one
changes nothing. [`reservations.cancel`](/docs/reference/api/reservations#cancel) withdraws one that has
not started (`pending` or `placed`); anything else is `INVALID_STATE`. Both are
`compute:reservations:release`. Freed capacity wakes the pool's pending reservations at once instead of
at their next 30-second retry.

```bash
better-compute extend rsv_… --tenant acme --until +12h
better-compute release rsv_… --tenant acme
```

## Delegated confirmation [#delegated-confirmation]

When an agent acting for a person in a delegated session is refused, and the person's delegation holds
the action for confirmation, Better IAM opens a confirmation request and the call fails with
`CONFIRMATION_REQUIRED` (`details.confirmationId`, and `expiresAt` when IAM gives one). The draft moves
to `awaiting-confirmation` with a `ConfirmationRequired` condition and no quota charged. The deadline is
IAM's, clamped to at most 30 minutes.

Once the person approves, the agent retries **with the same idempotency key**. The retry reaches the
same draft, IAM decides it again, finds the approval, and the reservation is placed. After the
deadline, the reservation becomes `expired`, either by the reconciler or on a late retry. See
[delegated confirmation](/docs/guides/iam/delegated-confirmation) and the [MCP server](/docs/guides/clients/mcp),
which runs this loop for agents.

## The reservations API [#the-reservations-api]

| Method                                                             | Action                         | Decided on  | Phase |
| ------------------------------------------------------------------ | ------------------------------ | ----------- | ----- |
| [`reservations.create`](/docs/reference/api/reservations#create)   | `compute:reservations:create`  | the draft   | v0.1  |
| [`reservations.get`](/docs/reference/api/reservations#get)         | `compute:reservations:read`    | reservation | v0.1  |
| [`reservations.list`](/docs/reference/api/reservations#list)       | `compute:reservations:read`    | tenant      | v0.1  |
| [`reservations.release`](/docs/reference/api/reservations#release) | `compute:reservations:release` | reservation | v0.1  |
| [`reservations.cancel`](/docs/reference/api/reservations#cancel)   | `compute:reservations:release` | reservation | v0.2  |
| [`reservations.extend`](/docs/reference/api/reservations#extend)   | `compute:reservations:extend`  | reservation | v0.1  |
| [`reservations.renew`](/docs/reference/api/reservations#renew)     | `compute:reservations:extend`  | reservation | v0.1  |

All seven are served today. `reservations.list` returns the caller's own reservations by default
(`scope: 'mine'`); `scope: 'all'` lists the tenant's. It filters by `state` and `poolId`, returns newest
first, and pages with `limit` (1 to 200, default 50) and `cursor`, the last reservation's id. Errors are
listed in the [error reference](/docs/reference/errors).


# Spot and preemption (/docs/guides/capacity/spot-and-preemption)

> Spot reservations fill idle GPUs at the spot meter, and a regular request that would otherwise wait takes their atoms after a notice.



GPUs that nobody has reserved still cost money. Spot reservations let tenants
use them at a lower price, on one condition: when a regular request needs those GPUs and nothing
else fits, the spot reservation gets a short notice and gives them up. Spot shipped ahead of its
roadmap phase, v0.5.

## Reserve spot capacity [#reserve-spot-capacity]

Set `spot: true` on a reservation:

```ts
await compute.api.reservations.create(credential, {
  tenantId: 'acme',
  poolId,
  devices: [{ class: 'gpu', count: 4 }],
  duration: '12h',
  spot: true,
  idempotencyKey: 'sweep-042',
});
```

* A spot reservation is admitted and placed like any other, with the same quota charge.
* The draft carries the IAM attribute `spot: true`, so a tenant policy can allow spot work to people
  or agents it would not give regular capacity to.
* It bills `gpu.<model>.spot.hours` instead of `gpu.<model>.hours`.
* `binding: 'node'` leases cannot be spot (`INVALID_INPUT`).

`better-compute iam setup --meters` creates `gpu.<model>.spot.hours` for each model it covers. Run it (or let the
IAM module define the catalog when it starts) before you sell spot capacity; otherwise its usage
windows are dead-lettered (see [quotas and usage](/docs/guides/capacity/quotas-and-usage)).

## When preemption happens [#when-preemption-happens]

Preemption is tried only for a **regular** request that would otherwise go `pending`: at admission, in
a pool-lease batch, or when a pending reservation retries. Spot requests never preempt, not even other
spot work; regular reservations are never preempted; node leases never preempt.

1. The deadline is the later of the request's start and now plus the notice (`spotNoticeMs`, 2 minutes
   by default).
2. The placer plans the request for the window starting at the deadline **as if** the bookings of every
   live spot reservation (placed, binding or active) overlapping that window were free.
3. Among the plans that fit (at most 64 are compared), it takes the one that preempts the fewest spot
   reservations, then the best score. A gang considers whole candidate gangs.
4. Inside one savepoint, it locks the victims with `FOR UPDATE SKIP LOCKED`, gives each a notice, and
   places the request on the planned nodes from the deadline.

If a victim is locked by another transaction or no longer qualifies, or the placement still fails,
the savepoint rolls back: no spot reservation is touched and the request goes `pending` as usual. The
preempting reservation answers `placed`, with `during.start` at the deadline, so it starts about two
minutes after it was requested.

<Mermaid
  chart="sequenceDiagram
  participant R as Regular request
  participant P as Placer
  participant S as Spot reservation
  R->>P: reservations.create (nothing free)
  P->>P: plan as if spot bookings were free
  P->>S: notice: end at now + 2 min, Preempted
  P->>R: placed, from now + 2 min
  Note over S: runs until the deadline
  S->>S: released, reason preempted"
/>

## What happens to the spot reservation [#what-happens-to-the-spot-reservation]

Preemption is never a silent revoke. What the victim sees depends on whether it has started by the
deadline:

**It has started, or starts before the deadline.** Its bookings and allocations are cut back to end
at the deadline, and its `during.end` becomes the deadline. It gets a `Preempted` condition (reason
`SpotPreempted`), its nodes' desired state is refreshed with the new end, and
`reservation.preempting` is emitted with `at`, the deadline. The reservation keeps running until then;
at the deadline the lifecycle releases it with reason `preempted`, refunds its quota, and emits
`reservation.released`.

**It starts at or after the deadline.** It has not run yet, so it is not cut short. Its bookings are
released and it goes back to `pending` without a window, with `Unschedulable` and reason
`SpotPreempted`, keeping its quota charge. `reservation.preempted` is emitted with `requeued: true`.
It then retries every 30 seconds like any pending reservation.

Your workload should treat `reservation.preempting` as its signal to checkpoint. Subscribe to it with
[webhooks](/docs/guides/events/webhooks) or [watch](/docs/guides/events/watch), or poll
[`reservations.get`](/docs/reference/api/reservations#get) and compare `during.end` with the end you
booked.

### Notices never name another tenant [#notices-never-name-another-tenant]

When the preempting reservation belongs to the victim's own tenant, the condition message reads
`preempted by rsv_…` and the events carry `by` with its id. When it belongs to another tenant, the
message is `preempted by a non-spot reservation` and `by` is absent, in the condition and in every
event. Nothing a tenant can read reveals another tenant's reservation.

## Preview a preemption [#preview-a-preemption]

[`capacity.preview`](/docs/reference/api/capacity#preview) of a regular request that does not fit now
reports whether it would fit by preempting spot work:

```json
{
  "fits": false,
  "candidates": [],
  "reason": "no node fits (2 nodes): 2 × fits: 0 of 4 gpu free",
  "preemptible": { "reservations": 1, "startsAt": 1790000120000 }
}
```

`preemptible` gives a count and the start after the notice, never reservation ids. A spot request's
preview never has it.

## Billing [#billing]

Each side pays for what it held. The victim bills the spot meter until the deadline, and the
re-queued kind bills nothing for the booking it lost, because it never became active. The preempting
reservation bills its regular meter from the moment it is active, so a preempt followed by a re-place
bills each hour once.

## Configure the notice [#configure-the-notice]

```ts title="compute.ts"
betterCompute({
  // …
  capacity: { spotNoticeMs: 5 * 60_000 }, // five minutes instead of two
});
```

A longer notice gives workloads more time to checkpoint and delays the preempting reservation by the
same amount. The same value drives `capacity.preview`'s `startsAt`.


# Admin panel (/docs/guides/clients/admin-panel)

> The operator panel in apps/admin, a keyboard-first workspace over capacity, the fleet and the control plane that runs on a simulated fleet or, through a loopback-only proxy, on a real compute API as the signed-in operator.



`apps/admin` is a web panel for the people who run a better-compute fleet: GPU capacity and bookings, nodes and
workloads, rollouts and destructive batches, and the control plane, all in one workspace with dashboards, a command
palette and keyboard shortcuts. It is a Next.js 16 app (`@better-compute/admin`, private, not published), styled with
BoardUI components copied from `apps/docs` in the docs' "instrument" theme (d0e4e8f).

It differs from the [console](/docs/guides/clients/console) in where it lives and whom it serves. The console hosts
Better IAM and Better Compute in its own process and is the UI for tenants and operators alike. The admin panel
hosts neither: it is a separate server that talks to a compute API somewhere else, through its own server-side
proxy, and it is built for platform operators. By default it does not talk to any API at all; it runs on a
deterministic simulated fleet.

> **Development-grade, tested against stubs.** 
  The admin panel has unit tests over its pure logic and its server routes, with `fetch` stubbed (see
  [What is tested](#what-is-tested)). No test in the repository runs it against a real compute API or a real
  Better IAM, and its tests are not part of the root `tests/**` suite. Its auth state lives in one process's memory
  (see [Limits of the auth state](#limits-of-the-auth-state)). Run one instance, on loopback.

## Run it [#run-it]

The scripts in `apps/admin/package.json` bind to `127.0.0.1` on port 4200:

| Script      | Runs                                                                             |
| ----------- | -------------------------------------------------------------------------------- |
| `dev`       | `next dev --hostname 127.0.0.1 --port 4200`                                      |
| `build`     | `next build --webpack` (the README notes Turbopack's build trips on `next/font`) |
| `start`     | `next start --hostname 127.0.0.1 --port 4200`                                    |
| `typecheck` | `tsc --noEmit`                                                                   |
| `test`      | vitest from the repository root with `apps/admin/vitest.config.ts`               |

```bash
cd apps/admin
node node_modules/next/dist/bin/next dev --port 4200   # as the README shows; pnpm dev binds 127.0.0.1
pnpm --filter @better-compute/admin test
pnpm --filter @better-compute/admin build
```

`next.config.mjs` reads `NEXT_DIST_DIR` for the output directory (default `.next`) so that concurrent dev servers
and builds do not share one; `.claude/launch.json` starts the panel with `NEXT_DIST_DIR=.next-admin` on port 4200.
Open `http://127.0.0.1:4200/`.

With no further configuration the panel shows the simulation: 5 sites and 9 pools totalling 440 nodes
(`lib/data/topology.ts`), generated as a pure function of time. Because the data is a function of the clock, the
clock in the top bar can be paused and scrubbed back 7 days ("time travel"), and every view replays that moment.

## Connect a compute API [#connect-a-compute-api]

Live mode is configured **on the admin server**, never in the browser, so the proxy cannot be pointed elsewhere
from a page (2dacaf8). `upstreamConfig()` reads the environment first:

| Variable               | Meaning                                                                                                                                                                                                                               |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ADMIN_COMPUTE_URL`    | The compute API base including its base path, e.g. `http://127.0.0.1:3300/api/compute`. Must be `http` or `https`; a URL with a username, password, query or fragment is refused (live mode stays off) and a trailing `/` is trimmed. |
| `ADMIN_IAM_URL`        | Better IAM's API base for operator sign-in. Default: the compute URL's origin plus `/api/iam`.                                                                                                                                        |
| `ADMIN_COMPUTE_TENANT` | The tenant views default to and the tenant operators sign in to when they leave the organization field empty (normally the platform's root tenant).                                                                                   |
| `ADMIN_SESSION_SECRET` | At least 32 characters; seals operator sessions. Unset in development (`NODE_ENV` not `production`), a random key is written to `apps/admin/.admin-secret` (mode `0600`, gitignored). Unset or short in production: no sign-in.       |
| `ADMIN_COMPUTE_TOKEN`  | Optional. A bearer for the shared connection signed-out viewers read through. Give it a read-only role.                                                                                                                               |
| `ADMIN_ALLOWED_HOSTS`  | Comma-separated `host` or `host:port` names to accept besides loopback, for a deployment behind a reverse proxy.                                                                                                                      |

When `ADMIN_COMPUTE_URL` is unset, the panel reads `apps/admin/.upstream.json` from its working directory instead
(gitignored), with the keys `url`, `iamUrl`, `tenantId` and `token`:

```json title="apps/admin/.upstream.json"
{ "url": "http://127.0.0.1:3300/api/compute", "tenantId": "ten_…" }
```

Then pick **Live API** under Settings → Connection. The choice is stored per browser (`bc-admin:workspace` in
localStorage) and takes effect only while the API answers; until then, and whenever the status probe fails, the
workspace source stays `simulated`. Time travel always replays the simulation: live data is always "now".

## The security model [#the-security-model]

Three routes on the admin server stand between the browser and the API. Each starts with the same caller check,
`callerAllowed` (ab71e52):

* the `Host` header must be a loopback name (`localhost`, `127.0.0.1`, `[::1]` or `*.localhost`) or one listed in
  `ADMIN_ALLOWED_HOSTS`, which stops other machines on the network and DNS rebinding;
* an `Origin` header, when present, must name that same host.

The POST routes also require `x-better-compute: 1`, which a cross-site form cannot send, the same CSRF boundary
the compute API uses ([the HTTP API](/docs/guides/clients#the-http-api)).

| Route                                 | Does                                                                                                                                            |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /api/upstream/<group>/<method>` | The proxy. Forwards one method to `ADMIN_COMPUTE_URL` with the JSON body.                                                                       |
| `GET /api/upstream/status`            | Whether an API is configured, whether `system/version` answers (4 s timeout) and its latency, and who the admin acts as. Never returns a token. |
| `GET /api/upstream/health`            | The API's public `GET {base}/health`, with the round trip measured from the admin server.                                                       |
| `POST`, `DELETE /api/session`         | Operator sign-in and sign-out against Better IAM.                                                                                               |

### The proxy [#the-proxy]

`POST /api/upstream/<path>` refuses, in order:

| Answer                     | When                                                                            |
| -------------------------- | ------------------------------------------------------------------------------- |
| `403 ACCESS_DENIED`        | the caller check fails or `x-better-compute: 1` is missing                      |
| `503 NOT_CONFIGURED`       | no compute API is configured                                                    |
| `400 INVALID_INPUT`        | the path is not a method path (`group/method`, two to four lowerCamel segments) |
| `403 READ_ONLY`            | no operator is signed in and the method is not a read (below)                   |
| `413 INVALID_INPUT`        | the request body is over 1 MiB                                                  |
| `400 INVALID_INPUT`        | the body is not JSON                                                            |
| `502 UPSTREAM_UNREACHABLE` | the API did not answer; the message has any URL replaced with `<url>`           |

Otherwise it POSTs to `{ADMIN_COMPUTE_URL}/{path}` with `x-better-compute: 1`, no redirects, a 15 s timeout, and
the bearer described below. It passes back the API's status, JSON envelope and any `Retry-After` header. An answer
over 8 MiB becomes a `502 INVALID_RESPONSE`; an answer without JSON keeps its status with an `INVALID_RESPONSE`
error envelope; a 204, 205 or 304 status becomes a 502.

### Read methods and writes [#read-methods-and-writes]

Signed out, the proxy forwards only **read methods**, decided by name in `lib/methods.ts`: the method's last
segment must be one of `get`, `list`, `summary`, `version`, `status`, `timeline`, `preview`, `history`, `diff`,
`show`, `describe`, `catalog`, `search`, `find` or `estimate`, optionally followed by more words (`listSshCerts`,
`getPolicy`), and none of those words may be a write word such as `Or`, `And`, `Create`, `Delete`, `Set`, `Lock`,
`Revoke` or `Reserve`. So `capacity/preview` and `quotas/get` are reads, and `getOrCreate`, `listAndPrune` and
`findAndLock` are not.

This is a name check, not authorization. The README and the code both say so: give `ADMIN_COMPUTE_TOKEN` a read-only
role, because the compute API's own decision is the real boundary.

### Who the proxy acts as [#who-the-proxy-acts-as]

| State                                 | Bearer sent upstream                        | Methods forwarded                                                                         |
| ------------------------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Signed out, no `ADMIN_COMPUTE_TOKEN`  | none                                        | reads only; public methods such as `system/version` answer, others are refused by the API |
| Signed out, `ADMIN_COMPUTE_TOKEN` set | the shared token                            | reads only                                                                                |
| Operator signed in                    | the operator's own Better IAM session token | every method                                                                              |

A signed-in operator's calls are authorized and audited by the compute API as that person (4392bc3); the shared
token is never used for a change. The status route reports this as `readOnly`, `writable` and `operator` (identity
id, name, email, expiry), and Settings → Connection shows it as "Acting as": the operator with **read & write**, or
"shared connection" or "anonymous (public methods)" with **read-only** and a sign-in link.

### Operator sign-in [#operator-sign-in]

`/signin` asks for an optional organization, an email and a password. `POST /api/session`:

1. Requires the caller check and `x-better-compute: 1`; beyond loopback, also https (checked on the request URL or
   `x-forwarded-proto: https`), so passwords and cookies never cross the network in the clear.
2. Requires the browser's session key (below), a configured API and a usable session secret.
3. Resolves the tenant: empty means `ADMIN_COMPUTE_TENANT`; a `ten_…` id is used as is; a slug is looked up with
   Better IAM's `tenants/lookup`. An unknown organization answers like wrong credentials.
4. Calls Better IAM `auth/signIn`. Wrong email, wrong password and unknown account all answer the one message "Those
   credentials did not work." Failures count per organization and email: after 10 in 15 minutes the account is
   refused with `429` until the window ends.
5. If Better IAM asks for a second factor, the admin seals the challenge (tenant, challenge, the browser key's hash,
   a 5-minute expiry) and returns it; when an email code is available it calls `auth/requestMfaCode`. The code step
   calls `auth/verifyMfa`. A challenge takes at most 5 wrong codes and signs in once. An account that must first
   enroll a second factor is told to do that in the console.
6. With a session token, it calls `auth/getSession` and seals the operator session: the token, identity id, tenant,
   name, email and an expiry of the session's lifetime capped at 12 hours (and at its idle expiry).

The sealed session goes into an HttpOnly, `SameSite=Strict`, `Path=/` cookie named `bc_admin_session`, or
`__Host-bc_admin_session` with `Secure` on https. Sealing is AES-256-GCM under a SHA-256 of `ADMIN_SESSION_SECRET`,
with the purpose (`session` or `mfa`) bound in as associated data, so a session cannot pass as a challenge or the
reverse. The browser never sees the bearer.

Cookies are not isolated by port, so on `localhost` every local server the browser visits receives this cookie. The
cookie alone is therefore not a session (4fc4afc): at each sign-in the page makes a random key, keeps it in its own
origin's localStorage (`bc-admin:session-key`), and sends it as `x-bc-session-key` on every call to `/api/session`
and `/api/upstream`. The sealed payload holds the key's SHA-256, and a request opens a session only when one of its
cookies (any of them, under either name) opens and matches that hash. A copied cookie replayed by another local
server opens nothing.

`DELETE /api/session` signs out every session cookie in the request that opens, with or without the key. Each is
revoked in the admin until it expires, ended at Better IAM with `auth/signOut`, and the cookies are cleared. If
Better IAM could not be reached, the answer says the session there ends when it expires.

After sign-in, `/signin?next=…` returns only to a path on the admin itself: `safeNext` refuses anything that does not
start with a single `/`, contains control characters, spaces or backslashes, or resolves to another origin, and
falls back to `/`.

### Limits of the auth state [#limits-of-the-auth-state]

The sign-in throttle, the record of used MFA challenges and the sign-out denylist live in the admin process's
memory (cb8f618). A restart forgets them, and several instances behind one name do not share them. Run one instance,
and treat Better IAM's revocation as the real sign-out. The proxy route's own comment adds that the caller check is
not operator authentication: put real authentication in front before exposing the admin beyond loopback.

## How the browser calls the API [#how-the-browser-calls-the-api]

`lib/api.ts` exposes the proxy as `api.$request(path, input)`, the shape of console-kit's `ComputeRequester`, so
console-kit's view helpers can run on it. Errors come back as `ApiError` with the API's `code`, `message` and
`details`. `useApi(path, input, { enabled, refreshMs })` is a React hook over it.

Two pieces of connection state are shared by every page:

* `ConnectionProvider` probes `/api/upstream/status` on load, every 30 s and on demand; the status bar, Settings and
  the live views read `status.configured`, `status.ok`, `status.tenantId` and `status.writable` from it.
* The change journal ("Your changes", 40fae79): every call that is not a read is recorded with its outcome in
  localStorage (`bc-admin:journal`, the last 100), reachable from the operator menu and ⌘K. It keeps only
  identifying input fields (ids, names, `reason`, `tenant`, `pool`, `site`; at most 6, each cut at 120 characters)
  and drops any field whose name looks like a secret. A `READ_ONLY` refusal is not recorded. The compute API's
  [audit log](/docs/guides/events/audit) stays the record.

Settings → Connection also has **Try a method**: a method path and JSON input sent through the same proxy, with the
answer or error and its time. It is bound by the same read-only rule.

## Modules [#modules]

Everything the shell shows comes from `lib/modules.ts`. A module is one descriptor (`AdminModule`) that contributes
navigation, dashboard widgets, palette commands, search results, peeks (a compact side sheet for one entity),
status-bar readouts, attention items and watch describers. Three modules are installed: Overview, Capacity and
Fleet. An operator can hide Capacity or Fleet per browser in Settings → Installed modules (3e53bf8); its pages leave
the sidebar, palette, search and widget catalog and its alerts leave the queue, but its pages still resolve and its
peeks still open from other modules. The Overview cannot be hidden.

Each page below says what it reads in live mode. "Simulated only" means the page always shows the simulation, even
with Live API selected.

### Overview [#overview]

| Page            | Path             | Shows                                                                                                                                                                                                                            | Live mode                                                                                                                                                                                                                          |
| --------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Overview        | `/`              | Dashboards of widgets from every module (presets plus your own; drag, resize, keyboard arrange, share links, JSON export and import), a "Since you were away" digest, and a comparison digest while comparing with a past moment | widgets that declare `live` read the API; the rest say "sim"                                                                                                                                                                       |
| Needs attention | `/attention`     | The operator queue: platform items plus every module's attention items, most severe first, grouped by kind                                                                                                                       | each module's items come from its live data; nothing until the first answer                                                                                                                                                        |
| Control plane   | `/control-plane` | API, Better IAM bridge, database, workers, outbox and one gateway per site, with latency and throughput                                                                                                                          | simulated, except the **Live vitals** panel, which polls `/api/upstream/health` every 5 s (round trip, the database latency the API reports, share answered) and shows the server name and version from the status probe (6e76141) |
| Settings        | `/settings`      | Workspace preferences, dashboards export and import, installed modules, and the Connection panel                                                                                                                                 | n/a                                                                                                                                                                                                                                |

Overview widgets: Fleet KPIs, Utilization, Needs attention, Fleet wall, Control plane, Sites, Activity, Watching,
Notes and Power. Only Fleet KPIs, Watching and Notes declare live support.

### Capacity [#capacity]

The capacity module (c951695) covers pools, bookings over time, reservations, quotas, usage and flavors. In live mode
(324b8b4) it builds one snapshot for the connected tenant (`status.tenantId`) and refreshes it every 15 s while a
view reads it:

| Call                                                    | Used for                                                                               |
| ------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| [`pools/list`](/docs/reference/api/pools)               | the pools                                                                              |
| [`capacity/summary`](/docs/reference/api/capacity)      | GPU totals per pool and model                                                          |
| [`reservations/list`](/docs/reference/api/reservations) | `scope: 'all'`, 200 a page, at most 10 pages (2,000)                                   |
| [`quotas/get`](/docs/reference/api/quotas)              | quota rows                                                                             |
| [`usage/list`](/docs/reference/api/usage)               | the last 30 days, one day at a time newest first, 500 a page, capped at 10,000 windows |
| [`flavors/list`](/docs/reference/api/flavors)           | per pool of class `vps-host` or `provider`; a failure gives that pool no flavors       |
| `capacity/timeline`                                     | per pool on demand, cached 60 s, split into calls of at most 7 days and 336 buckets    |

When the reservation or usage caps are hit, or usage fails, the page says the data is partial. A failed snapshot shows
the API's reason with **Use the simulation**. In live mode tenants, identities, offers, prices and node placement are
not loaded: the snapshot has only the connected tenant.

| Page               | Path                                                | Shows and does                                                                                                                                                                          |
| ------------------ | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capacity           | `/capacity`                                         | KPIs, the booking heatmap per pool and hour (history and ahead; drill into a cell, drag to plan), GPU models, runway, insights                                                          |
| Pools              | `/pools`, `/pools/[poolId]`, `/pools/compare`       | each pool's capacity, schedule, tenants and policy; a week of bookings; pools side by side                                                                                              |
| Reservations       | `/reservations`                                     | every booking as a table or a Gantt timeline; a drawer with **Extend**, **Release** and **Cancel** (and bulk release or cancel); "Why it's waiting" for a pending reservation (6267a60) |
| Plan a reservation | `/reservations/new`                                 | the planner: a shape, pool and window, previewed, then created; other shapes of the same GPU count dry-run                                                                              |
| What-if            | `/capacity/scenarios`                               | maintenance, expansion or a burst evaluated against the current book; a maintenance plan can **Cordon these N now** through the fleet's cordon command                                  |
| Tenants            | `/capacity/tenants`, `/capacity/tenants/[tenantId]` | who holds what, ceilings, bookings ahead and spend                                                                                                                                      |
| Quotas             | `/quotas`                                           | tenant by pool ceilings and use, an editor, and right-sizing suggestions                                                                                                                |
| Usage & spend      | `/usage`                                            | metered usage priced at list, by tenant, project, pool or meter, with a CSV export                                                                                                      |
| Report             | `/capacity/report`                                  | a capacity report as a page and as Markdown to copy                                                                                                                                     |
| Flavors            | `/flavors`                                          | instance and sandbox shapes                                                                                                                                                             |

Capacity writes, with the API call each makes in live mode:

| Action               | Call                                                                                                                                               |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Preview a placement  | `capacity/preview` (a read: allowed signed out)                                                                                                    |
| Create a reservation | `reservations/create` with an `admin-…` idempotency key                                                                                            |
| Release, cancel      | `reservations/release`, `reservations/cancel`                                                                                                      |
| Extend               | `reservations/extend` with `by` in ms; an answer with `extended: false` is shown as "Can't extend by that much", with when the GPUs are free until |
| Set a quota          | `quotas/set`                                                                                                                                       |

Before acting, `useCapacityActions` refuses with a toast and no dialog while the clock is in the past, while live
mode has no tenant yet, or while signed out in live mode (f38fab8). Live, destructive actions ask for a reason "kept
in the audit log". A `READ_ONLY` answer means the session ended meanwhile: the panel re-probes the connection and
says "Your sign-in has ended", adding for bulk actions that anything done before it ended is kept (a5b5924). The
planner's search, `earliestFit`, `timelineView`, `usageCsv` and `quotaHint` come from
[console-kit](/docs/guides/clients/console#console-kit).

The palette also answers capacity questions: typing a shape such as `16 h200 3d` lists the pools where it fits soonest,
each opening the planner prefilled (0acbc3d).

### Fleet [#fleet]

The fleet module (e2030b8) covers nodes, GPUs, workloads, the metal lifecycle and operations. In live mode (0d3bfed)
one poller serves every fleet view for the connected tenant, every 15 s while something subscribes. Each group is
optional: a module the server does not mount answers an error and only that part stays empty, named in the page's
live notice.

| Call                                                             | Input                                        |
| ---------------------------------------------------------------- | -------------------------------------------- |
| [`nodes/list`](/docs/reference/api/nodes#list)                   | 50 a page, every page up to 40 (2,000 nodes) |
| [`instances/list`](/docs/reference/api/instances)                | first 200                                    |
| [`sandboxes/list`](/docs/reference/api/sandboxes)                | first 200                                    |
| [`sandboxTemplates/list`](/docs/reference/api/sandbox-templates) | first page                                   |
| [`sandboxVolumes/list`](/docs/reference/api/sandbox-volumes)     | first page                                   |
| [`rollouts/list`](/docs/reference/api/rollouts)                  | first page                                   |
| [`commandBatches/list`](/docs/reference/api/command-batches)     | first page                                   |
| [`fleet/status`](/docs/reference/api/fleet)                      | drift                                        |

Live nodes are mapped without inventing data (`lib/data/fleet-live.ts`): health comes from connectivity, provision
state and NoSchedule reasons (cordons, maintenance and taints do not count as faults), and the fields the list does
not carry (GPU telemetry, labels, rack placement, daemon version, CPU and memory) are empty and shown as a dash. In
live mode the GPU pages therefore have no GPUs, the rack view has no placement, the metal page has no PXE discoveries
or workflows, and the Activity stream has no API events (the panel does not call the event stream).

| Page            | Path                        | Shows                                                                                                                                                                |
| --------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nodes           | `/nodes`, `/nodes/[nodeId]` | every machine: health, GPUs, lifecycle; a table with saved views, rack elevations (`?view=racks`), and a node page with conditions, labels, taints and recent events |
| GPUs            | `/devices`                  | every GPU: health, readings, holders; the GPU wall                                                                                                                   |
| Bare metal      | `/metal`                    | enrollment to deployment, workflows, PXE discoveries                                                                                                                 |
| Sandboxes       | `/sandboxes`                | sandboxes, warm pools, TTLs, time to ready                                                                                                                           |
| Instances       | `/instances`                | VPS instances by state, flavor, tenant and host                                                                                                                      |
| Volumes         | `/volumes`                  | sandbox volumes and where they live                                                                                                                                  |
| Fleet drift     | `/drift`                    | the applied FleetConfig against the live control plane                                                                                                               |
| Rollouts        | `/rollouts`                 | bcd releases wave by wave                                                                                                                                            |
| Command batches | `/batches`                  | destructive batches waiting for approver signatures                                                                                                                  |
| Activity        | `/activity`                 | the event stream, filterable and pausable: simulated events plus this browser's own actions                                                                          |

Fleet actions all go through `useFleetCommands` (`components/fleet/commands.tsx`), so pages, the node drawer and
palette commands behave alike (5e65153). Live, each sends one call per node with the connected tenant, toasts how
many succeeded, and refreshes the fleet:

| Action                          | Call                                                                                    | Confirmation                                                                     |
| ------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Cordon | [`nodes/cordon`](/docs/reference/api/nodes#cordon) with `version` and optional `reason` | dialog with the call and a reason                                                |
| Uncordon                        | [`nodes/uncordon`](/docs/reference/api/nodes#uncordon)                                  | none                                                                             |
| Drain   | [`nodes/drain`](/docs/reference/api/nodes#drain)                                        | dialog with a reason; over 8 nodes the reason is required and you type `drain N` |
| Revoke identity                 | [`nodes/revokeIdentity`](/docs/reference/api/nodes#revokeidentity)                      | reason required; you type the hostname                                           |
| Label                           | [`nodes/label`](/docs/reference/api/nodes#label) with `set` and `remove`                | `key=value` or `key-`; `bc.dev/` keys are refused                                |
| Taint                           | [`nodes/taint`](/docs/reference/api/nodes#taint) with `add` and `remove`                | none                                                                             |
| Pause, resume, abort a rollout  | `rollouts/pause`, `rollouts/resume`, `rollouts/abort` (abort with a required reason)    | abort only                                                                       |
| Cancel a batch                  | `commandBatches/cancel`                                                                 | dialog                                                                           |
| Confirm a PXE MAC               | `metal/confirmMac`                                                                      | dialog                                                                           |
| Delete, extend sandboxes        | `sandboxes/delete`, `sandboxes/extend` (`by` in ms)                                     | delete only                                                                      |

Some actions do not reach the API in live mode. Maintenance on and off says to use `metal.manage` and
`metal.provide` instead. Approving a batch shows the command `better-compute batch approve <batchId>`, because an
approval is a signature by the approver's key, which a browser never holds (see the CLI's
[destructive batches](/docs/guides/clients/cli#destructive-batches)). Acknowledging a condition and ignoring a MAC
only change this browser's view.

Every action is refused while the clock is in the past, and, live and signed out, refused before any dialog with
"Sign in to make changes" (05ce45e). A `READ_ONLY` answer re-reads the connection and says the sign-in has ended
(782c869). In the simulation the same actions apply to the simulated fleet, most with **Undo** in the toast.

## Operator features [#operator-features]

* **Time travel**: scrub back 7 days, step with `,` and `.`, pause with `p`. "Compare with now" pins a moment and the
  Overview shows what changed (`?compare=`). Links carry `?site=&range=&at=`. Nothing can be changed while viewing
  the past.
* **Peek and watch**: any id opens a peek; **watch** follows it in the Watching widget with a toast when it changes
  (df8ac1c). Critical attention items toast as they open.
* **Wall display**: `?kiosk=1` (or ⌘K, Wall display) for a NOC screen, rotating dashboards with `&rotate=45`
  (c407294).
* **Keyboard**: ⌘K or `/` for the palette, `g` plus a key to jump (for example `g n` Nodes, `g c` Capacity), `e` to
  arrange a dashboard, `[` for the sidebar, `?` for every key.

Preferences, dashboards, saved table views, watches, hidden modules and the journal are kept per browser in
localStorage; nothing about the workspace is stored on the admin server.

## What is tested [#what-is-tested]

`pnpm --filter @better-compute/admin test` runs vitest over `apps/admin/tests/*.test.ts` in a Node environment with
fixed clocks. Components are not rendered; the config notes that the dev server and a browser cover them.

| File                 | Covers                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `upstream.test.ts`   | the caller check (loopback, DNS rebinding, cross-origin `Origin`, `ADMIN_ALLOWED_HOSTS`); the read-method rule; URL cleaning; the proxy's refusals (header, host, read-only, path, body size); a 502 that leaks neither URL nor token; 503 unconfigured; status never returning the token; share-link and note-link sanitizing                                                                |
| `signin.test.ts`     | sign-in against a stubbed Better IAM: the sealed, key-bound cookie; https beyond loopback; one message for wrong credentials and unknown organizations; the per-account throttle; MFA through a sealed one-use challenge and its 5-code limit; forged challenges; sign-out at Better IAM and when it is unreachable; the proxy forwarding the operator's bearer only with both cookie and key |
| `session.test.ts`    | sealing round trips; expired, tampered, foreign and wrong-purpose values; the browser key; decoy cookies; revocation; cookie attributes; short secrets; the `next` redirect                                                                                                                                                                                                                   |
| `foundation.test.ts` | the simulation core, fleet pulse, attention merging, the module registry and per-browser hiding, the change journal (no secrets, no reads, no `READ_ONLY`), widget pins, dashboard share links                                                                                                                                                                                                |
| `fleet.test.ts`      | the simulated fleet's invariants, `eventsBetween`, operator actions over time, `liveSnapshot` mapping, fleet attention and watch describers (nothing in live mode before the first answer)                                                                                                                                                                                                    |
| `capacity-*.test.ts` | the simulated capacity source, scenarios, runway, lanes, quota cells and suggestions, "Why it's waiting", the palette's capacity questions, and watch describers                                                                                                                                                                                                                              |

The server routes are tested with a stubbed `fetch`, and the live mappings with hand-built API answers. Nothing here
calls a running compute server or Better IAM.

Related: [Clients](/docs/guides/clients), [Console](/docs/guides/clients/console),
[CLI](/docs/guides/clients/cli), [API reference](/docs/reference/api).


# CLI (/docs/guides/clients/cli)

> Use the better-compute command line as a client - authenticate with a token, run a command for every API group, reserve GPUs, run sandboxes, plan the fleet.



`better-compute` is one binary for two jobs. Operators use it to run and maintain a deployment (`migrate`, `serve`,
`doctor`, `pki`). Tenants, operators and CI jobs also use it as an API client: every token command sends the same
request as the TypeScript client and is decided by Better IAM as the token's identity. This page covers the client
side. The [CLI reference](/docs/reference/cli) lists every command and flag.

```bash
npx better-compute help            # the commands, grouped
npx better-compute help reserve    # one command's flags and examples
npx better-compute help --json     # the whole manifest, as the reference is generated
```

With some 180 commands, load shell completion (581d1fb): `better-compute completion bash`, `zsh` (through
`bashcompinit`) or `fish` prints a script, generated from the program's commands (a configuration's project commands
included), that completes command words, the flags of the command on the line and the values of flags with choices.
Only plainly shaped names are emitted, and every word list and description is single-quoted without control
characters, so nothing a command defines runs in your shell.

```bash
source <(better-compute completion bash)     # bash: in ~/.bashrc
source <(better-compute completion zsh)      # zsh: in ~/.zshrc
better-compute completion fish | source      # fish: in ~/.config/fish/config.fish
```

## Two kinds of command [#two-kinds-of-command]

Every command declares a target, and the target decides what it needs:

| Target   | Needs                                                                                                                                                            | Commands                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `token`  | a credential (`BETTER_COMPUTE_TOKEN` or a saved one, see [authenticate](#authenticate)), and either `--url` or a configuration                                   | `api`, every command of the [command groups](#command-groups) below (`nodes`, `instances`, `rollouts`, `metal`, `webhooks`, ...), `reserve`, `release`, `extend`, `reservations list/get`, `pools list`, `capacity`, `sandbox …`, `sandbox-template …`, `sandbox-snapshot …`, `console`, `exec`, `forward`, `ssh-cert`, `fleet export/plan/status/apply`, `batch list/show/approve/create/members/cancel`, `auth login` |
| `config` | the configuration (`--config`, `BETTER_COMPUTE_CONFIG`, the nearest `better-compute.config.{mjs,js,ts,mts}`, or `BETTER_COMPUTE_DATABASE_URL`) and no credential | `migrate`, `doctor`, `serve`, `audit checkpoint/verify/prove`                                                                                                                                                                                                                                                                                                                                                           |
| `none`   | neither                                                                                                                                                          | `init`, `version`, `completion`, `dev` (it finds a configuration itself, or runs PGlite), `sim`, `pki`, `daemon install-script`, `iam setup`, `release keygen/sign/cosign/verify`, `batch keygen/sign/sign-approvers`, `audit check-proof`                                                                                                                                                                              |

A `config` command talks to the database directly. It is an operator tool: whoever can run it can already read the
database, so it is not decided by IAM. A `token` command never reads the database itself.

## Authenticate [#authenticate]

Token commands act as a Better IAM session token, API key or session JWT. They take the first credential they find,
in PLAN.md section 9.3's order:

1. `BETTER_COMPUTE_TOKEN`;
2. Better IAM's saved session (`better-iam login`), when the optional `@better-iam/cli` peer is installed: its profile
   store, profile `BETTER_IAM_PROFILE` or its current one;
3. compute's own credentials file, written by `better-compute auth login`: profile `BETTER_COMPUTE_PROFILE`, else the
   file's current profile, else `default`.

Expired saved sessions are skipped, and a broken Better IAM file never blocks compute's own. A saved credential
(2 and 3) is used only for the deployment it belongs to. A Better IAM session issued by a remote endpoint
(`better-iam login --url https://compute.example.com/api/iam`) goes only to a `--url` of that same origin, and one
issued in process only to an in-process deployment; any other is passed over, so
`nodes list --url https://staging.example.com` never carries the production session. If your IAM and compute are
served from different origins, save a profile for the compute URL with `better-compute auth login --url …`, or set
`BETTER_COMPUTE_TOKEN`, which is sent wherever the command goes. Without any credential token commands stop with
`MISSING_ENV` (the hint names a saved credential that was passed over, and why); public methods such as
`system.version` need no token.

```bash
export BETTER_COMPUTE_URL=https://compute.example.com
export BETTER_COMPUTE_TOKEN=biam_…          # from Better IAM: an API key or a session token
export BETTER_COMPUTE_TENANT=acme           # the default --tenant
better-compute api system.whoami
```

**Saved credentials.** `auth login` reads the token from standard input or `BETTER_COMPUTE_TOKEN`, never from an
argument, checks it with `system.whoami` against the deployment (`--url`, or the configuration in process;
`--no-verify` skips the check), and saves it with its tenant under `--profile`, which becomes the current profile:

```bash
better-compute auth login --url https://compute.example.com < token.txt
BETTER_COMPUTE_TOKEN=biam_key_… better-compute auth login --profile ci --no-verify
better-compute auth status          # which credential commands use: source, profile, path; never a token
better-compute auth logout --profile ci
```

The file is `BETTER_COMPUTE_CREDENTIALS`, else `%APPDATA%\better-compute\credentials.json` on Windows, else
`$XDG_CONFIG_HOME/better-compute/credentials.json` or `~/.config/better-compute/credentials.json`. It is found only
from the environment, and every write replaces it atomically with mode 0600; treat it like a password. `auth logout`
forgets a profile but does not revoke the session in Better IAM (`better-iam logout` does). A saved profile records
the `--url` it was saved for, but commands still take the URL from `--url` or `BETTER_COMPUTE_URL`. The profile's
token is used only for that deployment: a command whose URL has another origin, or a profile saved in process
(without `--url`) when a command calls a remote server, gets no token from it, so a token for one deployment is
never sent to another.

**Where the call goes.**

* With `--url` (or `BETTER_COMPUTE_URL`) the command calls a running server over HTTP through the typed client, with
  a request id on every call and one retry after a short `RATE_LIMITED`. The URL is an origin
  (`https://compute.example.com`, base path `/api/compute`) or a full endpoint (`https://host/custom/compute`).
  Plain `http` is allowed only for `localhost`, `127.0.0.1`, `[::1]` and `*.localhost`, so a token never crosses a
  network in clear text.
* Without `--url` the command loads your configuration and calls `compute.handler` in process. It still goes through
  the route table, the envelope, the CSRF rules and the 64 KiB body limit, and IAM still decides as the token, so a
  command behaves the same locally and remotely.

`--tenant` names the tenant acting, and defaults to `BETTER_COMPUTE_TENANT`. It must be the token's own tenant, or the
root tenant for platform operators; anything else is `ACCESS_DENIED` before compute reads its database.

## Output [#output]

Commands that print a result share two flags:

| Flag               | Does                                                                                             |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| `--format json`    | indented JSON (the default)                                                                      |
| `--format compact` | JSON on one line, for piping                                                                     |
| `--format table`   | a table, for lists                                                                               |
| `--query PATH`     | prints part of the result, like `binding`, `items[].id` or `findings[].check`; strings print raw |

A page of a list method (`{ items, nextCursor }`) printed as a table ends with `(more rows: --cursor …)` when there
is a next page (5df9455).

Output is terminal-safe, since results carry the server's words (names, reasons, messages): a string result, or a
`--query` that selects one, prints with its control characters turned into spaces (line breaks and tabs stay); table
cells are one line without control characters; JSON also escapes DEL and the C1 controls (U+007F to U+009F).

A failure prints its code and message, plus a hint line saying what to run or set next. Usage mistakes (an unknown
flag, a missing value) exit 2 with `INVALID_ARGUMENT`, and unknown commands and flags get a "did you mean"
suggestion. `BETTER_COMPUTE_DEBUG=1` prints the stack trace of a failure.

## Call any method [#call-any-method]

`better-compute api GROUP.METHOD` calls any method, including plugin methods (`plugins.ID.GROUP.METHOD`). Build the
input from `key=value` items, `--data`, or both:

| Item                                           | Sets                                                        |
| ---------------------------------------------- | ----------------------------------------------------------- |
| `name=gpu-a`                                   | a string                                                    |
| `limit:=10`, `labels:='["a","b"]'`             | a JSON value                                                |
| `spec:=@spec.json`                             | a JSON value read from a file                               |
| `notes=@notes.md`                              | a string read from a file                                   |
| `resource.type=pool`                           | a nested field                                              |
| `ids[]=a`                                      | an item appended to an array                                |
| `--data JSON`, `--data @file.json`, `--data -` | the whole body, from the argument, a file or standard input |

```bash
better-compute api --list                  # every route, whether it needs a token, and its IAM action
better-compute api --list reservations     # one group
better-compute api events.list tenantId=acme type=reservation. limit:=20 --format table
better-compute api webhooks.create tenantId=acme url=https://hooks.example.com/compute 'events[]=reservation.*'
better-compute api reservations.get tenantId=acme reservationId=rsv_… --query binding
```

`api --list` reads the deployment's route table, so it shows plugin routes and marks the public ones. Anything
`api` can reach, the [API reference](/docs/reference/api) documents.

## Command groups [#command-groups]

PLAN.md section 9.3 has the commands mirror the API groups, and since 171d9be every API group has commands or an
existing equivalent (44915e5, e2fe6f8, bb89083, d1e83ca, 573b0b0, 171d9be). Each one is the API call as the token's
identity, with typed flags in place of `key=value` items; `api` stays
the way to any method a group's commands leave out (a raw `commands.run`, for one) and to plugin methods.

| Commands                                                                                                                                                                                                                                           | What they do                                                                                                                                                                                                                                                                                                                                                      | API group                                                                                                                                                                                                               |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`nodes`](/docs/reference/cli#nodes-list) `list`, `get`, `events`, `cordon`, `uncordon`, `drain`, `label`, `taint`, `delete`                                                                                                                       | Read nodes and their events; cordon, drain, label and taint them (`--if-version` for optimistic concurrency). [`delete`](/docs/reference/cli#nodes-delete) (4049621) removes a retired or cordoned adopted node for good and refuses without `--yes` before any call; see [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host) | [nodes](/docs/reference/api/nodes)                                                                                                                                                                                      |
| [`join-tokens`](/docs/reference/cli#join-tokens-create) `create`, `list`, `revoke`                                                                                                                                                                 | Mint the tokens `bcd join` presents                                                                                                                                                                                                                                                                                                                               | [joinTokens](/docs/reference/api/join-tokens)                                                                                                                                                                           |
| [`join-requests`](/docs/reference/cli#join-requests-list) `list`, `approve`, `reject`                                                                                                                                                              | Admit or turn down machines that asked to join                                                                                                                                                                                                                                                                                                                    | [joinRequests](/docs/reference/api/join-requests)                                                                                                                                                                       |
| [`inventory`](/docs/reference/cli#inventory-get) `get`, `history`, `diff`                                                                                                                                                                          | A node's hardware inventory and what changed                                                                                                                                                                                                                                                                                                                      | [inventory](/docs/reference/api/inventory)                                                                                                                                                                              |
| [`site-tokens`](/docs/reference/cli#site-tokens-create) `create`, `revoke`                                                                                                                                                                         | The tokens a site agent joins with                                                                                                                                                                                                                                                                                                                                | [siteTokens](/docs/reference/api/site-tokens)                                                                                                                                                                           |
| [`instances`](/docs/reference/cli#instances-list) `list`, `get`, `create`, `start`, `stop`, `delete`, `resize`                                                                                                                                     | VPS instances; see [instances](/docs/guides/vps/instances#from-the-command-line)                                                                                                                                                                                                                                                                                  | [instances](/docs/reference/api/instances)                                                                                                                                                                              |
| [`events`](/docs/reference/cli#events-list) `list`, `watch`                                                                                                                                                                                        | Page through events, or follow them live (below)                                                                                                                                                                                                                                                                                                                  | [events](/docs/reference/api/events)                                                                                                                                                                                    |
| [`releases`](/docs/reference/cli#releases-register) `register`, `list`                                                                                                                                                                             | Signed bcd release manifests                                                                                                                                                                                                                                                                                                                                      | [releases](/docs/reference/api/releases)                                                                                                                                                                                |
| [`rollouts`](/docs/reference/cli#rollouts-create) `create`, `list`, `get`, `pause`, `resume`, `abort`                                                                                                                                              | Wave rollouts of a bcd release; see [rollouts](/docs/guides/rollouts)                                                                                                                                                                                                                                                                                             | [rollouts](/docs/reference/api/rollouts)                                                                                                                                                                                |
| [`workflows`](/docs/reference/cli#workflows-list) `list`, `get`, `abort`, `pause`, `resume`, `retry`                                                                                                                                               | A node's workflows (`--node`)                                                                                                                                                                                                                                                                                                                                     | [workflows](/docs/reference/api/workflows)                                                                                                                                                                              |
| [`commands`](/docs/reference/cli#commands-list) `list`, `get`, `cancel`                                                                                                                                                                            | A node's queued commands (`--node`); there is no `commands run`                                                                                                                                                                                                                                                                                                   | [commands](/docs/reference/api/commands)                                                                                                                                                                                |
| [`sites`](/docs/reference/cli#sites-list) `list`, `get`, `create`, `update`, `delete`                                                                                                                                                              | Sites                                                                                                                                                                                                                                                                                                                                                             | [sites](/docs/reference/api/sites)                                                                                                                                                                                      |
| [`projects`](/docs/reference/cli#projects-list) `list`, `get`, `create`, `archive`                                                                                                                                                                 | Projects                                                                                                                                                                                                                                                                                                                                                          | [projects](/docs/reference/api/projects)                                                                                                                                                                                |
| [`images`](/docs/reference/cli#images-list) `list`, `get`, `register`, `promote`                                                                                                                                                                   | The image catalog (`register` takes `--disk-format`)                                                                                                                                                                                                                                                                                                              | [images](/docs/reference/api/images)                                                                                                                                                                                    |
| [`flavors`](/docs/reference/cli#flavors-list) `list`, `create`, `update`, `delete`                                                                                                                                                                 | VPS flavors of a pool                                                                                                                                                                                                                                                                                                                                             | [flavors](/docs/reference/api/flavors)                                                                                                                                                                                  |
| [`quotas`](/docs/reference/cli#quotas-get) `get`, `set`                                                                                                                                                                                            | Quotas per pool and resource                                                                                                                                                                                                                                                                                                                                      | [quotas](/docs/reference/api/quotas)                                                                                                                                                                                    |
| [`usage`](/docs/reference/cli#usage-list) `list`, `export`                                                                                                                                                                                         | Usage windows, and a billing export (`--export-format csv` or `jsonl`)                                                                                                                                                                                                                                                                                            | [usage](/docs/reference/api/usage)                                                                                                                                                                                      |
| [`webhooks`](/docs/reference/cli#webhooks-list) `list`, `get`, `create`, `update`, `delete`, `ping`, `rotate-secret`, `deliveries`, `redeliver`, `event-types`                                                                                     | Webhook subscriptions (`--endpoint` is the receiver's URL)                                                                                                                                                                                                                                                                                                        | [webhooks](/docs/reference/api/webhooks)                                                                                                                                                                                |
| [`pools`](/docs/reference/cli#pools-get) `get`, `create`, `update`, `archive`, `offer`, `unoffer`, and [`pools list`](/docs/reference/cli#pools-list)                                                                                              | Pools and their offers                                                                                                                                                                                                                                                                                                                                            | [pools](/docs/reference/api/pools)                                                                                                                                                                                      |
| [`drivers list`](/docs/reference/cli#drivers-list)                                                                                                                                                                                                 | The registered drivers                                                                                                                                                                                                                                                                                                                                            | [drivers](/docs/reference/api/drivers)                                                                                                                                                                                  |
| [`secrets`](/docs/reference/cli#secrets-put) `put`, `delete`                                                                                                                                                                                       | Write-only secrets, the value from a file or standard input only                                                                                                                                                                                                                                                                                                  | [secrets](/docs/reference/api/secrets)                                                                                                                                                                                  |
| [`metal`](/docs/reference/cli#metal-enroll) `enroll`, `manage`, `inspect`, `provide`, `rescue`, `unrescue`, `recover`, `retire`, `deploy`, `undeploy`, `abort`, `discovered`, `confirm-mac`, `power`, `bulk`                                       | The bare-metal lifecycle; see [deploying](/docs/guides/bare-metal/deploying#from-the-command-line). `discovered` and `confirm-mac` (002dd95) list a site's PXE discovery and bind a machine's PXE MAC; see [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines)                                                                     | [metal](/docs/reference/api/metal)                                                                                                                                                                                      |
| [`profiles`](/docs/reference/cli#profiles-list) `list`, `get`, `validate`, `render`, `create`, `update`, `delete`                                                                                                                                  | Provision profiles, from JSON documents                                                                                                                                                                                                                                                                                                                           | [profiles](/docs/reference/api/profiles)                                                                                                                                                                                |
| [`provider-accounts`](/docs/reference/cli#provider-accounts-list) `list`, `get`, `create`, `update`, `delete`, `offers`, `bootstrap`                                                                                                               | Provider accounts (`--config-file`, credentials in it as references)                                                                                                                                                                                                                                                                                              | [providerAccounts](/docs/reference/api/provider-accounts)                                                                                                                                                               |
| [`reserve`](/docs/reference/cli#reserve), `release`, `extend`, `reservations list`, `reservations get`                                                                                                                                             | Reservations ([below](#capacity))                                                                                                                                                                                                                                                                                                                                 | [reservations](/docs/reference/api/reservations)                                                                                                                                                                        |
| [`capacity`](/docs/reference/cli#capacity)                                                                                                                                                                                                         | Free capacity per pool and GPU model, and a preview                                                                                                                                                                                                                                                                                                               | [capacity](/docs/reference/api/capacity)                                                                                                                                                                                |
| [`sandbox`](/docs/reference/cli#sandbox-create), [`sandbox-template`](/docs/reference/cli#sandbox-template-create), [`sandbox-snapshot`](/docs/reference/cli#sandbox-snapshot-list), [`sandbox-volume`](/docs/reference/cli#sandbox-volume-create) | Sandboxes, their templates, their snapshots and their persistent volumes ([below](#sandboxes))                                                                                                                                                                                                                                                                    | [sandboxes](/docs/reference/api/sandboxes), [sandboxTemplates](/docs/reference/api/sandbox-templates), [sandboxSnapshots](/docs/reference/api/sandbox-snapshots), [sandboxVolumes](/docs/reference/api/sandbox-volumes) |
| [`console`](/docs/reference/cli#console), `exec`, `forward`; [`ssh-cert`](/docs/reference/cli#ssh-cert)                                                                                                                                            | Tunnels and break-glass SSH ([below](#interactive-access))                                                                                                                                                                                                                                                                                                        | [tunnels](/docs/reference/api/tunnels), `nodes.issueSshCert`                                                                                                                                                            |
| [`fleet`](/docs/reference/cli#fleet-export) `export`, `plan`, `status`, `apply`                                                                                                                                                                    | [Fleet configuration](#fleet-configuration)                                                                                                                                                                                                                                                                                                                       | [fleet](/docs/reference/api/fleet)                                                                                                                                                                                      |
| [`batch`](/docs/reference/cli#batch-list) `list`, `show`, `approve`, `create`, `members`, `cancel`                                                                                                                                                 | [Destructive batches](#destructive-batches)                                                                                                                                                                                                                                                                                                                       | [commandBatches](/docs/reference/api/command-batches)                                                                                                                                                                   |
| [`auth login`](/docs/reference/cli#auth-login) and `api system.whoami`                                                                                                                                                                             | Check a token (`auth login` calls `system.whoami`)                                                                                                                                                                                                                                                                                                                | [system](/docs/reference/api/system)                                                                                                                                                                                    |

What they share:

* **Ids are checked before anything is sent.** An id that does not look like its kind (`node_…`, `inst_…`,
  `rollout_…`, `wf_…`, `cmd_…`, `pool_…`, `jreq_…`, `pacct_…`, `snap_…`, `vol_…`) is a usage error, exit 2.
* **Lists print tables.** List commands default to `--format table`, and a page with more rows ends with
  `(more rows: --cursor …)` (5df9455); pass that cursor to get the next page.
* **Creates are retry-safe.** A command that creates something sends a new idempotency key on every run unless you
  pass `--key`; repeat with the same `--key` to retry without creating twice.
* **Secrets never pass through arguments or the output unless you ask.** `secrets put` reads `--value-file` or
  `--stdin`; `metal enroll` takes a BMC password only from `--bmc-password-file`; `join-tokens create` and
  `site-tokens create` with `--out`, and `webhooks create` and `rotate-secret` with `--secret-out`, write the one-time
  secret to a new 0600 file instead of printing it, never over an existing file (`ALREADY_EXISTS`), and so does
  `usage export --out` with its export. `join-tokens revoke` and `site-tokens revoke` refuse a pasted token (anything
  with a `.`), so a secret is never sent as an id.
* **Bulk work goes through the quorum.** `metal bulk` runs one verb on the nodes of `--nodes` or `--nodes-file`
  (deduplicated) as one batch, and its required `--reason` is what the batch's approvers see.

`tests/cli-framework.test.ts` ties the commands to the API (fb6304f): every `group.method` a command summary names,
and every route a command calls, must exist in the method table and the live registry, so a renamed route fails the
suite rather than a user's first call. The [CLI reference](/docs/reference/cli) has every command's flags.

### Follow events live [#follow-events-live]

`events watch` (e2fe6f8) follows the host's server-sent events route, the same stream as the client's `$watch`, and
prints one JSON line per event:

```bash
better-compute events watch --url https://compute.example.com --tenant acme --types 'node.*,reservation.*'
better-compute events watch --url https://compute.example.com --count 100
better-compute events watch --url https://compute.example.com --path /api/compute/watch   # the console app's route
```

* It needs `--url` (`UNSUPPORTED` without it): it follows a running server's stream. `--path` defaults to `/watch`,
  the route the examples' hosts mount; the console app serves `/api/compute/watch`.
* `--tenant` is sent as `?tenantId=`, which the host's watch authorizer decides as `events.list` for that tenant.
* Each line carries the event's `sequence`, the stream's cursor, and `--since` resumes after one: pass the last
  `sequence` a previous run printed (86706b4). It reconnects after a broken connection from the last event, and stops
  on Ctrl-C, after `--count` events, or on a refusal (a revoked session, a withdrawn `compute:events:read`).
* `--types` takes up to 20 type patterns; they and `--since` are checked before it connects.
* Lines are terminal-safe: DEL, the C1 controls and U+2028/U+2029, which JSON leaves raw, are escaped as `\uXXXX`.

Both event commands take an outbox position, written two ways. `events watch --since` accepts either form and sends
the stream the dotted one (867c837), so a watch can resume from where an `events list` page ended; `events list --cursor` takes only the colon form:

| Command        | Flag       | Form                                                        | Where it comes from                                                                                                  |
| -------------- | ---------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `events watch` | `--since`  | `<txid>.<seq>`, with a dot (`184467.12`), or `<txid>:<seq>` | The `sequence` of a line the stream printed (`formatCursor` in `@better-compute/events`), or an `events.list` cursor |
| `events list`  | `--cursor` | `<txid>:<seq>`, with a colon (`184467:12`)                  | The `nextCursor` of the previous `events.list` page                                                                  |

```bash
# resume a watch after the last event a previous run printed
better-compute events watch --url https://compute.example.com --since 184467.12 --count 100
# page through the tenant's events
better-compute events list --limit 100 --cursor 184467:12
```

See [watch](/docs/guides/events/watch).

## Capacity [#capacity]

The capacity commands wrap the reservation methods with friendlier flags:

```bash
better-compute pools list
better-compute capacity                                  # per pool and per GPU model
better-compute capacity --preview --pool pool_01k… --gpus 8 --for 1d --selector "vramGiB >= 80"
better-compute reserve --pool pool_01k… --gpus 4 --for 8h --nvlink
better-compute reservations list --state active --format table
better-compute extend rsv_01k… --by 4h
better-compute release rsv_01k…
```

`reserve` sends a new idempotency key on every run unless you pass `--key`, and prints the key it used. To retry the
same request, whether after a timeout or after a person approved a `CONFIRMATION_REQUIRED` call, repeat the command
with that `--key`. It waits up to `--wait` seconds (default 20) while the reservation is queued. `--selector` takes a
shorthand such as `gpuProduct in (H100-SXM5-80GB), vramGiB >= 80`, and `--model` and `--product` add a `gpuModel`
or `gpuProduct` match. See [reservations](/docs/guides/capacity/reservations).

## Sandboxes [#sandboxes]

The `sandbox` commands (help group `Sandboxes`) wrap the `sandboxes` and `sandboxTemplates` methods of
[sandboxes](/docs/guides/vps/sandboxes):

```bash
better-compute sandbox templates --format table                 # sandboxTemplates.list, with warm pools
better-compute sandbox create --template sbxt_01k5w… --ttl 30m   # sandboxes.create
better-compute sandbox list --all --template sbxt_01k5w…
better-compute sandbox get sbx_01k5w…                           # with its snapshots
better-compute sandbox extend sbx_01k5w… --by 30m
better-compute sandbox snapshot sbx_01k5w… --keep
better-compute sandbox exec sbx_01k5w… -- pytest -q
better-compute sandbox delete sbx_01k5w…
```

| Command                              | Flags                                                                                                                                                                                 | Calls                                                                                                             |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `sandbox create`                     | `--template ID` (required), `--ttl DURATION` (`15m`, `2h`; whole seconds), `--from-snapshot ID`, `--wait SECONDS` (0 to 30, default 30), `--key KEY`                                  | `sandboxes.create`, with a new idempotency key unless `--key` is given; waits until the sandbox leaves `starting` |
| `sandbox list`                       | `--all` (the tenant's, not only yours), `--deleted`, `--template ID`, `--limit N` (1 to 200, as the API), `--cursor`                                                                  | `sandboxes.list`                                                                                                  |
| `sandbox get SANDBOX`                |                                                                                                                                                                                       | `sandboxes.get`                                                                                                   |
| `sandbox extend SANDBOX`             | exactly one of `--by DURATION` or `--until TIME` (ISO 8601, epoch ms, or `+DURATION` from now)                                                                                        | `sandboxes.extend`                                                                                                |
| `sandbox snapshot SANDBOX`           | `--no-memory` (disk only), `--keep` (outlive the sandbox), `--key KEY`                                                                                                                | `sandboxes.snapshot`, always with an idempotency key                                                              |
| `sandbox delete SANDBOX`             |                                                                                                                                                                                       | `sandboxes.delete`                                                                                                |
| `sandbox exec SANDBOX [-- COMMAND…]` | `--key PATH` (default `~/.ssh/id_ed25519`), `--minutes N` (1 to 15, default 10), `--ssh PROGRAM` (default `ssh`), `--accept-host-key` (trust an uncertified guest's key on first use) | `sandboxes.exec`, then OpenSSH through the tunnel                                                                 |
| `sandbox templates`                  |                                                                                                                                                                                       | `sandboxTemplates.list`                                                                                           |

A sandbox id that does not look like `sbx_…` is a usage error (exit 2) before anything is sent.

`sandbox-snapshot list` (`--all`, `--sandbox ID`), `get` and `delete` (171d9be) wrap `sandboxSnapshots`, so a kept
snapshot of a deleted sandbox can be found and removed.

**`sandbox exec`** gets a short-lived certificate for your own key and runs `ssh` through a single-use tunnel; there
is no agent in the guest. It sends the public key beside `--key` (`<key>.pub`; a file that holds a private key is
refused) and never the private key, writes the certificate into a temporary directory, listens on a random
`127.0.0.1` port for one connection that it bridges to the tunnel, and runs `--ssh` with `-F none` (no user or
system `ssh_config`), `-i <key>`, `CertificateFile`, `IdentitiesOnly=yes` and a known-hosts file of its own (as both
`UserKnownHostsFile` and `GlobalKnownHostsFile`), as the account the server names. A guest with a still-valid host
certificate from the sandbox CA (the grant's `hostCertified`) is pinned: the file holds
`@cert-authority <hostPrincipal> <sandbox CA key>`, with `HostKeyAlias=<hostPrincipal>` and
`StrictHostKeyChecking=yes`; a malformed `hostPrincipal` is `INVALID_RESPONSE`. Any other guest (from before host
certificates, or whose certificate expired) is refused with `INVALID_STATE` and nothing is spawned, unless you pass
`--accept-host-key`: then it gets an empty file, `StrictHostKeyChecking=accept-new` and `HostKeyAlias=<sandbox id>`,
with a note that its key is trusted on first use for this connection. The account must be a Unix
account name (`INVALID_RESPONSE` otherwise). The command after `--` runs remotely; without one you get a login shell. The temporary directory is
removed afterwards, whatever happened, and the CLI exits with ssh's own status (`REMOTE_EXIT`). Like the other tunnel
commands it needs `--url`, and `sandboxes.exec` needs a signed-in session rather than an API key. It is refused
(`INVALID_STATE`, `restore-shares-identity`) while a memory restore and its source, or two restores of one memory
snapshot, are live together and still share one SSH identity, that is until the host gives the restore its own. The
CLI then adds a hint (9958865): delete the sandbox it was restored from or its sibling restores, or wait until
`sandbox get` shows `identityRebound: true`. See
[a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity).

The guest is untrusted, so the connection is hardened against it and against the command you pass:

* a `--` precedes the host, so the remote command is never parsed as an ssh option (`sandbox exec sbx_… --
  -oProxyCommand=…` runs that text in the guest, not on your machine), which matters when an agent or a model writes
  the command;
* none of your ssh configuration that would hand the guest anything applies: `-a` and `-x`, `ForwardAgent=no`,
  `ForwardX11=no`, `ClearAllForwardings=yes`, `ProxyCommand=none`, `ProxyJump=none`, `ControlMaster=no`,
  `ControlPath=none` and `PermitLocalCommand=no`;
* the `CertificateFile` and `UserKnownHostsFile` paths are quoted, since ssh splits `-o` values on whitespace and a
  home or temporary directory may contain a space.

### Sandbox templates [#sandbox-templates]

Tenant administrators define templates with `sandbox-template`, decided as `compute:sandboxes:manage`:

```bash
better-compute sandbox-template create --name python --pool pool_01k5w… --flavor small --image img_01k5w… \
  --warm-min 2 --max-ttl 4h --env OPENAI_API_KEY=secret://acme/sec_01k5w… --network private
better-compute sandbox-template update sbxt_01k5w… --warm-min 4 --image img_01k5w…
better-compute sandbox-template delete sbxt_01k5w…
```

| Command                            | Flags                                                                                                                                                          | Calls                                                |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `sandbox-template create`          | `--name NAME` and `--pool ID` (required), `--flavor NAME` and `--image ID` (needed), `--runtime ID` (checked against the pool's), and the template flags below | `sandboxTemplates.create`                            |
| `sandbox-template update TEMPLATE` | the template flags, `--no-user-data` (remove it)                                                                                                               | `sandboxTemplates.update` with only the fields given |
| `sandbox-template delete TEMPLATE` |                                                                                                                                                                | `sandboxTemplates.delete`                            |

The template flags are `--flavor`, `--image`, `--user-data FILE` (cloud-init read from a file), `--default-ttl` and
`--max-ttl` (durations such as `30m` or `4h`, whole seconds), `--warm-min N` and `--warm-max N` (the API accepts 0 to
64; `--warm-max` needs `--warm-min`), `--env NAME=secret://<tenant>/<id>,…` (secret references, never values) and
`--network egress|private|none`, and `--volumes NAME:/PATH[:ro],…` (f181f43): the persistent volumes every sandbox
mounts, in order, each by name or by id (an entry that looks like `vol_…` is sent as `volumeId`), with `:ro` for a
read-only mount; `--volumes ""` removes them all, and an entry without an absolute path or with a mode other than `ro`
is a usage error. An `update` without any field flag, or with both `--user-data` and `--no-user-data`, is a usage
error, and so is a template id that does not look like `sbxt_…`. See
[templates](/docs/guides/vps/sandboxes#templates).

### Sandbox volumes [#sandbox-volumes]

`sandbox-volume` (f181f43) manages [persistent volumes](/docs/guides/vps/sandboxes#volumes), an agent's memory, with
the `sandboxVolumes` methods:

```bash
better-compute sandbox-volume create --pool pool_01k5w… --name agent-notes --size 20
better-compute sandbox-volume list --pool pool_01k5w… --format table
better-compute sandbox-volume get vol_01k5w…
better-compute sandbox-volume resize vol_01k5w… --size 40
better-compute sandbox-volume delete vol_01k5w…
better-compute sandbox-template update sbxt_01k5w… --warm-min 0 --warm-max 0 --volumes agent-notes:/workspace
```

| Command                        | Flags                                                                                                                                            | Calls                                                                                                     |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `sandbox-volume create`        | `--pool ID`, `--name NAME` and `--size GIB` (1 to 16,384) required, `--key KEY`                                                                  | `sandboxVolumes.create`, with a new idempotency key unless `--key` is given                               |
| `sandbox-volume list`          | `--all` (every volume of the tenant you may see, not only yours; e68aba7), `--pool ID` (only that pool's), `--deleted` (include deleted volumes) | `sandboxVolumes.list`, with `scope: 'all'` for `--all` and the API's default, your own volumes, otherwise |
| `sandbox-volume get VOLUME`    |                                                                                                                                                  | `sandboxVolumes.get`                                                                                      |
| `sandbox-volume resize VOLUME` | `--size GIB` (1 to 16,384, required); volumes only grow                                                                                          | `sandboxVolumes.resize`                                                                                   |
| `sandbox-volume delete VOLUME` |                                                                                                                                                  | `sandboxVolumes.delete`, which wipes the data                                                             |

A volume id that does not look like `vol_…` is a usage error before anything is sent. A template that mounts volumes
keeps no warm pool, so give it `--warm-min 0 --warm-max 0` (the API refuses one whose `warm.max` is not 0), and
naming a volume in a template takes a decision on that volume (see [attach](/docs/guides/vps/sandboxes#volumes)). The
console's **Volumes** page does the same list, create, grow and delete (see
[the console](/docs/guides/clients/console#volumes)).

## Interactive access [#interactive-access]

Four commands in the `Access` group reach a node or an instance from your terminal, decided like any other call as
the token's identity:

```bash
better-compute console --url https://compute.example.com --tenant acme --node node_01k5w…
better-compute console --url https://compute.example.com --tenant acme --instance inst_01k5w…
better-compute exec --url https://compute.example.com --tenant acme --instance inst_01k5w…
better-compute forward --url https://compute.example.com --tenant acme --instance inst_01k5w… --port 5432
better-compute ssh-cert --tenant acme --node node_01k5w… && ssh -i ~/.ssh/id_ed25519 root@10.0.0.5
```

* **`console`** opens a tunnel with op `console`: with `--node`, a root console on the node, decided as
  `compute:nodes:console` on the node; with `--instance`, the serial console of a running instance, decided as
  `compute:instances:exec` on the instance (798e710). Give exactly one of the two. An instance console needs a node
  daemon that advertises `tunnel.instance-dial`, like `forward` below, and a sandbox's instance is refused
  (`sandbox-managed`: use `sandbox exec&#x60;). &#x2A;*`exec`** (`--instance`) opens a shell in a running instance with op
  `exec`, decided as `compute:instances:exec`; the instance's node agent runs it. bcd refuses instance `exec` until
  guests have an agent, so on a real host `console --instance` is the terminal that works.
* Both open the tunnel's WebSocket on the same server as the API call, so they need `--url` (or
  `BETTER_COMPUTE_URL`). Without it they fail with `UNSUPPORTED` before a ticket is spent: an in-process call has no
  WebSocket server.
* The terminal is in raw mode until the remote program exits: every key goes to the remote, Ctrl-C included, and
  the window size follows resizes. &#x2A;*Ctrl-]** detaches. The command exits with the remote program's status, as ssh
  does, and with 255 when the tunnel itself breaks (`REMOTE_EXIT`).
* **`forward`** (`--instance`, `--port` 1 to 65535) works like `ssh -L`: it listens on `--bind` (default
  `127.0.0.1`) at `--local-port` (default the same as `--port`; `0` picks a free one) and forwards each accepted
  connection to that port of the running instance through a `tcp` tunnel of its own, a fresh `tunnels.open` per
  connection decided as `compute:instances:exec`. It runs until Ctrl-C. The token is resolved before anything
  listens, and it needs `--url` like `console` and `exec`. A connection that cannot be opened is closed with a note;
  the tunnel has no half-close, so a client that ends its side ends the connection. Memory stays bounded: the local
  socket pauses while the tunnel holds more than 4 MiB unsent and resumes below 1 MiB, and a local reader more than
  16 MiB behind is disconnected, since the WebSocket's receiving side cannot be paused. A `--bind` other than loopback
  lets anyone who reaches that address use your access. Instance `tcp` and `ssh` tunnels open only when the node's
  daemon advertises the `tunnel.instance-dial` capability in its `Hello`; otherwise `tunnels.open` answers
  `UNSUPPORTED` with `details.reason: 'daemon-capability'`.
* **`ssh-cert`** sends your own public key (`--key`, default `~/.ssh/id_ed25519.pub`, else `id_ecdsa.pub`) to
  `nodes.issueSshCert` and writes the node-bound certificate beside it as `<key>-cert.pub`, where `ssh -i <key>`
  finds it. `--account` (default `root`) is the Unix account and `--minutes` (default 15, 1 to 60) the validity,
  never past your session. It needs `compute:nodes:ssh-root` and, by default, an MFA session. The private key never
  leaves your machine: a `--key` that is not a `.pub` file, or that holds a private key, is refused before anything
  is sent. The answer is checked before use: an account that is not a Unix account name, or a certificate that is not
  an OpenSSH user certificate for your key, is `INVALID_RESPONSE&#x60; and nothing is written. With &#x2A;*`--connect HOST`** it then runs your OpenSSH client (`--ssh`, default `ssh`) to the node's
  address with the new certificate: `-i <key> -o CertificateFile=<cert> -o IdentitiesOnly=yes -l <account> -- HOST`,
  followed by the command after `--`, or a shell without one. `HOST` must be a host name or an IP address, never
  something ssh would read as an option (`-oProxyCommand=…`), and is checked before a certificate is issued; a
  command without `--connect` is a usage error. The command exits with ssh's status (`REMOTE_EXIT`). Unlike
  `sandbox exec`, it keeps your ssh configuration on purpose, so bastions and jump hosts still apply:

  ```bash
  better-compute ssh-cert --tenant acme --node node_01k5w… --connect 10.0.0.5 -- journalctl -u bcd -n 50
  ```

See [tunnels](/docs/guides/access/tunnels) and [break-glass SSH](/docs/guides/access/ssh).

## Fleet configuration [#fleet-configuration]

`fleet` manages a tenant's platform objects (sites, provider accounts, pools with quotas and offers, VPS flavors,
sandbox templates) as
one [FleetConfig](/docs/guides/fleet-config) document. The loop is export, edit, plan, review, apply:

      ### Export [#export]

    ```bash
    better-compute fleet export --out fleet.json
    ```

    Objects the fleet did not create are keyed by id. Rename them in the document and the next apply keeps them.
    Secrets appear only as references.
  
      ### Plan [#plan]

    ```bash
    better-compute fleet plan fleet.json            # + create, ~ update, - delete (only with --prune), BLOCKED
    better-compute fleet plan fleet.json --calls    # also the API calls apply would make
    ```

    A plan changes nothing and prints a `planHash`. A plan with a blocked change (a site's region, a pool's class or
    site, a dangling name) exits `PLAN_BLOCKED`.
  
      ### Apply the reviewed plan [#apply-the-reviewed-plan]

    ```bash
    better-compute fleet apply fleet.json --plan-hash 3f1c…
    ```

    Every underlying call is decided before anything runs, then all of them run in one transaction, so a denial or a
    failure applies nothing. If the fleet changed since the plan, the hash no longer matches and the command exits
    `CONFIG_DRIFT`. Without `--plan-hash` or `--yes`, `apply` only plans.
  
In CI, fail a job when the live fleet no longer matches the committed document, or when it drifted from the last
apply:

```bash
better-compute fleet plan fleet.json --fail-on-drift   # exits FLEET_DRIFTED when the plan is not empty
better-compute fleet status --fail-on-drift            # exits FLEET_DRIFTED while the fleet.drift job sees drift
```

The fleet methods are decided on `compute:fleet:admin`, which only the root tenant's administrators hold. The MCP
server's fleet tools stop at `fleet_plan` for the same reason: a person applies. See
[fleet configuration](/docs/guides/fleet-config).

## Destructive batches [#destructive-batches]

With the v1.0 destructive-batch quorum on (`pki.approvers`), fleet-wide destructive commands run only once enough
operator approver keys have signed their batch. The `batch` commands are the approver's side (b3fd712, 6a7f1b7).
They never take the server's word for what a batch holds: every review pages the batch's scope and recomputes its
Merkle root before anything is signed.

| Command                                                           | Needs                                   | Does                                                                                                                                                                                              |
| ----------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batch keygen --out PATH`                                         | nothing                                 | a P-256 approver key (mode 0600, never overwritten) and `PATH.pub`; prints the `kid`. Encrypted when `BETTER_COMPUTE_APPROVER_PASSPHRASE` is set                                                  |
| `batch list [--state STATE]`                                      | a credential, `compute:batches:read`    | `commandBatches.list`: reason, state, size, approvals had and needed                                                                                                                              |
| `batch show --batch ID [--scope-out FILE]`                        | a credential, `compute:batches:read`    | the batch with its verified scope; `--scope-out` writes the input of `batch sign`                                                                                                                 |
| `batch approve --batch ID --key PATH`                             | a credential, `compute:batches:approve` | reviews the batch, then signs and submits your approval                                                                                                                                           |
| `batch approve --batch ID --signature FILE`                       | a credential, `compute:batches:approve` | submits an approval made offline, after checking it is for this batch                                                                                                                             |
| `batch sign --key PATH --scope FILE`                              | nothing                                 | the same review and signature on an offline machine                                                                                                                                               |
| `batch create --reason TEXT --scope FILE [--hours N] [--key KEY]` | a credential, `compute:batches:create`  | `commandBatches.create` from `{ subject, kind, payload }` leaves, each payload encoded as its kind's message so the leaf binds that operation (def72c8)                                           |
| `batch members --batch ID [--state S]`                            | a credential, `compute:batches:read`    | the batch's commands, with `awaitingApproval` while held (45d217e)                                                                                                                                |
| `batch cancel --batch ID`                                         | a credential, `compute:batches:approve` | ends a pending batch; its held commands are cancelled                                                                                                                                             |
| `batch sign-approvers --key PATH --epoch N ...`                   | nothing                                 | a current approver's signature over a new approver set at its epoch, `N` one above the published set's (c6214dc; see [changing the approver set](/docs/operations/pki#changing-the-approver-set)) |

```bash
better-compute batch approve --tenant acme --batch batch_01k5w… --key ~/.better-compute/approver.pem
# Batch batch_01k5w…: reimage rack 12 after the firmware recall
# 12 command(s) on 12 subject(s): deploy.image ×12:
#   deploy.image: node_01…, node_02…, …
# Approvals: 0 of 2; the window closes at 2026-09-25T18:00:00.000Z.
# Not approved. Re-run with --yes, or with --root 3f1c….
better-compute batch approve --tenant acme --batch batch_01k5w… --key ~/.better-compute/approver.pem \
  --root 3f1c… --expect rack12.txt
```

`batch approve` signs only what it showed you. Without `--yes` it stops after the summary; `--root` approves only if
the batch still has the root you reviewed. `--expect FILE` (one subject per line, or `{ subject, kind }` commands)
approves only a batch that holds exactly those. Leaves that admit any payload of their kind are refused unless
`--allow-unbound`, and a batch that stays open longer than `--max-hours` (default 72) is refused. The signature
covers the batch id, root, size, expiry and the deployment's trust domain. On an offline machine, write the scope
with `batch show --scope-out batch.json`, carry it over, run `batch sign --key … --scope batch.json` there (a bare
`--root` and `--size` needs `--unchecked`), and bring the output back to `batch approve --signature`.

A batch an agent proposed for a person (MCP's `propose_command_batch`) carries the agent's own reason. `batch
approve` then notes `Proposed by agent X for Y (delegation D): the reason is the agent's own text.` after the reason
line, and `batch show` returns the same words as `proposedBy`; for a person's batch `proposedBy` is just their
identity (bd80209). See [the destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10).

Approver key files readable by other users are refused, as ssh does. Every refusal has its own exit code:

| Code               | Meaning                                                                                                                   |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `SCOPE_MISMATCH`   | the listed leaves do not give the root and size the server asks you to sign, or a `--signature` file signs something else |
| `SCOPE_CHANGED`    | the batch no longer has the `--root` you reviewed                                                                         |
| `SCOPE_UNEXPECTED` | the batch is not what `--expect` lists; the message names the differences                                                 |
| `UNBOUND_SCOPE`    | leaves admit any payload and `--allow-unbound` was not given                                                              |
| `INVALID_RESPONSE` | the server answered a scope page that is malformed or does not advance                                                    |
| `INVALID_STATE`    | the batch is not pending, its window is closed or longer than `--max-hours`, or the key file is readable by others        |

See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) for how batches are created, held and
checked on each machine.

## Audit [#audit]

The audit commands are operator commands: they read `audit_events` and the checkpoints straight from the
database through your configuration, not through the API.

```bash
better-compute audit verify --tenant acme                  # recompute every checkpoint; AUDIT_TAMPERED on any problem
better-compute audit verify --tenant acme --query problems
better-compute audit prove aud_01k… --tenant acme --out proof.json
better-compute audit check-proof proof.json --pin sha256/…  # offline, no database
better-compute audit checkpoint                            # seal now, as the hourly job does
```

`verify` and `check-proof` check signatures under the audit key's pin: `--pin`, `BETTER_COMPUTE_AUDIT_PIN`, or the
public half of the PKI bundle (`--pki`, default `.better-compute/pki.json`). `checkpoint` signs, so it needs the
bundle's private audit key, unsealed with `BETTER_COMPUTE_SECRET`. Anyone you hand a proof file and the pin to can
run `check-proof` without access to your deployment. See [audit](/docs/guides/events/audit).

## Operator commands [#operator-commands]

The rest of the CLI runs and maintains a deployment. Briefly:

| Command                                                                                      | Does                                                                                                                                                                                      |
| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `init`                                                                                       | writes a starter `better-compute.config.ts`                                                                                                                                               |
| `migrate`                                                                                    | creates or upgrades the schema and every module's migrations; run it on every deploy                                                                                                      |
| `doctor [--strict]`                                                                          | checks the database, schema, clock, method registry, IAM wiring, the PKI bundle and the host export                                                                                       |
| `serve`                                                                                      | runs the API and the worker roles (`--roles api`, `controllers,sinks`, `gateway`, …)                                                                                                      |
| `dev [--sim N]`                                                                              | a local server with PGlite and, with `--sim`, N simulated machines and an embedded Better IAM                                                                                             |
| `sim`                                                                                        | the deterministic fleet simulator and its invariants                                                                                                                                      |
| `pki init / show / issue-gateway / renew-ca / rotate-command-root / add-sandbox-ca / reseal` | the machine PKI bundle, its pins, gateway certificates, CA renewal and rotation, the sandbox SSH CA, re-sealing in the current format; see [PKI](/docs/operations/pki)                    |
| `daemon install-script`                                                                      | the POSIX installer that joins a server to a pool; `--plugins` prepares a node for bcx driver plugins (see [installing bcd](/docs/daemon/install#the-plugin-drop-in))                     |
| `iam setup [--meters]`                                                                       | prints the example policies; creates the billing meters in Better IAM                                                                                                                     |
| `release keygen / sign / cosign / verify`                                                    | signed bcd release manifests                                                                                                                                                              |
| `batch keygen / sign / sign-approvers`                                                       | an approver's offline tools: a P-256 approver key, an approval signed from a batch's scope file, and a signature over a new approver set; see [destructive batches](#destructive-batches) |

See [installation](/docs/guides/installation), [local development](/docs/operations/local-development) and the
[CLI reference](/docs/reference/cli).

## Exit codes and common errors [#exit-codes-and-common-errors]

The binary exits 0 on success and 2 for a usage mistake (`INVALID_ARGUMENT`, `INVALID_COMMAND`). Other failures
exit non-zero with the error's code:

| Code                                                                                       | Meaning                                                                                                                                                                                                                 |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MISSING_ENV`                                                                              | a token command with no credential: no `BETTER_COMPUTE_TOKEN` and no saved session or profile                                                                                                                           |
| `CONFIG_NOT_FOUND`                                                                         | no configuration found, and no `--url` or `BETTER_COMPUTE_DATABASE_URL`                                                                                                                                                 |
| `UNAUTHENTICATED`, `ACCESS_DENIED`, `CONFIRMATION_REQUIRED`, …                             | the API's own codes, from the server; see [errors](/docs/reference/errors). `TENANT_SUSPENDED` also prints a hint: the tenant is suspended or deleted in IAM, and an administrator of a parent tenant can reactivate it |
| `FLEET_DRIFTED`, `PLAN_BLOCKED`, `CONFIG_DRIFT`                                            | fleet plan, status and apply outcomes                                                                                                                                                                                   |
| `AUDIT_TAMPERED`, `INVALID_PROOF`, `AUDIT_KEY_MISSING`, `PKI_NOT_FOUND`                    | audit verification outcomes                                                                                                                                                                                             |
| `DOCTOR_FINDINGS`                                                                          | `doctor` found an error, or `doctor --strict` an error or a warning; the report is printed first                                                                                                                        |
| `ALREADY_EXISTS`                                                                           | an `--out` or `--secret-out` file exists; a token, secret or export is never written over a file. The file is reserved before the API call, so nothing was minted or rotated                                            |
| `FILE_NOT_WRITABLE`                                                                        | an `--out` or `--secret-out` file for a one-time secret cannot be created (no such directory, no permission); refused before the API call                                                                               |
| `SCOPE_MISMATCH`, `SCOPE_CHANGED`, `SCOPE_UNEXPECTED`, `UNBOUND_SCOPE`, `INVALID_RESPONSE` | `batch show`, `approve` and `sign` refused a batch; see [destructive batches](#destructive-batches)                                                                                                                     |
| `REMOTE_EXIT`                                                                              | `console`, `exec`, `sandbox exec` or `ssh-cert --connect`: the binary exits with the remote program's own status (1 to 255), or 255 when the tunnel broke (or ssh was ended by a signal)                                |

## Project commands [#project-commands]

A configuration can export its own commands, which get the same flag parsing, help, configuration loading and
output formatting as the built-in ones:

```ts title="better-compute.config.ts"
import { defineCommand } from 'better-compute/cli';

export const commands = [
  defineCommand({
    name: 'seed',
    summary: 'Create the demo sites and pools',
    target: 'config',
    flags: { region: { type: 'string', description: 'Region label', default: 'eu' } },
    async run({ compute, flags }) {
      const instance = await compute();
      // …
      return { created: true, region: flags.region };
    },
  }),
];
```

A `token` project command gets `context.api()`, the same transport the built-in commands use, so it is decided as
the resolved credential like any other call.

## Inside another CLI [#inside-another-cli]

PLAN.md section 9.3 asks for one binary for IAM and compute. `computeCommands(define, options?)` from
`better-compute/cli` turns every built-in command into a spec of another framework's `defineCommand`; with Better
IAM's:

```ts title="bin.ts"
import { createCli, defineCommand } from '@better-iam/cli';
import { computeCommands } from 'better-compute/cli';

const cli = createCli({ commands: computeCommands(defineCommand, { exclude: ['dev', 'serve'] }) });
await cli.run(process.argv.slice(2)); // better-iam compute-reserve …, compute-api …, compute-support-bundle …
```

Names are `compute-<command>` by default (`prefix`, and `separator: ' '` for a framework that matches several
words), in a `Compute` help group. Each spec declares compute's own flags plus `--config` and `--url`, and the host
adds its output flags; running it runs compute's program with the flags given, so compute reads its own
`BETTER_COMPUTE_*` environment, loads its deployment and prints its own output, errors included. `exclude` leaves
commands out, such as `dev` and `serve` for a client-only binary.

## Run commands from code [#run-commands-from-code]

`runCli(argv, io?)` runs one command line and rejects with the failure's `ComputeError`, `CliError` or
`ComputeClientError`; `createCli(options?)` returns a program whose `run(argv, io?)` does the same with your own
commands added. `main(argv?, cli?, io?)` behaves like the binary: it prints `CODE: message` on failure and resolves with
the exit status, 0, 2 for a usage mistake, 1 otherwise, and the remote program's own status when a `console`, `exec`,
`sandbox exec` or `ssh-cert --connect` ends with one (a `RemoteExitError`, whose `exitStatus` is 1 to 255, or 255 when the tunnel broke).

Everything a command touches comes through a `CliIO`, so programs and tests can run commands without a real terminal:

| Field                           | Used for                                                                                                                              | Default (`processIO()`)                                        |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `out(message)`, `err?(message)` | results on stdout; progress and failures on stderr                                                                                    | `console.log`, `console.error`                                 |
| `env`                           | the environment commands read; `process.env` is never read implicitly                                                                 | `process.env`                                                  |
| `stdin?()`, `cwd?`, `fetch?`    | `--data -`, relative paths and configuration discovery, remote calls                                                                  | the process's                                                  |
| `signal?`                       | an `AbortSignal` that ends long-running commands (`serve`, `dev`, `forward`) and detaches `console` and `exec`                        | SIGINT or SIGTERM                                              |
| `terminal?`                     | the interactive terminal `console` and `exec` attach to: `onInput`, `write(data, channel)`, and optional `size`, `onResize` and `raw` | the process's stdin and stdout, raw mode on a TTY              |
| `spawn?(command, args)`         | runs a program with inherited stdio and resolves with its exit status; `sandbox exec` and `ssh-cert --connect` run OpenSSH through it | `child_process.spawn`; a program ended by a signal reports 255 |

When you pass your own `CliIO` without `terminal`, `console` and `exec` fail with `UNSUPPORTED` before a ticket is
spent; without `spawn`, `sandbox exec` and `ssh-cert --connect` fail with `UNSUPPORTED` after the grant or the
certificate, since they cannot run ssh.

```ts title="run.ts"
import { RemoteExitError, runCli } from 'better-compute/cli';

try {
  await runCli(['sandbox', 'exec', sandboxId, '--', 'make', 'test'], {
    out: (line) => log.push(line),
    env: { ...process.env, BETTER_COMPUTE_URL: 'https://compute.example.com' },
    spawn: (command, args) => runWithInheritedStdio(command, args), // resolves with the exit status
  });
} catch (error) {
  if (error instanceof RemoteExitError) process.exitCode = error.exitStatus; // the remote command failed
  else throw error;
}
```


# Console (/docs/guides/clients/console)

> The Next.js console runs Better IAM and Better Compute on one origin, built on console-kit, framework-free stores with live updates and permission-aware actions.



The console is a web UI for people: tenants reserve GPUs, create instances and open terminals; platform operators
watch nodes, rollouts and fleet drift. It comes in two layers you can use separately:

* **`apps/console`**, a Next.js 15 app that runs Better IAM and Better Compute in one process, on one origin. It is
  private to the repository (not published) and serves as the reference deployment.
* **`@better-compute/console-kit`**, the framework-free view models the console's pages are built on: resource
  stores, the live event stream, permission-aware actions, forms, charts and a terminal. Use it to build your own
  UI in React, Solid, Vue or anything else.

> **Ahead of its phase.** 
  PLAN.md schedules the console for v0.5. Both layers are written and tested today (`tests/console-kit*.test.ts`),
  but the console's configuration is a development-first reference, not a hardened product.

## One origin, one sign-in [#one-origin-one-sign-in]

<Mermaid
  chart="flowchart LR
  B[&#x22;Browser&#x22;]
  subgraph P[&#x22;node server.mjs (one process)&#x22;]
    N[&#x22;Next.js pages&#x22;]
    IA[&#x22;/api/iam: Better IAM&#x22;]
    CA[&#x22;/api/compute: Better Compute&#x22;]
    W[&#x22;/api/compute/watch: SSE&#x22;]
    T[&#x22;/api/compute/tunnels/{ticket}: WebSocket&#x22;]
  end
  G[&#x22;gateway&#x22;]
  B -->|&#x22;IAM session cookie&#x22;| N
  B --> IA
  B --> CA
  B --> W
  B --> T
  T -->|&#x22;mTLS relay&#x22;| G"
/>

Better IAM is mounted at `/api/iam` and Better Compute at `/api/compute` on the same origin, so the host-only IAM
session cookie authenticates both. Compute is built with `acceptIamCookie: true`: it accepts the cookie only on
POSTs that carry `X-Better-Compute: 1` and an `Origin` equal to the console's own. The compute IAM bridge installs
compute's action catalog in Better IAM and serves compute's decisions from its projection, as in any
[embedded](/docs/guides/iam#wire-embedded-mode) deployment.

`server.mjs` runs Next.js plus the one thing a route handler cannot do: the WebSocket upgrade of
`/api/compute/tunnels/{ticket}` for browser terminals and node consoles, served by `handleTunnel` and relayed over
mTLS to the gateway that holds the node's session. Tunnels open only from the console's own origin, and a signed-in
browser can only use its own person's tickets.

## Run it [#run-it]

```bash
cd apps/console
node server.mjs --dev      # or: pnpm dev
```

`--dev` sets `CONSOLE_DEV=1`, the only mode in which the console generates secrets for itself: the session secret,
the exec ticket key, the break-glass and sandbox SSH CA keys (under `.console-data/`) and, on the first start, the root administrator's
password, printed once. It also allows gateways without TLS. Open `http://localhost:3100/login` and sign in as
`admin@console.local`. Members of an organization sign in at `/login?org=<slug>`.

For production, build and start without `--dev`; every secret then comes from the environment:

```bash
cd apps/console
pnpm build                   # next build
CONSOLE_ORIGIN=https://console.example.com CONSOLE_SECRET=… CONSOLE_ADMIN_PASSWORD=… \
BETTER_COMPUTE_DATABASE_URL=postgres://… CONSOLE_IAM_DATABASE_URL=postgres://… \
node server.mjs --prod       # or: pnpm start
```

Run it from `apps/console`: the stack file and the data directory are resolved against the working directory.

| Variable                                                                     | With `--dev`                                      | Otherwise                                                                                                                                         |
| ---------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONSOLE_ORIGIN`                                                             | `http://localhost:3100`                           | set it to the public origin: cookie, origin and tunnel checks compare against it                                                                  |
| `CONSOLE_SECRET`                                                             | generated into `.console-data/secret`             | required, at least 32 random characters                                                                                                           |
| `CONSOLE_DATA_DIR`                                                           | `.console-data`                                   | the same                                                                                                                                          |
| `CONSOLE_IAM_DATABASE_URL`                                                   | a libSQL file, `.console-data/iam.db`             | PostgreSQL for Better IAM when set                                                                                                                |
| `BETTER_COMPUTE_DATABASE_URL`                                                | PGlite in `.console-data/compute`                 | `postgres://…` for compute                                                                                                                        |
| `CONSOLE_ADMIN_EMAIL`, `CONSOLE_ADMIN_PASSWORD`                              | `admin@console.local`, a generated password       | the password is required on the first start                                                                                                       |
| `CONSOLE_EXEC_TICKET_KEY_FILE`                                               | a generated `exec-ticket.pem`                     | unset: tunnels are off                                                                                                                            |
| `CONSOLE_SSH_CA_KEY_FILE`                                                    | a generated `ssh-ca.pem`                          | unset: no break-glass SSH                                                                                                                         |
| `CONSOLE_SANDBOX_SSH_CA_KEY_FILE`                                            | a generated `sandbox-ssh-ca.pem` (Ed25519)        | unset: no `sandboxes.exec` (`UNSUPPORTED`), and sandbox guests trust no CA                                                                        |
| `CONSOLE_GATEWAY_TLS_CERT`, `_KEY`, `_CA`                                    | none needed (h2c gateways)                        | all three or none; tunnels need them                                                                                                              |
| `CONSOLE_AUDIT_KEY_FILE`                                                     | a generated `audit-checkpoint.pem` (P-256 PKCS#8) | unset: no audit checkpoints; every replica shares the key                                                                                         |
| `CONSOLE_RELEASE_PINS`, `CONSOLE_DOWNLOADS_URL`, `CONSOLE_RELEASE_THRESHOLD` | unset: no rollout module                          | set the first two to mount rollouts                                                                                                               |
| `CONSOLE_GATEWAY_URL`, `CONSOLE_GATEWAY_SERVER_NAME`                         | unset: no `gateway` section                       | published in `/.well-known/better-compute` as `gateway` for `bcd join --server`; the console has no PKI bundle, so no serving-CA pin is published |
| `CONSOLE_CONFIG`                                                             | `console.config.mjs`                              | another stack file                                                                                                                                |
| `PORT`, `HOSTNAME`                                                           | `3100`, `localhost`                               | the same                                                                                                                                          |

Every replica and gateway must share the exec ticket key, and nodes pin its public key and the SSH CA's, so
production supplies the key files rather than generating them. The sandbox SSH CA is a separate key, so a sandbox
certificate never opens a node and a break-glass one never opens a sandbox; the console passes its public key to
`vps({ sandboxExec })` so every sandbox guest trusts it, and certifies each guest's host key with it
(`certifyHostKey`), so `sandbox exec` against the console pins guests instead of trusting them on first use. Every
replica must share it too. The stack is built once per
process from `console.config.mjs` (loaded at run time, never bundled): Better IAM, then compute with the `vps`, `fleet` and
`events` (webhooks) modules and `retentionModule()` (the console builds `betterCompute()` itself, so the CLI's
automatic retention does not apply), `audit` when an audit key is available, and `rollout` when release pins are set.

Before the workers start, the stack calls `installIamHandlers(compute, { bridge, streams: [watches] })`, as `serve`
does for `host.iam`: a revoked delegation or suspended agent closes its tunnels on every gateway (over the console's
gateway mTLS) and its `/api/compute/watch` streams in this process, and revocations drop the IAM caches. See
[tunnels](/docs/guides/access/tunnels#closing-on-iam-revocation).

## Screens [#screens]

Every page needs a session; without one the layout redirects to `/login`. Pages render in the browser over
console-kit, for the tenant the session acts in. Platform pages appear only for the root tenant.

| Page               | Path                                                                                   | Shows and does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------ | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Overview           | `/`                                                                                    | capacity per pool, quota alerts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Reservations       | `/reservations`, `/reservations/[id]`                                                  | the form with a capacity preview, the earliest start and approvals, optional host CPUs and memory (GiB), and a Kubernetes checkbox only for a pool the tenant owns with `policy.kubernetes` (4533265); the list and each reservation's actions                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Instances          | `/instances`, `/instances/[id]`, `/instances/[id]/terminal`, `/instances/[id]/console` | the form with sizes per pool; list, detail, a browser terminal (op `exec`) and the instance's serial console (op `console`, 191898b), linked as **Open a terminal** and **Serial console**, both decided as `compute:instances:exec`. bcd refuses instance `exec` until guests have an agent, so the serial console is the terminal that works on real hosts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Sandboxes          | `/sandboxes`                                                                           | a create form over the tenant's templates (each with its flavor and warm pool size) and a lifetime in minutes, capped at the template's maximum; the person's own sandboxes, or everyone's, with state, a countdown to expiry (a warning in the last 5 minutes), **+30 min**, **Snapshot** and **Delete**, and the `better-compute sandbox exec` line for a shell; templates, with their volumes (attach and detach, dd5555b)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Volumes            | `/volumes`                                                                             | [sandbox volumes](/docs/guides/vps/sandboxes#volumes) (50add82): a create form (pool, name, size in GiB); the person's own volumes, or everyone's (e68aba7), with id, device path, pool, host, size and state; **Grow** and **Delete** (below)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Nodes              | `/nodes`, `/nodes/[id]`, `/nodes/[id]/console`                                         | a filterable table; conditions, cordon, uncordon and drain, the node's labels (key, value, source) and taints with forms to set or remove an operator label and add or remove a taint (`nodes.label`, `nodes.taint`, gated like the other node actions on `compute:nodes:update`; keys, values and taints are checked against the API's patterns before the button enables; 3ff9fb3), recent commands, the node console, and the break-glass SSH panel (below). A node in a bare-metal provision state (not `adopting` or `adopted`) also gets a **Lifecycle** panel (0cd1d79): `manage`, `inspect`, `provide`, `rescue`, `unrescue`, `abort`, `recover`, `undeploy` and `retire`, each offered as the provision state machine and IAM (`compute:nodes:provision`) allow; deploy and power stay in the CLI (`metal deploy`, `metal power`), and PXE MAC confirmation has its own page (below) |
| PXE discovery      | `/discovery`                                                                           | a site picker and what the site's agents saw PXE-booting from MACs with no boot entry (`metal.discovered`): MAC and request count, product UUID, arch and relay, last seen, and what each row is (a candidate for a machine, not enrolled, or an enrolled node); **Confirm** on a candidate and a form to bind a MAC by hand (`metal.confirmMac`), each behind a browser confirmation (15c11ee; below)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Batches            | `/batches`                                                                             | destructive batches held for the approver quorum: pending by default (or approved, expired, cancelled, all), each with its reason, command count, approvals of required, state and the window left; a batch's scope by kind, who approved, the `better-compute batch approve` line, and **Cancel** (below)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Pools & capacity   | `/pools`, `/pools/[id]`                                                                | pools owned or offered, a 7-day booking calendar, join tokens and join requests                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Quotas             | `/quotas`                                                                              | used of max per pool and resource                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Usage & spend      | `/usage`                                                                               | usage per meter, day, project and person, with a CSV export                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Activity           | `/activity`                                                                            | the live event stream: the last 300 events this session received, filterable by type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Webhooks           | `/webhooks`                                                                            | the tenant's endpoints: create one (URL, event patterns, description) with its signing secret shown once to copy, pause and resume, ping, rotate the secret with a 1 h overlap, recent deliveries with redelivery of dead ones, delete                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Rollouts (root)    | `/admin/rollouts`                                                                      | daemon rollouts wave by wave; pause, resume and abort                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Fleet drift (root) | `/admin/fleet`                                                                         | drift of the fleet from its applied FleetConfig                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

A "live" badge in the navigation shows the event stream's state. While the stream is down, the stores a page has
loaded refresh every 20 seconds instead, and only while the tab is visible.

Buttons follow two checks. A button is enabled only when the item's state allows the action (a released
reservation offers nothing) and Better IAM would allow it for the signed-in person. The second check is advisory: a
server action batches up to 50 checks through Better IAM's `can`, and the compute API still decides the real call.
A delegated call held back for approval shows "awaiting approval" with the confirmation id.

Actions that destroy or release something ask first, naming what happens: draining a node, stopping or deleting an
instance, releasing or cancelling a reservation (d40af29, 0919f86), a bare-metal node's `undeploy`, `retire`,
`abort` and `provide` (0cd1d79), and binding a machine's PXE MAC, after which it boots from that MAC (15c11ee).

### Sandboxes [#sandboxes]

The **Sandboxes** page (`createSandboxesView` in console-kit) works on [sandboxes](/docs/guides/vps/sandboxes) of the
session's tenant. With no template, the create form says an administrator creates them. Create sends `sandboxes.create` with a fresh `console-…` idempotency key and waits up to 20 seconds
for the sandbox to leave `starting`. The list shows the person's own sandboxes, or, with **Everyone's sandboxes**,
the tenant's (`scope: 'all'`), each with its template name, `warm` and the snapshot it was restored from. A memory
restore whose `identityRebound` is `false` (console-kit's `SandboxRow` carries it) shows a warning that it still
answers to its source's SSH identity, so exec waits until the source and sibling restores are gone or the host gives
it its own (9958865). The countdown ticks every second from `expiresAt` and reads `expired` at zero.

Its buttons are not gated by IAM checks in advance: the call is decided and an error is shown in place. **+30 min**
is `sandboxes.extend` with `by` of 30 minutes, **Snapshot** takes a memory snapshot that goes with the sandbox, and
**Delete** asks for confirmation first. A shell is not opened in the browser: the page shows the
`better-compute sandbox exec sbx_…` command, which runs OpenSSH through a tunnel. The console's own stack configures
the sandbox SSH CA but no snapshot store, so **Snapshot** answers `INVALID_CONFIG` there until you give
`vps({ snapshots })` a store in `console.config.mjs`. On that answer the page says snapshots are not configured on
this deployment and hides the button for the rest of the session.

A **Templates** section serves tenant administrators (`compute:sandboxes:manage`; the calls are decided, not the
section hidden). It lists the tenant's templates with pool, flavor, image, lifetime and warm pool, with **Warm +1**,
**Warm −1** (`sandboxTemplates.update` of `warm`) and delete, and a form that creates one from a name, pool, flavor,
image, warm minimum and longest lifetime in hours. console-kit's `SandboxesView` exposes the same as
`createTemplate`, `updateTemplate` and `deleteTemplate`. Templates can also come from
`better-compute sandbox-template create` or a FleetConfig document.

Since dd5555b the templates table also shows each template's [volumes](/docs/guides/vps/sandboxes#volumes) (volume
name, mount path, `(read-only)`), each with **Detach**, and an **Attach a volume** form picks a template, a `ready`
volume of that template's pool, a mount path (default `/data`; it must start with `/`) and **Read-only**. Every
sandbox from the template then mounts the volume and runs on the volume's host. Both send the template's whole volume
list through `sandboxTemplates.update`. **Warm +1** is disabled for a template with volumes, since the server keeps
no warm pool beside volumes. The volume picker lists everyone's volumes (`sandboxVolumes.list` with `scope: 'all'`);
without the right to list them it is empty.

### Volumes [#volumes]

The **Volumes** page (50add82) works on the session tenant's [sandbox volumes](/docs/guides/vps/sandboxes#volumes),
an agent's persistent memory, through the API directly (`sandboxVolumes.*`; there is no console-kit view for it).
The create form takes a pool from the tenant's pools, a name (lowercase letters, digits, `.`, `_` and `-`, at most 63,
checked before the button enables) and a size from 1 to 16,384 GiB, and sends `sandboxVolumes.create` with a fresh
`console-…` idempotency key. The list shows the person's own volumes (`scope: 'mine'`), or, with
**Everyone's volumes**, the tenant's (`scope: 'all'`, e68aba7), each with its id and
`/dev/disk/by-id/virtio-<serial>` path, pool and host ("not placed yet" without one), size and state. For a `ready`
volume, **Grow** asks for the new size and calls `sandboxVolumes.resize` only with a larger one, and **Delete** asks
for confirmation, warning that the data is wiped, before `sandboxVolumes.delete`. As on the Sandboxes page, the
buttons are not gated by IAM checks in advance; the call is decided and an error is shown in place. A volume is
attached to a template on the Sandboxes page's **Templates** section (dd5555b, above), or with
`better-compute sandbox-template update --volumes`, the API or FleetConfig.

### PXE discovery [#pxe-discovery]

The **PXE discovery** page (`/discovery`, 15c11ee) is the console's side of
[discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines): a machine whose BMC cannot name
its NICs (IPMI), enrolled with only its product UUID, waits in `metal.ensureRamdisk` until an operator confirms its
PXE MAC. The page is built on console-kit's `createPxeDiscoveryView`.

A site picker lists the session tenant's sites (`sites/list`, the first one selected), and the table shows what that
site's agents saw PXE-booting from MACs with no boot entry in the last 24 hours ([`metal.discovered`](/docs/reference/api/metal#discovered);
only the tenant that owns the site sees it). **Refresh** reloads it. `pxeCandidates` orders the rows by what needs a
decision first: candidates, then unknown machines, then enrolled ones, newest first within each.

| Column       | Shows                                                                                                                   |
| ------------ | ----------------------------------------------------------------------------------------------------------------------- |
| MAC          | the MAC, and how many requests it sent                                                                                  |
| Product UUID | the SMBIOS UUID from DHCP, or a dash                                                                                    |
| Arch / relay | the client architecture, and `via <relays>` for a relayed request or `agent segment` for one on the agent's own segment |
| Last seen    | `just now`, `N min ago` or `N h ago` (`seenAgo`)                                                                        |
| Machine      | what the row is to the operator: `candidate for <node>`, `not enrolled`, or the enrolled node's id                      |

Nothing on the page binds a MAC by itself, since discovery data is unauthenticated DHCP. A candidate row (the MAC
metal saw under the machine's UUID right after power-cycling it) has a **Confirm** button. It first asks, in a browser
confirmation, "Bind `<mac>` as the PXE MAC of `<node>`? The machine will then boot from it.", and only then calls
[`metal.confirmMac`](/docs/reference/api/metal#confirmmac) and reloads the list. Unknown and enrolled rows have no
button: an unknown machine is never answered or enrolled from here.

**Bind a MAC by hand** serves a machine enrolled without MACs whose PXE MAC you know from its label or its BMC. It
takes a node id (it must start with `node_`) and a MAC; `normalizeMac` accepts upper or lower case and `-` or `:` as
the separator, and the button stays disabled until both are well formed. It goes through the same browser
confirmation and `metal.confirmMac` call. As with the CLI's `metal confirm-mac`, the call is decided as
`compute:nodes:provision` on the node and audited as `node.pxe-mac.confirmed`; the buttons are not gated by IAM
checks in advance, so a refusal (for example `CONFLICT` for a MAC another machine of the site has, or
`INVALID_STATE` for a node that already has MACs) is shown in place.

### Destructive batches [#destructive-batches]

The **Batches** page (`createBatchesView` in console-kit, b24d043) shows the tenant's
[destructive batches](/docs/operations/pki#the-destructive-batch-quorum-v10): fleet-wide reimages, wipes and power
actions held until enough approvers sign their scope. It lists pending batches by default, with a filter for
`approved`, `expired`, `cancelled` or all, and counts down each pending batch's window from `notAfter`. Each row shows
its reason and who created it; a batch an agent proposed shows a warning line instead, "Proposed by agent X for Y
(delegation D): the reason is the agent's own text" (bd80209, from `createdByAgent` and `delegationId`).

**Details** loads the batch (`commandBatches.get`) and its whole scope (`commandBatches.scope`, paged 5,000 at a time
and capped at 20,000 entries, with a warning when cut short), grouped by command kind. Subjects read as a person
would: a node id, `BMC 10.0.0.7 (via node_site)` for a BMC reached through a site agent, or `HTTP to node_…`. The
panel also shows the scope root, the approvals so far (key id, approver and time) and the member commands by state.
**Cancel** asks for confirmation and calls `commandBatches.cancel`, so none of the batch's commands run.

The page cannot approve. A browser never holds an approver key, so for a pending batch it shows the command for the
approver's terminal, `better-compute batch approve --batch <id> --key <approver key>`, where the CLI lists the scope,
recomputes the root from it and signs. Reading needs `compute:batches:read` and cancelling `compute:batches:approve`,
both decided when called.

### Break-glass SSH [#break-glass-ssh]

A node's page carries the **Break-glass SSH** panel for [SSH certificates](/docs/guides/access/ssh), for platform
administrators. It takes your public key (a pasted private key, several lines or an unsupported key type is refused
before anything is sent), the account (default `root`) and a lifetime of 5, 15, 30 or 60 minutes, and issues a
certificate with `nodes.issueSshCert`; the certificate is shown once, with a copy button and the commands to save it
as `<key>-cert.pub` and connect. A session without MFA is told to sign in again with a second factor.

Below it, **Who holds access** lists the node's certificates from `nodes.listSshCerts`: holder, principal, issue
and expiry times, and state (`active`, `expired` or `revoked`, with the revocation reason), with **Include expired**
to show those too. Each active certificate has a **Revoke** button, which asks for confirmation and calls
`nodes.revokeSshCert` with the reason `revoked from the console`.

## console-kit [#console-kit]

`@better-compute/console-kit` depends only on the typed client and core. Every store has `subscribe(listener)` and
`get()`, the contract of React's `useSyncExternalStore`, so a framework adapter is a few lines.

```ts
import { createComputeClient } from '@better-compute/client';
import { createPermissions, describeAction, startLiveConsole } from '@better-compute/console-kit';

const client = createComputeClient({ baseURL, token: () => session.token });

const live = await startLiveConsole(client, {
  tenantId,
  watchUrl: `${baseURL}/watch`, // wherever your host mounts handleWatch
  headers: () => ({ authorization: `Bearer ${session.token}` }),
  permissions: createPermissions({ authorize: (checks) => iamAuthorizeMany(checks) }),
});

live.reservations.subscribe(() => render(live.reservations.get()));
live.connection.subscribe(() => showBadge(live.connection.get().status));
```

| Area            | Exports                                                                                                 | Does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| --------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Stores          | `createConsoleResources`, `createListResource`, `createValueResource`, `createStore`                    | `nodes`, `pools`, `reservations`, `instances` and `capacity` for one tenant; lists page with `loadMore()`, a stale response never overwrites a newer one, and an error keeps the last good items                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Live            | `watchEvents`, `startLiveConsole`, `matchesType`, `parseSseBlock`                                       | Server-Sent Events over `fetch`, resumed with `Last-Event-ID`; each event re-reads the item it names                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Actions         | `describeAction`, `actionsFor`, `createPermissions`                                                     | map a button to its IAM action and resource from core's API surface; batch checks (50 per call) in one tick and cache answers for 10 s                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Detail pages    | `createDetail`, `applicable`                                                                            | one reservation, node, instance or pool with each action gated by state and by IAM; `run(path)` answers `done` or `confirmation-required`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Forms           | `createReservationForm`, `validateReservation`, `createInstanceForm`, `validateInstance`                | validation that mirrors the server, a debounced `capacity.preview`, one idempotency key per attempt kept across retries and confirmations. Since e833433 the reservation form also takes `vcpu` (0 to 4096) and `memoryGiB` (0 to 65,536), sent as `resources` (left out: the pool's per-GPU share), and `kubernetes` for `binding: 'kubernetes'`, and the preview books them too. The console app's reservation page shows them since 4533265, with the Kubernetes checkbox only for a pool the tenant owns that has `policy.kubernetes`                                                                                                                                          |
| Views           | `capacityChart`, `quotaView`, `createQuotaView`, `timelineView`, `earliestFit`, `usageView`, `usageCsv` | stacked capacity bars, quota meters (warning from 80 %), the booking calendar and earliest free window, usage and estimated spend                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Admin           | `createRolloutView`, `rolloutProgress`, `createFleetDriftView`, `nodeTable`                             | rollout waves, fleet drift by kind, a filterable, stably sorted node table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Sandboxes       | `createSandboxesView`, `timeLeft`, `sandboxExecCommand`, `withTemplateVolume`, `maxTemplateVolumes`     | a tenant's sandboxes (`scope` mine or all) and templates, `detail`, `create`, `extend`, `snapshot` and `remove`, each refreshing the list; the remaining lifetime as `12m 03s` or `1h 05m`, urgent in the last 5 minutes; the CLI line that opens a shell. Since dd5555b a `SandboxTemplateRow` carries its `volumes` (`volumeId`, `mountPath`, `readOnly`), `TemplateFields.volumes` replaces a template's whole list, `volumes()` lists the tenant's volumes a template can attach, and `withTemplateVolume(current, { add } or { remove })` returns the new list, refusing a second volume at the same mount path or more than `maxTemplateVolumes` (8) before anything is sent |
| Batches         | `createBatchesView`, `scopeByKind`, `describeSubject`, `batchApproveCommand`                            | a tenant's destructive batches over `commandBatches` (`filter(state or 'all')`, default `pending`), `detail`, `scope(batchId, max?)` (every leaf, paged, at most 20,000 by default, stopping on a page that does not advance; `truncated` when capped) and `cancel`; the scope grouped by kind with sorted subjects; a subject as text; the approver's CLI line                                                                                                                                                                                                                                                                                                                    |
| PXE discovery   | `createPxeDiscoveryView`, `pxeCandidates`, `seenAgo`, `normalizeMac`                                    | one site's `metal.discovered` list (`{ siteId, machines }`, each a `DiscoveredMachineRow`), `site(siteId)` to switch sites, and `confirmMac(nodeId, mac)`, which normalizes the MAC (throwing before any call for one that is not a MAC), calls `metal.confirmMac` and reloads the list; the rows as `{ kind, row }` with `kind` `candidate`, `unknown` or `enrolled`, in that order, newest first; a last-seen time as text; a MAC as the API takes it (lower case, colons) or `undefined` (15c11ee)                                                                                                                                                                              |
| SSH access      | `createSshAccessView`, `checkSshPublicKey`, `sshUsage`                                                  | one node's break-glass certificates: `issue` (held in `view.issued` until dismissed), `revoke`, `showExpired`; a public key check that refuses private keys; the commands to use a certificate                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Webhooks        | `createWebhooksView`, `deliverySummary`                                                                 | a tenant's webhooks over the `webhooks` group (list, create, pause, rotate, delete, ping, deliveries, redeliver, event types); the secret from create or rotate sits in `view.secret` until `dismissSecret()` and is never stored or re-read; delivery counts by state and the latest failure                                                                                                                                                                                                                                                                                                                                                                                      |
| Break-glass SSH | `createSshAccessView`, `checkSshPublicKey`, `sshUsage`                                                  | one node's certificates over `nodes.listSshCerts` (expired ones with `showExpired(true)`), `issue({ publicKey, account?, minutes? })` (1 to 60 minutes, 15 by default) and `revoke(serial, reason?)`; a pasted private key, a multi-line paste or a key type other than Ed25519 or ECDSA P-256 is refused before anything is sent; the issued certificate sits in `issued` until `dismissIssued()`, and `sshUsage` gives the commands to use it. The console's node page uses it                                                                                                                                                                                                   |
| Terminal        | `createTerminalSession`, `createScreen`, `keyInput`                                                     | `tunnels.open`, then the WebSocket (an instance with op `exec`, `ssh` or `console` for its serial console, or a node's console); a VT100/xterm screen model with scrollback, no DOM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| React           | `bindReact`, and `@better-compute/console-kit/react`                                                    | `useStore`, `useList`, `usePermission`, `useDetail`, `useAction`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

### Live updates [#live-updates]

`watchEvents({ url, types?, lastEventId?, headers?, onEvent, onStatus? })` reads the stream with `fetch` rather
than `EventSource`, so it can send an `Authorization` header (or, in the console, `X-Better-Compute: 1`). It
remembers the `id:` of the last event and sends it as `Last-Event-ID` when it reconnects, so it resumes exactly
after that event. It reconnects after the server's `retry:` delay (2 s by default) when the server closes the
stream, and with doubling backoff, capped at `maxBackoffMs` (60 s by default), after a failure. `onStatus` reports
`connecting`, `open`, `retrying` and `stopped`.

`applyEvent` routes each event by its subject: `reservation/…`, `node/…` and `instance/…` re-read that one item
(and drop it on `NOT_FOUND` or `ACCESS_DENIED`), and `pool/…` refreshes the pools. Reservation, pool and node events
(except `node.status`) also refresh the capacity summary, debounced by 250 ms. `startLiveConsole` does all of this
in one call and clears cached permission answers after events whose type starts with `iam.`, `binding.`,
`delegation.`, `agent.` or `project.members`. See [watch](/docs/guides/events/watch) for the server side.

### Permission-aware actions [#permission-aware-actions]

```ts
const release = describeAction('reservations', 'release', reservation.id);
// { path: 'reservations/release', action: 'compute:reservations:release',
//   resource: { type: 'compute-reservation', id: 'rsv_…' }, kind: 'mutation' }
const allowed = await permissions.can(release);
```

`describeAction` throws for a method outside the API table, and `actionsFor(kind, id)` lists every mutation on a
resource kind, optionally limited to the phases and modules your deployment runs. `createPermissions` takes your
`authorize(checks)`, usually Better IAM's `authorizeMany` for the caller's tenant behind a server endpoint. A failed
batch rejects its checks and leaves them unanswered; it never answers "denied", so a button stays disabled rather
than lying. The server still decides every call, so a stale "allowed" only ever ends in a clean `ACCESS_DENIED`.

### React [#react]

```tsx
import type { LiveConsole } from '@better-compute/console-kit';
import { useList } from '@better-compute/console-kit/react';

function Reservations({ live }: { live: LiveConsole }) {
  const { items, status } = useList(live.reservations);
  if (status === 'loading') return <p>Loading</p>;
  return <ul>{items.map((item) => <li key={item.id}>{item.id}: {String(item.state)}</li>)}</ul>;
}
```

The `/react` entry binds the hooks to the app's `react`, an optional peer (React 19). Apps that bring their own
React build, or `preact/compat`, call `bindReact(React)` from the main entry instead. `useList` and `useDetail` load
on first use; `usePermission` returns `undefined` while IAM answers (render the button disabled), then `true` or
`false`; `useAction(detail, path)` returns `run`, `pending`, and the `outcome` or `error`. The hooks use
`useSyncExternalStore` with the same snapshot on the server, so they render during SSR.

Related: [TypeScript client](/docs/guides/clients/typescript-client), [tunnels](/docs/guides/access/tunnels),
[delegated confirmation](/docs/guides/iam/delegated-confirmation).


# Clients (/docs/guides/clients)

> Every way to call Better Compute, from raw HTTP to the typed client, the CLI, MCP tools and the console, and the credentials each one carries.



Better Compute has one API. Every client, whether it is a shell script, the typed TypeScript client, the
`better-compute` CLI, an AI agent's MCP tool or the web console, ends up sending the same request to the same
route, and every call is authorized the same way: Better IAM decides one action on one resource, as the caller. No
client has a side door, and no client can do more than the credential it carries.

<Mermaid
  chart="flowchart LR
  subgraph callers[&#x22;Callers&#x22;]
    H[&#x22;curl / any HTTP client&#x22;]
    T[&#x22;TypeScript client&#x22;]
    C[&#x22;better-compute CLI&#x22;]
    M[&#x22;MCP tools&#x22;]
    K[&#x22;console (console-kit)&#x22;]
  end
  A[&#x22;POST /api/compute/{group}/{method}&#x22;]
  I[&#x22;Better IAM&#x22;]
  D[(&#x22;PostgreSQL&#x22;)]
  H --> A
  T --> A
  C --> A
  M -->|&#x22;as the caller&#x22;| T
  K --> T
  A -->|&#x22;one decision&#x22;| I
  A --> D"
/>

## The HTTP API [#the-http-api]

Every method is `POST {basePath}/{group}/{method}` with a JSON object body. The default base path is
`/api/compute`, so `reservations.create` is `POST /api/compute/reservations/create`. The generated
[API reference](/docs/reference/api) lists all groups and methods; plugin methods live under
`/api/compute/plugins/{id}/{group}/{method}`.

```bash
curl -X POST https://compute.example.com/api/compute/pools/list \
  -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Better-Compute: 1" \
  -d '{"tenantId":"acme"}'
# → { "data": { "items": [ … ] } }
```

The rules every request follows:

| Rule        | Detail                                                                                                                                                         |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Method      | `POST` for API calls. `GET` answers only `{basePath}/health` and `{basePath}/.well-known/better-compute`; anything else is `METHOD_NOT_ALLOWED`.               |
| Headers     | `Content-Type: application/json` and `X-Better-Compute: 1`, or the call fails with `CSRF_REJECTED` before anything is read.                                    |
| Body        | A JSON object, at most 64 KiB, checked against the method's input schema (`INVALID_INPUT` otherwise).                                                          |
| Answer      | `{ "data": … }` on success, `{ "error": { "code", "message", "details"? } }` on failure. See [errors](/docs/reference/errors).                                 |
| Request id  | Send `X-Request-Id` (up to 128 letters, digits, `.`, `_`, `:` or `-`) and the server echoes it and passes it to IAM; otherwise it makes one up and returns it. |
| Rate limits | A `RATE_LIMITED` answer carries `retryAfterMs` in the error body and a `Retry-After` header in seconds.                                                        |

`X-Better-Compute: 1` is the CSRF guard. A browser cannot send a custom header cross-origin without a CORS
preflight, and nothing in Better Compute approves one, so a forged form post from another site never reaches a
handler.

Before any database access, the server authenticates the caller and refuses a caller whose tenant is neither the
input's `tenantId` nor the root tenant, with the same `ACCESS_DENIED` whether or not the thing exists. Then it
decides once: see [how a call is decided](/docs/guides/iam/actions-and-resources#how-a-call-is-decided).

## Credentials [#credentials]

Better Compute authenticates Better IAM credentials only; it has no users, passwords or API keys of its own.

| Credential                                                                                           | How it is sent            | Who uses it                                                                        |
| ---------------------------------------------------------------------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------- |
| IAM bearer token (`biam_…`): a user session, an API key, a role session or a delegated agent session | `Authorization: Bearer …` | scripts, services, the CLI, agents                                                 |
| Session JWT from `sts.getSessionToken({ format: 'jwt', audience: [...] })`, with compute's audience  | `Authorization: Bearer …` | services that verify tokens offline, browser apps on another origin                |
| The IAM session cookie                                                                               | the browser's cookie      | only a console on the same origin, with `betterCompute({ acceptIamCookie: true })` |

The cookie counts only when `acceptIamCookie` is on and the request's `Origin` is the deployment's own origin: a
cookie request without `Origin` is `CSRF_REJECTED`, and one from another origin is `UNTRUSTED_ORIGIN`. Without the
option, compute drops the cookie before it calls IAM. [Credentials compute accepts](/docs/guides/iam#credentials-compute-accepts)
covers what each credential resolves to.

`system.version` and `system.whoami` are public: they answer without a credential, which makes them a quick check
that a URL points at a compute server.

## Ways to call [#ways-to-call]

| Client                                                      | Package                                       | Runs                                 | Credential                                     | Best for                             |
| ----------------------------------------------------------- | --------------------------------------------- | ------------------------------------ | ---------------------------------------------- | ------------------------------------ |
| HTTP                                                        | none                                          | anywhere                             | bearer token                                   | other languages, quick checks        |
| In-process API                                              | `@better-compute/server`                      | inside your server                   | `{ headers }` or `{ token }` per call          | your own backend that embeds compute |
| [TypeScript client](/docs/guides/clients/typescript-client) | `@better-compute/client`                      | browsers, Node.js 22+, edge runtimes | bearer token, or the cookie on the same origin | apps and services in TypeScript      |
| [CLI](/docs/guides/clients/cli)                             | `@better-compute/cli`                         | a terminal or CI                     | `BETTER_COMPUTE_TOKEN`                         | operators, tenants, CI drift checks  |
| [MCP](/docs/guides/clients/mcp)                             | `@better-compute/mcp`                         | an MCP host you run                  | the agent's bearer token, forwarded            | AI agents                            |
| [Console](/docs/guides/clients/console)                     | `apps/console`, `@better-compute/console-kit` | a Next.js app                        | the IAM session cookie                         | people in a browser                  |

### In process [#in-process]

A backend that builds the control plane with `betterCompute()` can skip HTTP. `compute.api.<group>.<method>` takes
the caller's credential first and the input second, and runs exactly the same envelope: the tenant pre-check, the
IAM decision, the transaction and the audit row.

```ts title="server.ts"
import { compute } from './compute';

const pools = await compute.api.pools.list({ headers: request.headers }, { tenantId: 'acme' });
```

Pass the incoming request's headers so that IAM sees the real caller. There is no "system" credential that skips
the decision.

### The typed client [#the-typed-client]

`createComputeClient<typeof compute>()` infers every group and method from your server's type, with a type-only
import, so there is no code generation. It is a small `fetch` wrapper that adds the headers above and turns error
envelopes into `ComputeClientError`. See [TypeScript client](/docs/guides/clients/typescript-client).

```ts
import { createComputeClient } from 'better-compute/client';
import type { compute } from './compute.server';

const client = createComputeClient<typeof compute>({
  baseURL: 'https://compute.example.com',
  token: () => session.getToken(),
});
const { items } = await client.pools.list({ tenantId: 'acme' });
```

### The CLI [#the-cli]

`better-compute` calls the API as `BETTER_COMPUTE_TOKEN`, either against a running server with `--url` or in process
through `better-compute.config.ts`. `better-compute api GROUP.METHOD key=value` reaches any method; `reserve`,
`capacity`, `fleet plan` and friends wrap the common ones. See [CLI](/docs/guides/clients/cli).

```bash
export BETTER_COMPUTE_URL=https://compute.example.com BETTER_COMPUTE_TOKEN=biam_…
better-compute api pools.list tenantId=acme --format table
```

### MCP [#mcp]

`createComputeMcp()` serves a set of tools (`list_pools`, `reserve_gpus`, `node_status` and more) as a Streamable
HTTP MCP endpoint behind Better IAM's MCP gate. Each tool calls the API with the agent's own token, and a reservation
that a person must approve comes back as `confirmation-required` with the idempotency key to retry with. See
[MCP](/docs/guides/clients/mcp).

### The console [#the-console]

`apps/console` runs Better IAM and Better Compute in one Next.js process on one origin, so the IAM session cookie
signs people in to both. Its pages are built on `@better-compute/console-kit`, framework-free stores with live
updates and permission-aware buttons that you can reuse in your own UI. See [console](/docs/guides/clients/console).

## Events [#events]

Clients that need to know when something changes do not poll list methods. The [events](/docs/guides/events)
section covers the three ways out of the outbox:

* `events.list` pages a tenant's history from a cursor, over the same API.
* [`GET /watch`](/docs/guides/events/watch) streams live events as Server-Sent Events. The host mounts it (the
  console serves it at `/api/compute/watch`), and console-kit's `watchEvents` reads it.
* [Webhooks](/docs/guides/events/webhooks) POST signed events to a tenant's HTTPS endpoints.

## Idempotency and retries [#idempotency-and-retries]

Methods that create things take an `idempotencyKey`. Keep the key for the life of one attempt and reuse it on
every retry, whether the retry follows a timeout, a `RATE_LIMITED` answer or a person's approval of a
`CONFIRMATION_REQUIRED` call. The same key with the same input returns the stored result; the same key with a
different input is `CONFLICT`. [Reservations](/docs/guides/capacity/reservations#drafts-and-idempotency) shows the
rules in full, and [delegated confirmation](/docs/guides/iam/delegated-confirmation) shows the approve-and-retry
loop.

  - [TypeScript client](/docs/guides/clients/typescript-client): Create a typed client, pass credentials, handle errors and open tunnels.

  - [CLI](/docs/guides/clients/cli): Authenticate, call any method, reserve GPUs, run fleet plans and verify the audit trail.

  - [MCP](/docs/guides/clients/mcp): Tools for AI agents behind Better IAM's gate, with delegated confirmation.

  - [Console](/docs/guides/clients/console): The Next.js console and console-kit, the stores it is built on.


# MCP (/docs/guides/clients/mcp)

> The Better Compute MCP server gives AI agents tools to find capacity, reserve GPUs and check nodes, as the agent, behind Better IAM's gate with delegated confirmation.



`@better-compute/mcp` turns the compute API into [Model Context Protocol](https://modelcontextprotocol.io) tools, so
an AI agent can find GPU capacity, reserve and release GPUs, create, stop, start and delete VPS instances, create,
extend and delete [sandboxes](/docs/guides/vps/sandboxes), look at a node and run a diagnostic. It is built for
the v0.3 goal "an agent reserves via MCP with human confirmation, retrying with the same key", and the end-to-end
test `tests/mcp-e2e-confirmation.test.ts` runs exactly that against a real server and an embedded Better IAM.

The design has one rule: &#x2A;*every tool calls the compute API as the caller.** A tool forwards the agent's own
`Authorization` header, so compute and Better IAM make the real decision on each call, and a tool can never do more
than the agent's credential allows. The server holds no credential of its own.

## How it fits [#how-it-fits]

<Mermaid
  chart="sequenceDiagram
  participant A as Agent (MCP client)
  participant G as Better IAM gate
  participant M as createComputeMcp
  participant C as compute API
  participant I as Better IAM
  A->>G: POST tools/call reserve_gpus (Bearer agent token)
  G->>I: authenticate; compute:pools:read on the pool?
  G->>M: forwarded request, caller
  M->>M: validate arguments against the tool's schema
  M->>C: reservations.create (same Bearer token)
  C->>I: decide compute:reservations:create on the draft
  C-->>M: reservation, or CONFIRMATION_REQUIRED
  M-->>A: tool result (structuredContent)"
/>

The server is a small, stateless Streamable HTTP endpoint that speaks JSON-RPC 2.0 for `initialize`, `ping`,
`tools/list` and `tools/call`, and accepts notifications. It has no MCP SDK dependency; its only runtime dependency
is `@better-compute/client`. `@better-iam/mcp`, which provides the gate, is an optional peer.

> **Mounted when you ask for it.** 
  `serve` and `dev` mount the endpoint when the configuration's host sets `host.mcp` (below); nothing is mounted
  otherwise. You can also host the handler yourself in any runtime with the Fetch API, with Better IAM embedded in
  the same process or reached over HTTP ([remote IAM](#remote-iam)).

## Mount it with serve [#mount-it-with-serve]

`computeMcpEndpoint({ compute, gate, delegations?, ...options })` builds the endpoint for a control plane in the same
process: the gate first, then the tools, each calling `compute.handler` in-process with the caller's own
`Authorization` header, so compute decides every call exactly as it would over HTTP. With `delegations` (Better IAM's
delegations API) the tools wait inside the call for a person to approve a held-back action. Hand its `handler` to the
host:

```ts title="better-compute.config.ts"
import { createMcpGate } from '@better-iam/mcp';
import { defineHost } from 'better-compute/cli';
import { computeMcpEndpoint, gateTools } from 'better-compute/mcp';

export const host = defineHost(({ compute }) => ({
  mcp: {
    path: '/mcp', // the default
    handler: computeMcpEndpoint({
      compute,
      gate: createMcpGate({ iam, tenantId: platformTenantId, tools: gateTools(), unlisted: 'deny', confirmations: false }),
      delegations: iam.api.delegations,
    }).handler,
  },
}));
```

`serve` and `dev` then answer `POST`, `GET` and `DELETE` on the path, and `GET
/.well-known/oauth-protected-resource{path}` for the gate's RFC 9728 metadata. A path that is not absolute, or a
missing handler, is `INVALID_CONFIG`. The gate is required. Give it a fixed `tenantId` (the organization that runs the
endpoint), or choose one per caller deliberately: deciding in each caller's own tenant would let any organization
grant itself access. Build it with `confirmations: false`: the gate only checks visibility, and confirmations belong
to compute's own decisions, which carry the whole request. With the gate's default, a wildcard in a delegation's
confirm list would also match the gate's read actions and file confirmation requests built from unvalidated tool
arguments (daf58eb, security review 14 M2). The compose audit test mounts it through a real `serve`: no token is
401, and an administrator lists the tools.

`better-compute doctor` checks that the mounted handler is gated (b491aed, review 14 L6). It sends the handler one
unauthenticated JSON-RPC `tools/list` (read-only, given up after 5 seconds). A gated endpoint answers 401 with its
OAuth challenge; any 2xx is the error finding `host.mcp.ungated` ("host.mcp answers MCP at /mcp without a
credential"), since a bare `createComputeMcp().handler`, or one with a fixed client, serves MCP to anyone or makes
every caller act as one credential. A probe that fails or gets a non-2xx answer reports nothing.

## Mount the endpoint yourself [#mount-the-endpoint-yourself]

```ts title="mcp.ts"
import { createMcpGate } from '@better-iam/mcp';
import { createComputeMcp, gateTools, iamConfirmations } from 'better-compute/mcp';
import { iam } from './iam';

const mcp = createComputeMcp({
  baseURL: 'https://compute.example.com',
  // Optional: wait inside the tool call for the person to approve (see below).
  confirmations: iamConfirmations(iam.api.delegations),
});

const gate = createMcpGate({ iam, tenantId, tools: gateTools(), unlisted: 'deny', confirmations: false });

export default {
  fetch: (request: Request) => gate(request, (forwarded, caller) => mcp.handler(forwarded, caller)),
};
```

With `baseURL`, each request gets a fresh compute client that forwards the caller's `Authorization` header and a
request id. The gate authenticates the caller, applies the visibility rules below, and hands the handler the
caller's `tenantId`, which the tools act in.

<TypeTable
  type="{
  baseURL: { type: 'string', description: 'The compute API origin. Needed unless client is given.' },
  basePath: { type: 'string', description: &#x22;Where the API is mounted.&#x22;, default: &#x22;'/api/compute'&#x22; },
  client: {
    type: 'ComputeApi | ((request, caller?) => ComputeApi)',
    description: 'How tools reach compute instead: a fixed client, or one per request. Anything with $request fits, such as createComputeClient().',
  },
  tenantId: { type: 'string | ((caller?) => string | undefined)', description: &#x22;The tenant tools act in. Default: the caller's tenantId. Without one, every tool answers ACCESS_DENIED.&#x22; },
  execCommand: { type: 'boolean', description: 'Build exec_command. Off: running node commands through a model is a deliberate choice.', default: 'false' },
  fleet: { type: 'boolean', description: 'Build the read-only fleet_export and fleet_plan tools (platform administrators only).', default: 'false' },
  providers: { type: 'boolean', description: 'Build the read-only list_provider_accounts, check_provider_account, provider_account_findings, provider_account_billing, list_billing_leaks and provider_account_connect_url tools. Pass the same option to gateTools().', default: 'false' },
  confirmations: {
    type: 'ConfirmationPolling | ((request, caller?) => ConfirmationPolling | undefined)',
    description: 'Wait for people to approve held-back actions inside the tool call. iamConfirmations() builds one per request.',
  },
  serverInfo: { type: '{ name: string; version: string }', description: 'Reported at initialize.', default: &#x22;{ name: 'better-compute', version: '0.1.0' }&#x22; },
  instructions: { type: 'string', description: 'Shown to the model at initialize. The default explains the list, preview, reserve and retry flow.' },
  maxBodyBytes: { type: 'number', description: 'Largest JSON-RPC request body.', default: '1048576 (1 MiB)' },
  maxConcurrentCallsPerCaller: { type: 'number', description: &#x22;Tool calls one caller may have in flight at once, counted per caller identity (tenant and identity), else per Authorization header. Above it a call answers the tool error RATE_LIMITED with retryAfterMs 2000 (48d41a4). Counted by callLimiter.&#x22;, default: '4' },
  callLimiter: { type: 'McpCallLimiter', description: &#x22;Where the per-caller count lives (76e4e37): { acquire(key, max) } returns a release, or undefined to refuse. The default, memoryCallLimiter(), counts in this process, so with several MCP instances the cap multiplies; pass a shared one (Redis, your database) to hold it across them. A shared limiter must expire slots it never saw released, for example after 2 minutes. A limiter that throws refuses the call with UNAVAILABLE and retryAfterMs 2000, never lets it through uncounted. Keys never carry credentials: a caller known only by its Authorization header counts under the header's SHA-256.&#x22;, default: 'memoryCallLimiter()' },
  newKey: { type: '() => string', description: 'Idempotency keys for tools that create things.', default: 'mcp- and 24 random hex characters' },
}"
/>

### Remote IAM [#remote-iam]

When Better IAM runs in another process, give the gate `remoteMcpIam()` instead of a `betterIam()` instance (v0.4).
It wraps a `createIamClient<BetterIam>()` from `@better-iam/client` and answers the gate's calls over Better IAM's
HTTP API, each with the caller's own credential, so every decision is still made and audited as the caller. Only
`authorization`, `user-agent` and `x-request-id` are forwarded; since 7890821 (review 14 L3) neither cookies nor the
client's own `x-forwarded-for`, which behind an appending proxy would let the caller choose `request.sourceIp` for
policy conditions:

| Gate call                | Better IAM API                                                                   |
| ------------------------ | -------------------------------------------------------------------------------- |
| authenticate             | `sts.getCallerIdentity`                                                          |
| authorize, authorizeMany | `authorize`, `authorizeMany`                                                     |
| confirmations            | the caller's delegation: `delegations.get` and `delegations.requestConfirmation` |

```ts
import { createIamClient } from '@better-iam/client';
import type { BetterIam } from '@better-iam/server';
import { remoteMcpIam } from 'better-compute/mcp';

const gate = createMcpGate({
  iam: remoteMcpIam(createIamClient<BetterIam>({ baseURL: 'https://iam.example.com' })),
  tenantId,
  tools: gateTools(),
  unlisted: 'deny',
  confirmations: false,
});
```

`iamConfirmations()` reads confirmations through the server API's shape (`getConfirmation(credential, input)`), so a
remote host that wants tools to wait for approval passes an equivalent reader over its IAM client.
`tests/mcp-e2e-confirmation.test.ts` runs the whole confirmation flow through both the embedded and the remote gate,
including refusing another organization's owner.

`createComputeMcp()` returns the `tools`, `listTools()` (the `tools/list` entries) and `callTool(name, args, request,
caller?)` besides `handler`, so you can run a tool without the JSON-RPC wrapper, for example from your own agent
framework.

## The tools [#the-tools]

| Tool                                                    | Calls                                                                                                                                                                                                        | Gate rule                                                | Hints                                   |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | --------------------------------------- |
| `list_pools`                                            | `pools.list`                                                                                                                                                                                                 | any authenticated caller                                 | read-only                               |
| `capacity_summary`                                      | `capacity.summary`, or `capacity.byPool` with `poolId`                                                                                                                                                       | any authenticated caller                                 | read-only                               |
| `preview_reservation`                                   | `capacity.preview`                                                                                                                                                                                           | `compute:pools:read` on the pool                         | read-only                               |
| `reserve_gpus`                                          | `reservations.create` (`binding: 'hold'`, or `'kubernetes'`)                                                                                                                                                 | `compute:pools:read` on the pool                         | idempotent                              |
| `release_reservation`                                   | `reservations.release`                                                                                                                                                                                       | any authenticated caller                                 | destructive, idempotent                 |
| `list_reservations`                                     | `reservations.list`: the caller's own by default, `scope: 'all'`, filters `state` and `poolId`, `limit` 1 to 200, `cursor`                                                                                   | any authenticated caller                                 | read-only                               |
| `get_reservation`                                       | `reservations.get`                                                                                                                                                                                           | any authenticated caller                                 | read-only                               |
| `extend_reservation`                                    | `reservations.extend` by a duration, confirmation-aware                                                                                                                                                      | any authenticated caller                                 | idempotent                              |
| `node_status`                                           | `nodes.get`                                                                                                                                                                                                  | `compute:nodes:read` on the node                         | read-only                               |
| `run_diagnostic`                                        | `commands.run` with `diag.nvml`                                                                                                                                                                              | `compute:nodes:read` on the node                         | idempotent                              |
| `create_instance`                                       | `instances.create`                                                                                                                                                                                           | `compute:pools:read` on the pool                         | idempotent                              |
| `list_instances`                                        | `instances.list`: the caller's own by default, `scope: 'all'`, filter `poolId`, `limit` 1 to 200, `cursor`                                                                                                   | any authenticated caller                                 | read-only                               |
| `get_instance`                                          | `instances.get`                                                                                                                                                                                              | any authenticated caller                                 | read-only                               |
| `start_instance`                                        | `instances.start`                                                                                                                                                                                            | any authenticated caller                                 | idempotent                              |
| `stop_instance`                                         | `instances.stop`                                                                                                                                                                                             | any authenticated caller                                 | destructive, idempotent                 |
| `delete_instance`                                       | `instances.delete`                                                                                                                                                                                           | any authenticated caller                                 | destructive, idempotent                 |
| `list_sandbox_templates`                                | `sandboxTemplates.list`, without each template's `userData` and `env`                                                                                                                                        | any authenticated caller                                 | read-only                               |
| `create_sandbox`                                        | `sandboxes.create`, confirmation-aware                                                                                                                                                                       | `compute:sandboxes:read` on the template                 | idempotent                              |
| `list_sandboxes`                                        | `sandboxes.list`: the caller's own by default, `scope: 'all'`, filter `templateId`, `limit` 1 to 200, `cursor`                                                                                               | any authenticated caller                                 | read-only                               |
| `get_sandbox`                                           | `sandboxes.get`                                                                                                                                                                                              | any authenticated caller                                 | read-only                               |
| `extend_sandbox`                                        | `sandboxes.extend` by a duration                                                                                                                                                                             | any authenticated caller                                 | not idempotent: each call extends again |
| `delete_sandbox`                                        | `sandboxes.delete`                                                                                                                                                                                           | any authenticated caller                                 | destructive, idempotent                 |
| `list_sandbox_snapshots`                                | `sandboxSnapshots.list`: the caller's own by default, `scope: 'all'`, filter `sandboxId`, `limit` 1 to 200, `cursor`                                                                                         | any authenticated caller                                 | read-only                               |
| `delete_sandbox_snapshot`                               | `sandboxSnapshots.delete`                                                                                                                                                                                    | any authenticated caller                                 | destructive, idempotent                 |
| `list_sandbox_volumes`                                  | `sandboxVolumes.list` (v1.1), filter `poolId`, `scope` (`mine` by default, or `all`)                                                                                                                         | any authenticated caller                                 | read-only                               |
| `create_sandbox_volume`                                 | `sandboxVolumes.create`, confirmation-aware                                                                                                                                                                  | `compute:pools:read` on the pool                         | idempotent                              |
| `resize_sandbox_volume`                                 | `sandboxVolumes.resize`                                                                                                                                                                                      | any authenticated caller                                 | idempotent                              |
| `delete_sandbox_volume`                                 | `sandboxVolumes.delete`                                                                                                                                                                                      | any authenticated caller                                 | destructive, idempotent                 |
| `list_command_batches`                                  | `commandBatches.list` (v1.0): filter `state`, `limit` 1 to 200, `cursor`                                                                                                                                     | any authenticated caller                                 | read-only                               |
| `get_command_batch`                                     | `commandBatches.get` with a `batchId` (`batch_…`)                                                                                                                                                            | any authenticated caller                                 | read-only                               |
| `exec_command` (with `execCommand: true`)               | `commands.run`: `inventory.refresh`, `logs.collect` or `node.reboot`, optionally with a `batchId`                                                                                                            | `compute:nodes:read` on the node                         | destructive                             |
| `propose_command_batch` (with `execCommand: true`)      | `commandBatches.create` (v1.0): a `node.reboot` batch over the given nodes                                                                                                                                   | any authenticated caller                                 | idempotent                              |
| `fleet_export` (with `fleet: true`)                     | `fleet.export`                                                                                                                                                                                               | `compute:fleet:admin` on the fleet                       | read-only                               |
| `fleet_plan` (with `fleet: true`)                       | `fleet.plan`                                                                                                                                                                                                 | `compute:fleet:admin` on the fleet                       | read-only                               |
| `list_provider_accounts` (with `providers: true`)       | `providerAccounts.list`                                                                                                                                                                                      | any authenticated caller                                 | read-only                               |
| `check_provider_account` (with `providers: true`)       | `providerAccounts.check`: what the account's credential may do, read-only at the provider                                                                                                                    | `compute:providers:read` on the account                  | read-only                               |
| `provider_account_findings` (with `providers: true`)    | `providerAccounts.findings`: what the provider says needs an operator (DigitalOcean: its security scan)                                                                                                      | `compute:providers:read` on the account                  | read-only                               |
| `provider_account_billing` (with `providers: true`)     | `providerAccounts.billing`: the account's own bill at its provider (balance, invoices, history)                                                                                                              | `compute:providers:manage` on the account                | read-only                               |
| `list_billing_leaks` (with `providers: true`)           | `billing.leaks` over the last `days` days (default 31): instances the provider billed more compute hours for than compute metered                                                                            | `compute:billing:read` on the fleet (platform operators) | read-only                               |
| `provider_account_connect_url` (with `providers: true`) | `providerAccounts.connectSetup`: the provider's OAuth authorization URL for a person to open. No tool calls `providerAccounts.connect`: its code arrives on the redirect, in the browser that kept the state | `compute:providers:read` on the tenant                   | read-only                               |

The GPU tools share one argument shape: `poolId`, `gpus` (1 to 64 whole GPUs), `duration` (such as `30m`, `8h`,
`3d`), and optionally `gpuModel` (`H100`), `gpuProduct` (`H100-SXM5-80GB`), `nvlink` (all GPUs in one NVLink
domain) and `selector` (up to 32 `{ key, op, values }` expressions with `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`,
`Gte`, `Lt` or `Lte`). The tool turns them into one device request of class `gpu`. Since 17f600b both also take
`vcpu` (0 to 4096) and `memoryGiB` (0 to 65,536), the host CPUs and memory to book with the GPUs (sent as
`resources`; without them the pool's per-GPU share), and `kubernetes: true`, which asks for `binding: 'kubernetes'`
on a pool whose machines are [Kubernetes workers](/docs/guides/capacity/kubernetes) instead of a hold; the preview books
the same host resources as the create would. `reserve_gpus` also takes an
`idempotencyKey` and `waitSeconds` (0 to 30, default 20), which it sends as `wait: { timeoutMs }`, so on a pool-lease
pool the call returns once the reservation is placed or the wait runs out; see
[waiting for placement](/docs/guides/capacity/reservations#waiting-for-placement).

`create_instance` takes `poolId`, `flavor` and `imageId`, and optionally an `idempotencyKey` and
`waitSeconds` (0 to 30, default 20). It takes no `sshKeys` (7890821, review 14 H1): an agent under prompt injection
could install an attacker's key, so people add keys themselves and the agent reaches the VM through tunnels. The wait
is sent as `wait: { timeoutMs }`: the call returns once the instance is running or has
failed, or the wait runs out, so the agent does not have to poll `get_instance`; see
[waiting for the instance](/docs/guides/vps/instances#waiting-for-the-instance).

The instance tools take one `instanceId` (`inst_…`), except `list_instances`. They pass it to compute as is, with the
caller's tenant: `stop_instance` is a graceful shutdown that keeps the disk and addresses, `start_instance` runs a
stopped instance again, and `delete_instance` removes the instance and its disk for good. Unlike `create_instance`,
they do not turn `CONFIRMATION_REQUIRED` into a result; it comes back as a tool error.

The sandbox tools act on the caller's tenant. `create_sandbox` takes a `templateId` (`sbxt_…`, from
`list_sandbox_templates`) and optionally `ttlSeconds` (60 s to 30 days, at most the template's maximum), an
`idempotencyKey` and `waitSeconds` (0 to 30, default 20), sent as `wait: { timeoutMs }`: the call returns once the
sandbox is ready or failed, or the wait runs out. It answers `{ idempotencyKey, sandbox }` and, like
`create_instance`, turns `CONFIRMATION_REQUIRED` into a result to retry with the same key. `extend_sandbox` takes a
`sandboxId` and `by` (such as `30m`), and `get_sandbox` and `delete_sandbox` a `sandboxId`; `delete_sandbox` removes
the sandbox and its disk before its TTL ends. `list_sandbox_snapshots` lists snapshots (kept ones of deleted sandboxes
too), and `delete_sandbox_snapshot` deletes one (`snap_…`) for good. No tool takes a snapshot or calls
`sandboxes.exec`.

**Volume tools are agent memory** (e6759df). [Sandbox volumes](/docs/guides/vps/sandboxes#volumes) are persistent
disks that outlive sandboxes, which a template mounts in every sandbox it creates. `list_sandbox_volumes` lists the
caller's own volumes, or with `scope: 'all'` the tenant's (e68aba7; the API's default is `mine` since 77bf9f5),
optionally of one `poolId`. `create_sandbox_volume` takes a `poolId`, a `name` unique in the tenant
(`^[a-z0-9][a-z0-9._-]{0,62}$`, such as `agent-notes`), `sizeGiB` (1 to 16,384, the API's own limit, so a larger size is
refused by the schema before compute is called; f181f43) and optionally an `idempotencyKey`; like `create_sandbox` it answers
`{ idempotencyKey, volume }` and turns `CONFIRMATION_REQUIRED` into a result to retry with the same key. The volume
lives on one host and is billed until deleted. `resize_sandbox_volume` grows a volume (`vol_…`) to `sizeGiB`, never
shrinks it (the same 1 to 16,384), and sandboxes see the new size at their next boot. `delete_sandbox_volume` deletes a volume and wipes its
data; a volume a sandbox still mounts is refused. No tool attaches a volume to a template: a person does that with
`sandboxTemplates.update`, the CLI's `sandbox-template update --volumes` or a FleetConfig template.

Every argument is checked against the tool's JSON Schema before anything is called: ids must match their prefix
(`pool_…`, `node_…`, `rsv_…`, `inst_…`, `sbxt_…`, `sbx_…`, `snap_…`, `vol_…`, `batch_…`), and unknown arguments are refused. A bad argument is a tool error `INVALID_INPUT`
with the path of the first problem. The check reads own properties only (`constructor`, `__proto__` and the like are
never arguments) and enforces `minItems` and `uniqueItems`, and the list tools put the caller's `tenantId` after
the arguments, so an argument can never replace it (7890821, review 14 L1).

**Batch tools read and propose, never approve.** `list_command_batches` and `get_command_batch` show the tenant's
[destructive-batch](/docs/operations/pki#the-destructive-batch-quorum-v10) quorum: a batch's reason, size, the
approvals it has and needs, and its commands by state. With `execCommand: true`, `propose_command_batch` (171ad2f)
creates a batch for operators to approve: `nodeIds` (1 to 1,000 distinct `node_…` ids), `reason` (1 to 500
characters, what the approvers read), optionally `kind` (only `node.reboot`, the default), `notAfterSeconds` (60 to
259,200, how long approvals and commands are accepted; default 24 h) and an `idempotencyKey`. Every leaf's subject is
a node. Nothing runs until people approve it with their approver keys (`better-compute batch approve`); the agent then
passes the `batchId` to `exec_command` for each node, and the command waits, held, until the batch is approved. With
approver keys configured, `exec_command` with `node.reboot` and no `batchId` is refused `INVALID_STATE`, reason
`batch-required`. There is no approve tool: approving is a person's signature with an approver key held off the
control plane.

**Fleet tools plan but never apply.** `fleet_plan` returns the changes and the `planHash`, with the list of calls
replaced by `callCount`. There is deliberately no apply tool: a person reviews the plan and runs
`better-compute fleet apply FILE --plan-hash HASH`. Pass `fleet: true` to both `createComputeMcp` and `gateTools`.

## The gate checks visibility only [#the-gate-checks-visibility-only]

`gateTools()` returns the rules for `@better-iam/mcp`'s `createMcpGate({ tools, unlisted: 'deny' })`. They are
deliberately light:

* A tool that names a pool is visible to a caller with `compute:pools:read` on `compute-pool/{poolId}`; one that
  names a node needs `compute:nodes:read` on `compute-node/{nodeId}`.
* `list_pools`, `capacity_summary`, the reservation tools (`release_reservation`, `list_reservations`,
  `get_reservation`, `extend_reservation`) and the instance tools (`list_instances`, `get_instance`,
  `start_instance`, `stop_instance`, `delete_instance`) are open to any authenticated caller, because the compute API
  filters and decides every call anyway, on the reservation or instance itself (`OwnReservations`, `BoundedExtend`,
  a delegation's confirm list).
* The fleet tools need `compute:fleet:admin` on `compute-fleet/global`, which only resolves in the root tenant.
* `list_command_batches` and `get_command_batch` are open to any authenticated caller; compute decides
  `compute:batches:read` on the tenant. So is `propose_command_batch`, where compute decides
  `compute:batches:create` on the tenant, and the batch then needs people's approver keys.
* `create_sandbox` is visible to a caller with `compute:sandboxes:read` on `compute-sandbox-template/{templateId}`:
  creating a sandbox starts with seeing its template. The other sandbox tools (`list_sandbox_templates`,
  `list_sandboxes`, `get_sandbox`, `extend_sandbox`, `delete_sandbox`, `list_sandbox_snapshots`,
  `delete_sandbox_snapshot`) are open to any authenticated caller, since compute decides each call on the tenant,
  the sandbox, the snapshot or the draft.
* `create_sandbox_volume` is visible with `compute:pools:read` on its pool; `list_sandbox_volumes`,
  `resize_sandbox_volume` and `delete_sandbox_volume` are open to any authenticated caller, and compute decides each
  call on the tenant or the volume (e6759df).

The rules accept only well-formed ids: a `poolId` of letters, digits and `._:-` (at most 128 characters), a
`node_…` or a `sbxt_…` id. Anything else throws, so the gate denies the call (7890821, review 14 M2): the gate runs
before compute validates anything, and unvalidated text must never reach a confirmation a person reads.

Every tool the server can list has a rule: a test builds a server with every option on and requires a gate rule for
each of its tools, so a gate with `unlisted: 'deny'` never hides a tool by omission.

The gate never checks the action a tool performs, such as `compute:reservations:create`. That decision happens once,
in compute, on the reservation draft with its full attributes. Actions a delegation holds back for confirmation are
never gate actions, so a person is never asked twice about two different resources for one request. Pass
`typePrefix` to `gateTools` if your deployment uses a resource type prefix other than `compute-`.

## Results and errors [#results-and-errors]

A tool result carries the compute response twice: as pretty-printed JSON text in `content`, and as
`structuredContent` (an object; other values are wrapped as `{ result }`). `reserve_gpus` answers
`{ idempotencyKey, reservation }`, `create_instance` `{ idempotencyKey, instance }` and `create_sandbox`
`{ idempotencyKey, sandbox }`, so the model always has the key to retry with.

A compute error becomes a tool error with `isError: true`, the text `CODE: message`, and
`structuredContent.error` holding `code`, `message`, and `retryAfterMs` and `details` when compute sent them. Any
other failure is reported as `INTERNAL_ERROR` with a generic message, never internals. JSON-RPC problems use the
standard codes: `-32700` for unparsable JSON (HTTP 400) or a non-JSON content type (HTTP 415), `-32600` for an
invalid request or a batch (HTTP 400) or a body over `maxBodyBytes` (HTTP 413), `-32601` for an unknown method and
`-32602` for a missing `params.name`. Methods other than POST get 405. Notifications get 202 with no body.

## Delegated confirmation [#delegated-confirmation]

An agent usually acts through a Better IAM **delegation**: a person lets the agent use some of their permissions,
and can list actions that the person must approve each time. When the agent calls one of those,
compute answers `CONFIRMATION_REQUIRED` and Better IAM opens a confirmation request. See
[delegated confirmation](/docs/guides/iam/delegated-confirmation) for the whole mechanism.

`reserve_gpus`, `extend_reservation`, `create_instance` and `create_sandbox` turn that error into a normal result the
model can act on (`extend_reservation` answers `{ idempotencyKey, extension }` once it goes through):

```json title="reserve_gpus, waiting for a person"
{
  "status": "confirmation-required",
  "confirmationId": "cnf_…",
  "idempotencyKey": "mcp-3f9c0a…",
  "message": "A person must approve this action. Once they have, call the tool again with the same idempotencyKey."
}
```

The model calls the tool again with the same arguments and that `idempotencyKey` once the person has approved. The
retry reaches the same reservation draft, Better IAM finds the approval, and the reservation is placed. The default
server instructions tell the model to do exactly this.

**Waiting inside the call.** With `confirmations`, the tool polls instead of returning at once: it asks for the
confirmation's state every `intervalMs` (default 2 s) for up to `timeoutMs` (default 25 s). On `approved` it
retries once with the same key and returns the result; on `denied` or `expired` it returns
`{ status: 'confirmation-denied' | 'confirmation-expired', confirmationId, idempotencyKey }`; on a timeout it returns
`confirmation-required` as above. If checking the approval fails, it still returns that pending result with its
`idempotencyKey` and a note to retry with the same key (7890821, review 14 L4), so the agent never files a second
draft and a second request for the same person.

`iamConfirmations(iam.api.delegations, { timeoutMs?, intervalMs?, now? })` builds that poller per request from
Better IAM's `delegations.getConfirmation`, called with the caller's own bearer token in the caller's tenant, so an
agent only ever sees its own requests. It maps `rejected` to `denied`, and a `pending` request past its `expiresAt`
to `expired`. Without a bearer token or a tenant it returns nothing and the tool answers `confirmation-required`.

> **Stop on a rejection.** 
  A retry after the person rejected is decided again, denied again, and opens a new confirmation request. Tell your
  agent to stop on `confirmation-denied`, and to start over with a new request (and key) after
  `confirmation-expired`.

## Limits worth knowing [#limits-worth-knowing]

* `exec_command` is off unless you pass `execCommand: true`, and even then it only runs `inventory.refresh`,
  `logs.collect` and `node.reboot`. `node.reboot` restarts the machine; under the quorum it needs a batch from
  `propose_command_batch` that people approved.
* One caller has at most `maxConcurrentCallsPerCaller` (default 4) tool calls in flight. A call can wait about
  85 seconds for an approval or an instance to start, so an agent looping on retries would otherwise pile up held
  requests; above the cap a call answers `RATE_LIMITED` with `retryAfterMs: 2000` (48d41a4, review 14 L5). With
  several MCP server instances, pass a shared `callLimiter` so the cap holds across them (76e4e37); the default
  counts per process.
* The person approving a `create_instance`, `reserve_gpus`, `extend_reservation` or `create_sandbox` reads compute's
  description of the request, and the approval admits only that request; see
  [delegated confirmation](/docs/guides/iam/delegated-confirmation#what-the-person-sees).
* `run_diagnostic` runs `diag.nvml` only. It returns the queued command; the result arrives when the node finishes,
  and `node_status` or the command's own methods show it.
* `reserve_gpus` books with `binding: 'hold'`, or `kubernetes` with `kubernetes: true`; whole-node leases, spot, gangs
  and calendar starts are not exposed as tool arguments.
* `release_reservation`, `stop_instance`, `delete_instance`, `delete_sandbox`, `delete_sandbox_snapshot` and
  `delete_sandbox_volume` are marked destructive: many MCP hosts ask the user before calling such a tool.
* The sandbox tools cannot take a snapshot or run code in a sandbox; code runs through the CLI's `sandbox exec`.
* The server is stateless, with JSON responses only: it does not open SSE streams or send server notifications, and
  `tools.listChanged` is `false`. It negotiates protocol versions `2025-06-18`, `2025-03-26` and `2024-11-05`,
  answering with the newest when the client asks for another.

Related: [reservations](/docs/guides/capacity/reservations), [policies](/docs/guides/iam/policies) (the
`agent-guardrails` example policy), and the [TypeScript client](/docs/guides/clients/typescript-client) the tools
are built on.


# TypeScript client (/docs/guides/clients/typescript-client)

> createComputeClient infers every group and method from your server's type, sends bearer tokens or the IAM cookie, and raises ComputeClientError with stable codes.



`@better-compute/client` is the typed, browser-safe client for the compute API. Its types come from your server
through a type-only import, so there is no code generation step and no schema file to keep in sync: add a module
to `betterCompute()` and its methods appear on the client. At run time it is a small `fetch` wrapper behind a Proxy,
with one dependency (`@better-compute/core`, for types).

It is exported twice: as `@better-compute/client`, and as `better-compute/client` from the umbrella package. Both are
the same module.

## Create a client [#create-a-client]

```ts title="client.ts"
import { createComputeClient } from 'better-compute/client';
import type { compute } from './compute.server'; // type-only: nothing from the server is bundled

export const client = createComputeClient<typeof compute>({
  baseURL: 'https://compute.example.com',
  token: () => session.getToken(), // read at every request, never stored
  requestId: true,
  retryRateLimited: true,
});

const { version } = await client.system.version();
const pools = await client.pools.list({ tenantId: 'acme' });
```

`compute` is the value `betterCompute({ ... })` returns in your server code. The client needs only its type, so
`import type` keeps the server, its database driver and its secrets out of a browser bundle.

<TypeTable
  type="{
  baseURL: {
    type: 'string',
    description: &#x22;The API's origin, an absolute http(s) URL without credentials, query or fragment. Defaults to the page's origin in a browser; required elsewhere.&#x22;,
  },
  basePath: { type: 'string', description: 'Where the API is mounted: an absolute path without a trailing slash.', default: &#x22;'/api/compute'&#x22; },
  token: {
    type: 'string | (() => string | undefined | Promise<string | undefined>)',
    description: 'The bearer token: a Better IAM token (biam_...) or a session JWT whose audience includes compute. A function is called before every request.',
  },
  headers: { type: 'HeadersInit | (() => HeadersInit | Promise<HeadersInit>)', description: 'Extra headers for every request.' },
  fetch: { type: 'typeof fetch', description: 'The Fetch implementation.', default: 'globalThis.fetch' },
  credentials: {
    type: &#x22;'omit' | 'same-origin' | 'include'&#x22;,
    description: &#x22;Whether the browser sends cookies. Compute authenticates bearer tokens, so the default sends none; a same-origin console that relies on the IAM cookie uses 'same-origin'.&#x22;,
    default: &#x22;'omit'&#x22;,
  },
  requestId: {
    type: 'boolean | (() => string)',
    description: 'Send an X-Request-Id on every request (true generates one per request) and expose it on errors.',
    default: 'false',
  },
  retryRateLimited: {
    type: 'boolean | { maxWaitMs?: number }',
    description: 'Retry a RATE_LIMITED answer once, after the server\'s retryAfterMs, when that wait is at most maxWaitMs (5000 by default).',
    default: 'false',
  },
  onUnauthenticated: { type: '(error: ComputeClientError) => void', description: 'Called once per UNAUTHENTICATED answer, after the error is raised. Send people to sign in here.' },
}"
/>

A bad option fails at once, before any request, with `ComputeClientError('INVALID_CONFIG', …, 0)`: a relative or
credential-bearing `baseURL`, a `basePath` with a trailing slash, a missing `fetch`, or an unknown `credentials`
value.

## Call methods [#call-methods]

Every server method has the shape `(credential, input)`. On the client the credential is supplied for you, so each
method becomes `(input, options?)`:

```ts
// Server: compute.api.reservations.get(credential, { tenantId, reservationId })
// Client:
const reservation = await client.reservations.get({ tenantId: 'acme', reservationId: 'rsv_…' });

// The input is optional when the method has no required field.
const me = await client.system.whoami();

// Per call: an AbortSignal and extra headers.
const controller = new AbortController();
await client.capacity.summary({ tenantId: 'acme' }, { signal: controller.signal });
```

The types follow the server exactly:

* **Groups exist only for the modules you built with.** `client.instances` is a type error unless `vps()` is in the
  server's `modules`, and `client.webhooks` needs `eventsModule()`.
* **Plugins** are under `client.plugins[id][group][method]`, for plugins registered on the server:
  `await client.plugins['acme-autoinstall'].isos.render({ tenantId, nodeId })`.
* **Results** are the handler's return type, awaited.

`$request(path, input?, options?)` calls any route below the base path by name. Use it for groups the server adds
at run time and the type does not list, such as `tunnels`, or when you have no server type to import:

```ts
const opened = await client.$request<{ tunnelId: string; path: string }>('tunnels/open', {
  tenantId: 'acme',
  instanceId: 'inst_…',
  op: 'exec',
});
```

A path is plain segments that start with a letter (`system/version`, `plugins/acme/widgets/poke`). Anything else
is refused with `INVALID_PATH` before a request is made. The segments `then`, `catch`, `finally`, `toJSON`,
`constructor`, `prototype` and `__proto__` never route, so the client is not thenable and `await client` or
`JSON.stringify(client)` does no network call.

### On the wire [#on-the-wire]

Each call is one request:

```text
POST /api/compute/reservations/get HTTP/1.1
Host: compute.example.com
Content-Type: application/json
X-Better-Compute: 1
Authorization: Bearer biam_…
X-Request-Id: 6f1c…

{"tenantId":"acme","reservationId":"rsv_…"}
```

The client always sets `content-type` and `x-better-compute`, after your own headers, because the server's CSRF
check needs both. It sends `redirect: 'error'` and `cache: 'no-store'`, and the input must serialize as JSON
(`INVALID_INPUT` otherwise).

## Credentials [#credentials]

**Bearer tokens.** `token` accepts only the two shapes the server can authenticate: an opaque token of 16 to 512
URL-safe characters (`biam_…`), or a session JWT of three base64url segments, at most 4096 characters. Anything
else fails with `INVALID_TOKEN` before a request, so a stray value never ends up in an `Authorization` header. Pass
a function (sync or async) so a refreshed session is picked up on the next call; the client never stores what it
returns, and `undefined` sends no `Authorization` header.

A session JWT is accepted only when its audience includes compute's; see
[credentials compute accepts](/docs/guides/iam#credentials-compute-accepts).

**The IAM cookie.** A console served from the same origin as compute, with `betterCompute({ acceptIamCookie: true })`,
can rely on the person's Better IAM session cookie instead of a token. Leave `token` unset and send cookies:

```ts title="The console's client"
const api = createComputeClient({
  baseURL: origin,
  credentials: 'same-origin',
  onUnauthenticated: () => window.location.assign(`/login?next=${encodeURIComponent(location.pathname)}`),
});
```

The server accepts the cookie only with an `Origin` header equal to its own origin; the browser sends one on these
POSTs. See [console](/docs/guides/clients/console).

## Errors [#errors]

Every failure throws a `ComputeClientError`. It keeps the server's stable code and message and never exposes the raw
response body.

<TypeTable
  type="{
  code: { type: 'string', description: 'The error code: ACCESS_DENIED, NOT_FOUND, CONFLICT, CONFIRMATION_REQUIRED, RATE_LIMITED, or a client-side code below.', required: true },
  message: { type: 'string', description: 'A human-readable message.', required: true },
  status: { type: 'number', description: 'The HTTP status; 0 when the client refused before sending anything.', required: true },
  retryAfterMs: { type: 'number', description: 'For RATE_LIMITED and retryable conflicts: how long to wait. From the error body, else the Retry-After header.' },
  requestId: { type: 'string', description: &#x22;The request's X-Request-Id (yours, or the one the server echoed), for support and log lookups.&#x22; },
  details: { type: 'Record<string, unknown>', description: 'Machine-readable context, such as confirmationId for CONFIRMATION_REQUIRED or reservationId for CONFLICT.' },
}"
/>

| Code               | Status         | Raised when                                                              |
| ------------------ | -------------- | ------------------------------------------------------------------------ |
| `INVALID_CONFIG`   | 0              | an option is invalid (at creation, or a bad `requestId` value)           |
| `INVALID_PATH`     | 0              | a route segment is not plain                                             |
| `INVALID_TOKEN`    | 0              | `token` is not a bearer shape the server accepts                         |
| `INVALID_INPUT`    | 0              | the input does not serialize as JSON                                     |
| `INVALID_RESPONSE` | the response's | the body is not a `{ data }` or `{ error }` envelope                     |
| `HTTP_ERROR`       | the response's | a non-2xx status without an error envelope                               |
| anything else      | the response's | the server's own code; see the [error reference](/docs/reference/errors) |

```ts title="Handling errors"
import { ComputeClientError } from 'better-compute/client';

try {
  await client.reservations.create(input);
} catch (error) {
  if (!(error instanceof ComputeClientError)) throw error;
  switch (error.code) {
    case 'CONFIRMATION_REQUIRED':
      return { waitFor: error.details?.confirmationId, retryWith: input.idempotencyKey };
    case 'QUOTA_EXCEEDED':
      return { denied: error.message };
    default:
      console.error(`${error.code} (${error.requestId}): ${error.message}`);
      throw error;
  }
}
```

**Rate limits.** With `retryRateLimited`, a `RATE_LIMITED` answer is retried once after `retryAfterMs`, but only
when that wait is at most `maxWaitMs`. Aborting the call's `signal` during the wait rejects at once without a second
request. Longer waits surface as the error, with `retryAfterMs` set, for you to schedule.

**Sign-in.** `onUnauthenticated` runs once for every `UNAUTHENTICATED` answer, after the error is created. An
exception thrown from it is swallowed; it never changes what the caller sees.

## Idempotent writes and confirmation [#idempotent-writes-and-confirmation]

Methods that create things take an `idempotencyKey`. Generate one per attempt, keep it across retries, and pass the
same input again. The server returns the stored result for a repeated key, so a retry after a network error never
books twice.

An agent in a delegated Better IAM session may get `CONFIRMATION_REQUIRED` when the person it acts for holds the
action back. Wait for the person, then call again with the same input and key: the retry reaches the same draft and
is decided again. [Delegated confirmation](/docs/guides/iam/delegated-confirmation#approve-and-retry) has the full
polling loop with `@better-iam/client`.

## Watching events [#watching-events]

`$watch(options?)` follows the [watch stream](/docs/guides/events/watch) (`GET {basePath}/watch`, served by the
events module's `handleWatch`) as an async iterable. It reads Server-Sent Events over `fetch` with the client's
`token`, `headers` and `credentials`, so it can send an `Authorization` header, which the browser's `EventSource`
cannot:

```ts title="Wait for a reservation to become active"
const controller = new AbortController();
for await (const event of client.$watch({ types: ['reservation.*'], signal: controller.signal })) {
  console.log(event.type, event.subject, event.sequence);
  if (event.type === 'reservation.active' && event.subject === `reservation/${reservationId}`) break;
}
```

<TypeTable
  type="{
  types: { type: 'readonly string[]', description: 'Event type patterns such as reservation.* or node.connected, sent as ?types=. Every type when absent.' },
  tenantId: { type: 'string', description: &#x22;The tenant whose events to follow, sent as ?tenantId= for the host's watchAuthorizer to decide (e2fe6f8). The caller's own tenant when absent.&#x22; },
  lastEventId: { type: 'string', description: &#x22;Resume after this cursor: an event's sequence (the SSE id).&#x22; },
  signal: { type: 'AbortSignal', description: 'Ends the iteration; so does break or return().' },
  path: { type: 'string', description: 'The watch route, an absolute path below the origin.', default: &#x22;'{basePath}/watch'&#x22; },
  maxBackoffMs: { type: 'number', description: 'The longest wait between reconnects after failures.', default: '30000' },
}"
/>

Each event is `{ id, type, tenantId, subject, time?, sequence?, data? }`, in commit order, with the SSE `id` as its
`sequence`; save it and pass it back as `lastEventId` to resume after a restart. The iterator keeps the last id and
reconnects with `Last-Event-ID` whenever the stream ends: after the server's `retry:` delay when the server closed it
(its maximum duration), and with doubling back-off up to `maxBackoffMs` after network failures or other non-2xx
answers. A 401 ends it with `ComputeClientError('UNAUTHENTICATED')` (and calls `onUnauthenticated`), a 403 with
`ACCESS_DENIED`. `event: error` blocks and malformed data are skipped. The stream is always the caller's tenant: the
server's `authorize` decides which one.

The default path assumes the stream is mounted under the API's base path, as the console's
`/api/compute/watch` route is. A host that serves it at `/watch` (the `serve` host export in
[watch](/docs/guides/events/watch#mount-the-endpoint)) passes `path: '/watch'`.

**Callbacks instead of a loop.** `watchEvents` from `@better-compute/console-kit` reads the same stream with
`onEvent` and `onStatus` callbacks and a `stop()`, which suits UI stores. It works in browsers and in Node.js 22:

```ts title="Follow reservation events"
import { watchEvents } from '@better-compute/console-kit';

const watcher = watchEvents({
  url: 'https://compute.example.com/watch', // wherever your host mounts handleWatch
  types: ['reservation.*'],
  headers: async () => ({ authorization: `Bearer ${await session.getToken()}` }),
  onEvent: (event) => console.log(event.type, event.subject, event.sequence),
  onStatus: (status) => console.log('watch', status),
});
// later
watcher.stop();
await watcher.done;
```

For history, page `client.events.list({ tenantId, cursor })`. [Watch](/docs/guides/events/watch) covers the stream,
cursors and authorization, and [events](/docs/guides/events) the event shape.

## Tunnels [#tunnels]

The client also carries the caller side of a [tunnel](/docs/guides/access/tunnels): after `tunnels.open` returns a
single-use ticket and its `path`, `connectTunnel` opens the WebSocket and exchanges `TunnelFrame`s. The frames are
encoded by hand for the few fields a caller uses, so the client needs no protobuf runtime.

```ts title="An exec session"
import { connectTunnel, tunnelUrl } from 'better-compute/client';

const opened = await client.$request<{ path: string }>('tunnels/open', {
  tenantId: 'acme',
  instanceId: 'inst_…',
  op: 'exec',
});
const tunnel = connectTunnel({ url: tunnelUrl('https://compute.example.com', '/api/compute', opened.path) });
tunnel.onData((data, channel) => process[channel === 1 ? 'stderr' : 'stdout'].write(data));
await tunnel.ready; // the node verified the ticket
tunnel.resize(40, 120);
tunnel.write('nvidia-smi\n');
const { exitCode, reason } = await tunnel.exited;
```

| Member               | Does                                                                                                               |
| -------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `ready`              | resolves when the node opened its end; rejects if the socket closes first                                          |
| `exited`             | resolves with `{ exitCode, reason }` when the tunnel ends (`-1` and the close reason when the node sent no status) |
| `write(data)`        | sends bytes or UTF-8 text to stdin or the byte stream                                                              |
| `resize(rows, cols)` | sets the terminal size for exec and console sessions                                                               |
| `onData(handler)`    | receives output: channel 0 is stdout or the byte stream, 1 is stderr                                               |
| `close()`            | ends the session; the node reports its exit status, then the socket closes                                         |

The ticket is valid for 60 seconds and once, so open the WebSocket right after `tunnels.open`. `tunnelUrl` turns
`https` into `wss`. Outside a runtime with a global `WebSocket`, pass one: `connectTunnel({ url, WebSocket })`.
Console-kit's `createTerminalSession` wraps all of this for browser terminals.

## Typing [#typing]

The exported types let you name what the client infers:

| Type                                       | Is                                                                                      |
| ------------------------------------------ | --------------------------------------------------------------------------------------- |
| `ComputeClient<typeof compute>`            | the whole client: its groups, `plugins`, `$request` and `$watch`                        |
| `ClientGroups<Api>`                        | the groups of an API type, each method mapped to `(input, options?) => Promise<Output>` |
| `ComputeClientOptions`                     | the options above                                                                       |
| `ComputeCallOptions`                       | `{ signal?, headers? }` per call                                                        |
| `ClientTransport`                          | `{ $request, $watch }`; console-kit's stores and the MCP tools use only `$request`      |
| `ComputeWatchOptions`, `ComputeWatchEvent` | `$watch`'s options and the events it yields                                             |

A server parameter counts as the credential, and is dropped from the client signature, when every key it has is a
key of core's `Credential` (`headers`, `token`). `tests/client-types.test.ts`, compiled with
`tests/tsconfig.client.json`, pins the inference through modules and plugins: `client.instances` is an error
without `vps()`, and plugin methods keep their input and output types.

Without a server type to import (another repository, a plain script), the groups are not typed: call routes with
`$request<Output>(path, input)`, which is also all that console-kit's stores and the MCP tools use.


# Architecture (/docs/guides/concepts/architecture)

> The planes and components of Better Compute, the api, gateway and worker process modes, and how joins, reservations, bare metal and VPS flow through them.



Better Compute separates who may ask for something from where the answer is kept and from the machines that carry
it out. Requests arrive over HTTP and are decided once by Better IAM; the answer is written to PostgreSQL; daemons
learn about it through long-lived sessions they opened themselves. This page describes those parts and how the four
main flows move through them.

## Planes [#planes]

| Plane            | Components                                                                                                                                                                                           | Trust                                                                                               |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Interface        | The typed client, the CLI, MCP tools, the console kit, webhook receivers                                                                                                                             | Untrusted callers holding Better IAM credentials                                                    |
| Identity         | Better IAM, embedded in the api and worker processes or remote, plus Better Compute's IAM plugin                                                                                                     | Trusted                                                                                             |
| Control          | `api`: stateless HTTP, admission, JoinService, the secret broker. `gateway`: daemon sessions and tunnels. `worker`: reconcilers, placer, workflows, event sinks, usage, IAM jobs, the CommandSweeper | Trusted; the gateway only relays                                                                    |
| Data             | PostgreSQL (with PgBouncer in transaction mode), an object store, a time-series database, a KMS or Vault                                                                                             | Trusted                                                                                             |
| Site             | `bcd site`: proxy DHCP, TFTP, iPXE and HTTP boot, image cache, BMC access; one active boot server per site                                                                                           | High value; secrets are sealed per request                                                          |
| Node and ramdisk | `bcd node` with its runtimes and CDI; `bcd ramdisk` during commissioning                                                                                                                             | Trusted while the platform holds root; the ramdisk gets a restricted certificate of at most 2 hours |

## Components [#components]

<Mermaid
  chart="flowchart LR
  subgraph Interface
    CL[&#x22;Client, CLI, MCP, console&#x22;]
  end
  subgraph Control[&#x22;Control plane&#x22;]
    API[&#x22;api: authn, admission, JoinService&#x22;]
    WK[&#x22;worker: reconcilers, sinks, IAM jobs&#x22;]
    GW[&#x22;gateway: sessions, dispatch, tunnels&#x22;]
  end
  IAM[&#x22;Better IAM&#x22;]
  PG[(&#x22;PostgreSQL&#x22;)]
  TS[(&#x22;Time-series DB&#x22;)]
  subgraph Machines
    N[&#x22;bcd node&#x22;]
    S[&#x22;bcd site&#x22;]
    R[&#x22;bcd ramdisk&#x22;]
  end
  CL -->|&#x22;POST /api/compute&#x22;| API
  API -->|&#x22;one decision per mutation&#x22;| IAM
  API --> PG
  WK --> PG
  GW --> PG
  GW -->|&#x22;Join, Renew, Secret&#x22;| API
  N -->|&#x22;gRPC over mTLS, 443&#x22;| GW
  S --> GW
  R --> GW
  N -.->|&#x22;OTLP&#x22;| TS"
/>

* **PostgreSQL** holds nodes, sessions, devices, bookings, reservations, commands, the work queue, the outbox,
  leases and the authorization projection. It is the only place state is authoritative.
* **Gateways** hold daemon sessions in memory and write liveness to the database in batches. They sit behind an L4
  load balancer with TLS passthrough, because mutual TLS terminates in the gateway.
* **Workers** claim work from the database with `SKIP LOCKED` under shard and pool leases. Optional pokes (a
  notifier) wake gateways sooner, but every loop also converges without them.
* **Telemetry** goes from the daemon or `dcgm-exporter` to a time-series database over OTLP or remote write, never
  through PostgreSQL or the session.

## The control plane in your process [#the-control-plane-in-your-process]

`betterCompute(options)` is synchronous and side-effect free: it validates options (`INVALID_CONFIG` otherwise) and
registers `system`, the server's own groups, the built-in node controllers and the always-on `inventory` and
`capacity` modules, then your `modules` and `plugins`. Nothing touches the database or Better IAM until you call it.

| Member                                                              | What it is                                                                                                                                      |
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `api`                                                               | Every method as `(credential, input)`, through the same envelope as HTTP. It also drives the typed client's inference.                          |
| `handler`                                                           | The fetch transport: `(request: Request) => Promise<Response>`.                                                                                 |
| `nodeHandler`                                                       | The Node.js transport, for `createServer(compute.nodeHandler)`.                                                                                 |
| `gatewayServices`                                                   | JoinService's `join` and `renew` and the secret broker, consumed by `createGateway`.                                                            |
| `initialize()`                                                      | Checks every method's IAM action against the merged catalog, runs migrations, anchors the clock and awaits each module's `start()`. Idempotent. |
| `start({ roles })`, `stop()`                                        | Run and stop the worker roles on timers, under their leases.                                                                                    |
| `jobs`                                                              | One deterministic pass per `tick()`, for tests and the simulator.                                                                               |
| `events`                                                            | In-process subscribers of committed events.                                                                                                     |
| `context`, `registry`, `caches`, `authz`, `selfCheck()`, `endpoint` | The module context, the method registry, cache invalidation, the projection, read-only health checks, and where the handler is mounted.         |

```ts title="server.ts"
import { createServer } from 'node:http';
import { compute } from './compute';

await compute.initialize(); // registry checks, migrations, module start()
createServer(compute.nodeHandler).listen(3000); // POST /api/compute/{group}/{method}
await compute.start({ roles: ['controllers', 'sinks', 'iam'] });
```

The server package never imports the store, the IAM bridge, the gateway, or the injected `provision`, `metal` and
`vps` modules; they arrive through options, and `scripts/check-deps.mjs` fails CI if that changes.

## Process modes [#process-modes]

One configuration runs in any process mode, so you scale each part separately against one database. With the CLI,
`better-compute serve --roles` picks the mode:

| Mode      | Runs                                                                                                                                                 | Scales by                                        | `serve --roles`                   |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------------------- |
| `api`     | Stateless authentication and authorization, drafts, the admission transaction with inline placement, the outbox writes, JoinService                  | Requests, behind an L7 load balancer             | `api`                             |
| `gateway` | Daemon sessions in memory, liveness, dispatch, result and event ingestion, `Join`/`Renew`/`Secret` hand-off, tunnels                                 | Streams (at least three per region planned)      | `gateway`                         |
| `worker`  | `controllers` (reconcilers, workflows, the CommandSweeper), `sinks` (outbox dispatch), `iam` (Better IAM's jobs under singleton leases), `scheduler` | 256 shards spread over the workers               | `controllers,scheduler,sinks,iam` |
| `all`     | The API and every worker role in one process, plus the gateway when the configuration's `host.gateway` is set                                        | Nothing; for development and small installations | `all` (the default)               |

```bash
better-compute migrate                                                    # once per deploy
better-compute serve --roles api --port 3000 --no-migrate
better-compute serve --roles controllers,scheduler,sinks,iam --no-migrate
better-compute serve --roles gateway --no-migrate
```

Leases and shard ownership keep every job single-writer, so you can run several processes of each mode. The
`scheduler` role runs the reconcilers that declare `role: 'scheduler'` (placers), under their own rendezvous
membership (worker rows `scheduler/<id>`, leases `scheduler/shard:<n>`), so a scheduler-only process never takes node
shards from the controllers and each role spreads the 256 shards over its own processes. The pool-lease placer
(`capacity.placer`, used with `capacity: { placer: 'pool-lease' }`) is such a reconciler, so with that placer at
least one process must run the `scheduler` role. See [deployment](/docs/operations/deployment) and
[scaling](/docs/operations/scaling).

## Nodes only dial out [#nodes-only-dial-out]

A daemon opens no listening port. It dials `gw.<region>:443` and keeps one HTTP/2 connection carrying the `Session`
stream and, from v0.3, on-demand `Tunnel` streams. Mutual TLS terminates in the gateway, which checks the certificate
chain, revocation and role, and that the SPIFFE ID matches the node id in `Hello`.

The gateway package holds no CA and no signing key. It hands `Join`, `Renew` and `SecretRequest` to
`gatewayServices`, the control plane's JoinService and secret broker, through dependency inversion. The design puts
those in the api process behind internal mutual TLS; the `serve --roles gateway` host of this release constructs them
in the gateway's own process from the shared configuration. Either way, a compromised gateway can delay or drop frames
but cannot forge what matters: renewals need proof of possession of the node key, join tokens are sealed to
JoinService's key, secrets travel as JWE to a per-request key, and attested results carry the node's signature. See
[the gateway](/docs/daemon/gateway) and [security](/docs/operations/security).

## The four flows [#the-four-flows]

### A daemon joins (v0.1) [#a-daemon-joins-v01]

1. An operator mints a join token with [`joinTokens.create`](/docs/reference/api/join-tokens#create)
   (`compute:join-tokens:create` on the pool). The token, `bcjt_<id>.<secret>`, is shown once; only a hash is stored.
2. The install script puts `bcd`, its pinned configuration and the JoinService encryption key on the server.
3. `bcd join` sends `Join` with the token sealed to JoinService's key, a CSR and hardware claims.
4. JoinService runs the attestor chain, computes the hardware key, enforces the token's `expect` and reclaim rules,
   inserts the node as `adopting`, decrements the token and signs a 24-hour SPIFFE certificate.
5. The daemon opens its `Session`; the gateway bumps the session epoch and asks for inventory sections.
6. The inventory controller derives devices and conditions; the lifecycle controller runs the pool's acceptance
   checks (`diag.nvml`). On pass the node is `adopted` and receives desired-state generation 1.

No step calls Better IAM. See [joining](/docs/guides/nodes/joining).

### A user or agent reserves GPUs (v0.1) [#a-user-or-agent-reserves-gpus-v01]

This is the admission sequence every request-as-resource `create` follows:

<Mermaid
  chart="sequenceDiagram
  participant C as Caller
  participant A as api
  participant DB as PostgreSQL
  participant I as Better IAM
  C->>A: reservations.create(idempotencyKey)
  A->>I: authenticate (cached 30 s)
  A->>A: tenant pre-check
  A->>DB: tx 1 - idempotency lookup, draft, projection row
  A->>I: require compute:reservations:create
  A->>DB: tx 2 - quota, place, bookings, audit row
  A-->>C: reservation (placed)"
/>

A caller outside the input's tenant, and not in the root tenant, gets `ACCESS_DENIED` before any database access. A
repeated idempotency key from the same identity replays the stored result; a different request under the same key is
`CONFLICT`. The placer then books atoms in the second transaction, and the reservation binds through the node's
desired state. See [reservations](/docs/guides/capacity/reservations) and
[placement](/docs/guides/capacity/placement).

### Bare metal from PXE to ready (v0.2) [#bare-metal-from-pxe-to-ready-v02]

A site agent enrolled with a site token holds the site's boot lease and serves proxy DHCP and iPXE.
[`metal.enroll`](/docs/reference/api/metal#enroll) stores the BMC credentials as a secret reference; the machine is
inspected over Redfish, PXE-booted into `bcd ramdisk`, diagnosed and wiped (node-signed results), and reaches
`ready`. [`metal.deploy`](/docs/reference/api/metal#deploy) streams an image to disk, writes cloud-init with a
single-use token, and kexecs into the installed system, which rejoins as the same node. The control plane for this
flow has landed. `bcd site` and `bcd ramdisk` are written but not released and covered only by Go unit tests, so the
flow's tests run against simulated agents. See
[bare metal](/docs/guides/bare-metal).

### A VPS instance (v0.3) [#a-vps-instance-v03]

[`instances.create`](/docs/reference/api/instances#create) follows the same admission sequence on
`compute-instance`. On a hosted pool (`class: 'vps-host'`) the placer claims vCPU, memory, disk and optionally a GPU,
and the instance joins the node's desired state for the daemon to run. On a provider pool (`class: 'provider'`,
`agent: 'none'`) a workflow calls the provider's API, and no platform agent ever runs inside a tenant's VM. Later
operations change only the instance's desired state. See [VPS](/docs/guides/vps).


# Consistency (/docs/guides/concepts/consistency)

> Why double-booking cannot happen in Better Compute, and how one clock, exclusion constraints, the outbox, the work queue and fencing keep the fleet consistent.



A GPU fleet is a distributed system with slow, unreliable members: daemons that reconnect, workers that crash
mid-transaction, gateways that lose their database for thirty seconds. Better Compute stays correct through all of
that by putting its invariants where they cannot be bypassed, in PostgreSQL, and by making every message safe to
deliver twice. This page explains the mechanisms and what they guarantee.

## PostgreSQL is the truth [#postgresql-is-the-truth]

Everything authoritative is a row. Everything in memory is a cache that can be rebuilt:

| In memory                                                                | Rebuilt from                                               |
| ------------------------------------------------------------------------ | ---------------------------------------------------------- |
| A gateway's sessions and liveness ages                                   | `node_sessions`, and the daemons reconnecting              |
| The IAM resolver's authorization projection                              | `authz_projection`, tailed through `authz_changes`         |
| The 30-second authentication cache and the 10-second read-decision cache | Better IAM, on the next miss                               |
| Pokes that wake a gateway or a worker early                              | Nothing: every loop also polls, so a lost poke only delays |

Aggregates carry a `version`. A transition is `UPDATE … WHERE id = $id AND version = $v`; zero rows means
someone else changed the record first, and the caller gets `CONFLICT`. Which transitions are legal comes from the
state tables in `@better-compute/core` (`reservationStateTable`, `commandStateTable`, `provisionStateTable`, …), so
the rules live in one place.

## One clock [#one-clock]

Every time comparison in SQL (lease expiry, `not_before`, `not_after`, booking ranges, liveness) reads
`compute_now()`, never `now()`. The first migration defines it:

```sql title="core/0001_clock"
-- defines compute_now()
CREATE EXTENSION IF NOT EXISTS btree_gist;
-- Tests only: computeFixture({ clock: 'sim' }) points compute_now() here (useSimClock).
CREATE TABLE compute_sim_clock (id int PRIMARY KEY CHECK (id = 1), ts timestamptz NOT NULL);
CREATE FUNCTION compute_now() RETURNS timestamptz LANGUAGE sql STABLE AS $$ SELECT now() $$;
```

* **Simulated time.** Tests and the simulator replace the body with `SELECT ts FROM compute_sim_clock WHERE id = 1`
  (`useSimClock` and `setSimClock` in `@better-compute/postgres`; `computeFixture()` does it by default). Leases,
  deadlines and liveness then all expire in simulated time, deterministically.
* **Enforced.** `scripts/check-sql-now.mjs` fails CI on `now()`, `CURRENT_TIMESTAMP`, `LOCALTIMESTAMP`,
  `transaction_timestamp()`, `statement_timestamp()` or `clock_timestamp()` in any string in `packages/*/src`,
  except in the migration that defines `compute_now()` and on lines marked `check-sql-now: allow`.
* **TypeScript time.** Business time in TypeScript comes from a `Clock` anchored to `compute_now()`: sampled at
  `initialize()` and at every lease renewal. A process that drifts more than one second logs `ClockSkew`, and
  `selfCheck()` reports `clock.skew`.
* **Gateways report ages, not timestamps.** A gateway flushes "last seen 3.2 seconds ago", and the database turns it
  into `compute_now() - 3.2 s`, so a gateway with a wrong clock cannot fake liveness.

## Why double-booking cannot happen [#why-double-booking-cannot-happen]

Every claim on a device is a row in `device_bookings`, and the database refuses overlapping rows whatever the
scheduler, the API or a buggy module does:

```sql
CREATE TABLE device_bookings (
  device_id text NOT NULL REFERENCES devices,
  allocation_id text NOT NULL,
  during tstzrange NOT NULL CHECK (NOT isempty(during)),
  EXCLUDE USING gist (device_id WITH =, during WITH &&)
);
```

Five rules together close every gap:

1. **Device-time exclusivity.** The exclusion constraint allows one live booking per atom
   per instant.
2. **The atom rule.** A flat constraint cannot see that a MIG instance overlaps its parent GPU, so parents and
   children are never both bookable: a MIG GPU exposes only its instances, a time-slice or MPS GPU only its slots
   (`<uuid>/s<k>`). Changing a GPU's mode or layout books every atom of the GPU, parent included, for the change
   window, so the change commits only when nothing on the GPU is booked.
3. **Scalars.** vCPU, memory and disk are counters, changed only by guarded updates
   (`allocated + x <= floor(allocatable * ratio)`) under row locks, with a `CHECK` constraint as the backstop.
4. **Eligibility is linearizable.** The placer locks candidate node rows and the chosen atoms `FOR SHARE`. Cordoning,
   tainting, marking a node unreachable and quarantining a GPU are updates to those same rows, so whichever
   transaction commits second sees the first.
5. **Quota.** `UPDATE quota_usage SET used = used + n WHERE … AND used + n <= max` in the same transaction; zero
   rows is `QUOTA_EXCEEDED`. A reservation holds its charge in every non-terminal state, `pending` included.

As defense in depth, the daemon also refuses a device assignment that overlaps one it already holds, and reports the
allocation as failed. The capacity torture test
attacks all five rules at once: 1,000 concurrent operations with cordons, taints, health flips and releases, checking
every invariant.

### Conflicts do not abort the request [#conflicts-do-not-abort-the-request]

The guarded statements return zero rows instead of raising a constraint error, and bookings use
`INSERT … ON CONFLICT DO NOTHING`, which honors exclusion constraints. Each placement attempt runs under a savepoint:
a node, atom or counter conflict rolls back to it and replans without the conflicting node, at most three times.
After that the reservation becomes `pending` with an `Unschedulable` condition and is retried every 30 seconds. A
quota failure is final for the request.

Transactions that lock several of these tables take them in one order, which rules out deadlocks between reserve
and release: `quota_usage`, then `pools`, `nodes`, `devices`, `node_resources`, `pool_capacity_rollup`. Reserving
charges quota first; releasing refunds it first.

## Side effects leave through the outbox [#side-effects-leave-through-the-outbox]

Nothing external happens inside a transaction. Events are rows in `outbox`, written in the same transaction as the
change they describe, as `{ id, type, tenantId, subject, data, time }`. Webhook delivery, the SSE stream and
in-process subscribers read them afterwards. If the transaction rolls back, the event never existed.

Commit order is not sequence order, so a naive `WHERE seq > $cursor` reader could skip a row that commits late. The
reader only takes rows whose transaction is older than every transaction still running:

```sql
SELECT … FROM outbox o
 WHERE o.txid < pg_snapshot_xmin(pg_current_snapshot())
   AND (o.txid, o.seq) > ($cursorTxid, $cursorSeq)
 ORDER BY o.txid, o.seq
```

Nothing can then appear behind the cursor. Each event sink is delivered by the worker holding that sink's
`singleton:outbox-dispatch:<sink id>` lease, which saves the sink's position in `outbox_cursors` after every batch,
so a new lease holder resumes where the last one stopped. Delivery is at least once: consumers deduplicate by event `id`. The same cursor shape
serves the SSE stream's `Last-Event-ID` and the `authz_changes` feed. See [events](/docs/guides/events).

## The work queue never loses a wake-up [#the-work-queue-never-loses-a-wake-up]

Reconcilers are woken through `work_queue`, enqueued in the transaction that changed the state:

```sql
INSERT INTO work_queue (kind, key, shard, gen, not_before) VALUES (…)
  ON CONFLICT (kind, key) DO UPDATE SET gen = work_queue.gen + 1, …
```

A worker claims due rows of its own shards with `FOR UPDATE SKIP LOCKED` and stamps `claimed_until`. When the run
succeeds it deletes the row only if `gen` still equals the value it claimed; a key dirtied again during the run
survives and runs again. Because reconcilers are level-triggered and also resync everything they own every few
minutes, even a lost row costs only time. See [desired state and commands](/docs/guides/concepts/desired-state-and-commands).

## At least once, made safe [#at-least-once-made-safe]

Every hop may deliver twice, so every receiver can tell a duplicate from new work:

| Mechanism                                            | Where                                                                                                                           | What it prevents                                                     |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Idempotency keys scoped to `(tenant, identity, key)` | `reservations.create`, `reservations.extend`, `instances.create`, `commands.run`; optional on `sites.create` and `pools.create` | A retried request booking twice; another identity squatting a key    |
| `UNIQUE (node_id, idempotency_key)`                  | Commands (`wf:<workflowId>:<stepId>` for workflow steps)                                                                        | A step sending its command twice                                     |
| `cmp:<allocationId>:<windowStart>`                   | Usage windows sent to Better IAM billing                                                                                        | Double billing across worker restarts                                |
| Session epoch                                        | Routing and liveness writes carry `AND session_epoch = $epoch`; dispatch locks the session row                                  | A zombie gateway routing or marking a node after it lost the session |
| Lease epoch                                          | The first statement of every fenced transaction                                                                                 | A worker acting after its lease was taken over                       |
| Generation                                           | `min_generation` on commands, `observedGeneration` on nodes                                                                     | A command running against a spec the node has not applied            |
| Journal id                                           | A new daemon journal fails in-flight commands that are not retry-safe                                                           | A wipe or deploy replayed after the daemon lost its memory           |

### Leases and fencing [#leases-and-fencing]

Leases are rows in `leases`, not advisory locks: session-level advisory locks break under PgBouncer's transaction
pooling, and a table gives both a time to live and an epoch.

```sql
UPDATE leases SET owner = $me,
       epoch = CASE WHEN owner = $me THEN epoch ELSE epoch + 1 END,
       expires_at = compute_now() + make_interval(secs => $ttl)
 WHERE name = $name AND (owner = $me OR expires_at < compute_now())
RETURNING epoch;

-- the first statement of each fenced transaction
SELECT 1 FROM leases WHERE name = $name AND owner = $me AND epoch = $epoch
   AND expires_at > compute_now() FOR SHARE;
```

Shard, pool, site and singleton leases live 15 seconds and are renewed every 5; gateway leases live 20 seconds. A
takeover increments the epoch and blocks until the old holder's `FOR SHARE` transaction ends, so a zombie either
still sees its lease as valid or aborts. A plain `EXISTS` check would not be a fence under READ COMMITTED; the row
lock is.

## Disconnected is not dead [#disconnected-is-not-dead]

When a daemon stops talking, Better Compute changes a condition, not ownership:

| Transition                      | When                                                                                                                                                                                                                                               | What happens                                                                                                          |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `connected` to `disconnected`   | The session closes or its gateway's lease expires                                                                                                                                                                                                  | Nothing is released                                                                                                   |
| `disconnected` to `unreachable` | 40 seconds without a session, plus grace: 120 s after a gateway loss, 5 minutes after a database or controller outage. A node that joined but never opened a session counts from its join (its newest certificate), not from the beginning of time | The node leaves placement (`unreachable` joins `noScheduleReasons`, `NodeUnreachable` is raised); nothing is released |
| `unreachable` to `lost`         | The pool's `lostAfter`, only for `provider` and `vps-host` pools that set it                                                                                                                                                                       | Never automatic for bare metal                                                                                        |

Workloads on an isolated node keep running, and its reservations keep their devices and keep billing on reserved
time. A daemon that comes back supersedes its old session by epoch and replays its journal. If the machine really is
gone, an operator decides, by draining, releasing or retiring it. See [failure modes](/docs/daemon/failure-modes).


# Desired state and commands (/docs/guides/concepts/desired-state-and-commands)

> How Better Compute tells a node what to do, with generation-numbered NodeSpecs, level-triggered reconcilers and durable commands for one-shot actions.



The control plane has two ways to make a node do something. It can change what the node should look like, its
desired state, and let the node converge on it. Or it can queue a command: a one-shot action with a deadline. Choosing
the right one is what keeps the fleet correct through reconnects, restarts and partitions.

## The rule [#the-rule]

If losing the message would leave the node in the wrong steady state, it is desired state. If running it late could
do harm, it is a command with a deadline (`not_after`), plus a minimum generation when it presupposes a spec.

| Desired state (NodeSpec)                                  | Commands                                    |
| --------------------------------------------------------- | ------------------------------------------- |
| Device assignments of active reservations                 | `node.reboot`                               |
| VPS instances and whether they should run                 | `diag.nvml`, `diag.dcgm`                    |
| Host configuration units (drivers, MIG layout)            | `inventory.refresh`, `logs.collect`         |
| SSH CA keys and the exec-ticket JWKS                      | `gpu.reset`, `disk.wipe`, `deploy.image`    |
| Telemetry endpoint, daemon version, the site's boot lease | `boot.kexec`, `bmc.power`, `daemon.upgrade` |

A reservation's GPUs are desired state because a node that restarts, or reconnects after a partition, must still
expose exactly the booked devices. A reboot is a command because a reboot delivered an hour late, after the node was
leased to someone else, would do damage.

## Desired state: NodeSpec [#desired-state-nodespec]

A NodeSpec is the full desired state of one node, in sections:

| Section     | Holds                                                                                                             | Contributed by                                                                                       |
| ----------- | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `host`      | Config units, such as the NVIDIA driver or a MIG layout                                                           | `provision` (profiles' `hostConfig`)                                                                 |
| `devices`   | Device assignments: allocation id, device ids, optional `until`                                                   | `capacity` (active reservations and instances)                                                       |
| `instances` | VPS instances with runtime, flavor, image and desired state                                                       | `vps`                                                                                                |
| `access`    | SSH CA public keys, the exec-ticket JWKS                                                                          | tunnels (v0.3)                                                                                       |
| `telemetry` | OTLP endpoint and export interval                                                                                 | no built-in module yet; your own module can set it                                                   |
| `daemon`    | The bcd version, URL, SHA-256 and signature to run                                                                | `rollout`                                                                                            |
| `site`      | Whether this site agent holds the boot lease, and its epoch; the images its cache pre-warms (`cacheImages`, v0.2) | the site-boot controller (`boot`); provision (`cacheImages`, from the pools' `policy.prewarmImages`) |

Modules contribute sections through `nodeSpec(context, node)` (a `NodeSpecPart`), and the NodeSpec controller merges
the contributions in registration order; `site` merges field by field, so modules can contribute different fields
of it (see [NodeSpec sections](/docs/extending/modules#nodespec-sections)). It stores the result in `nodes.spec` as the protobuf JSON of `bettercompute.agent.v1.NodeSpec`
and increments `nodes.generation` only when the canonical JSON differs from what is stored. Desired state applies
once a node is `adopted` or `deployed`: adoption writes generation 1.

### How a change reaches the node [#how-a-change-reaches-the-node]

<Mermaid
  chart="sequenceDiagram
  participant M as Module (tx)
  participant DB as PostgreSQL
  participant R as NodeSpec controller
  participant G as Gateway
  participant D as bcd
  M->>DB: domain change + work_queue nodespec
  R->>DB: merge sections, compare, generation + 1
  R-->>G: poke gw after commit (best effort)
  G->>D: DesiredState(generation)
  D->>D: converge every section
  D->>G: StatusReport(observedGeneration)
  G->>DB: observed_generation = GREATEST(old, new)"
/>

1. Whatever changes a node's desired state calls `ctx.desired.invalidate(tx, nodeId)` in the same transaction,
   which enqueues `nodespec` work for the node. Nothing is pushed yet.
2. The NodeSpec controller recomputes the spec, and if it changed, writes the new generation and, after commit,
   pokes the node's gateway.
3. The gateway sends `DesiredState`. A daemon that reconnects behind the current generation gets the full spec
   after `Welcome`, so a lost poke delays convergence but never loses it.
4. The daemon converges each section with an idempotent plan and apply, and reports its `observedGeneration`. It
   advances only when every section has converged or failed terminally. The database keeps the highest value
   reported, so a late report cannot move it backwards.

A node is schedulable only once `observedGeneration` is at least 1: the placer never books devices on a node that
has not applied any desired state. `nodes.get` shows both numbers; `generation` greater than
`observedGeneration` means the node is still converging.

## Reconcilers [#reconcilers]

Controllers are reconcilers: level-triggered loops that read the current state, compare
it with the desired state, and act on the difference. They never trust that they saw every event, so a missed event,
a crash mid-run or a restart only delays them.

```ts
interface Reconciler {
  name: string;
  shardBy: 'node' | 'pool' | 'none';
  /** Level-triggered: SYNC_AGAIN re-queues the key. */
  run(context: ReconcileContext, key: string): Promise<'DONE' | 'SYNC_AGAIN'>;
  /** Keys also enqueued on a fixed period, e.g. [{ key: 'all', every: '5m' }]. */
  periodic?: readonly { key: string; every: Duration }[];
}
```

* **Edge-triggered wake-ups, level-triggered work.** A state change enqueues `(kind, key)` in the
  work queue in the same transaction. Enqueueing a key that is already queued bumps its
  generation instead of adding a row, and a successful run deletes the row only if the generation did not move, so a
  key dirtied again during its run is processed again. No wake-up is lost.
* **Resync.** Every worker re-enqueues the keys it owns every 5 minutes plus up to 5 minutes of jitter, in case
  anything slipped through.
* **Outcomes.** `DONE` removes the key; `SYNC_AGAIN` brings it back after 1 second; a thrown error backs off from 1
  second doubling to 5 minutes.
* **Sharding.** Node-sharded reconcilers spread keys over 256 shards (`nodes.shard`), and workers divide the shards
  among themselves by rendezvous hashing over the `workers` table, each shard under a fenced `shard:<n>` lease.
  `shardBy: 'none'` reconcilers run under a `singleton:<name>` lease.

With every module registered, the runtime runs these jobs: `node`, `nodespec`, `liveness`, `drain`, `site-boot`,
`inventory`, `inventory.hygiene`, `health`, `capacity.reservations`, `capacity.rollup`, `capacity.usage`,
`capacity.mig`, `workflow`, `metal`, `vps.instances`, `vps.hosts`, `vps.bootstraps`, `vps.usage`, `fleet.drift`,
`rollout`, `audit.checkpoint`, `events.fanout`, `events.deliver` and `events.prune`, plus the runtime's own
`command-sweeper`, `outbox-dispatch` and `event-subscribers`. The [modules reference](/docs/reference/modules)
lists them per module. In tests, `compute.jobs[name].tick()` runs one deterministic pass of any of them.

## Commands [#commands]

A command is a durable row in `commands`, written in the caller's transaction through the
command queue:

```ts
await context.commands.enqueue(tx, {
  nodeId,
  kind: 'inventory.refresh',
  payload: { sections: ['gpus'], rescan: true }, // the kind's payload message, as proto3 JSON
  idempotencyKey: `refresh:${nodeId}:${reason}`, // unique per node; a repeat returns the existing command
  notAfter: context.clock.now() + 10 * 60_000, // mandatory: the daemon rejects a command past it
});
```

The runtime gives it the node's next `seq` (from `nodes.next_cmd_seq`), encodes the payload into the protobuf
`CommandBody`, and pokes the node's gateway after commit. Routing data never goes in the payload: the target is
`nodeId`, and the gateway that currently holds the node's session is found by a join at dispatch time.

### Kinds and their flags [#kinds-and-their-flags]

Every command has a kind registered by the server or a module, and the kind carries
the rules for delivering it:

| Flag              | Meaning                                                                                                                                                                         |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lane`            | `control`, `provision` or `bulk`. Each lane has its own daemon worker pool (4, 2 and 1 workers) and its own credits (8, 4 and 2), so a long image stream never blocks a reboot. |
| `retrySafe`       | Safe to run again after a crash between acceptance and completion. `disk.wipe`, `deploy.image`, `gpu.reset` and `bmc.firmware` are not.                                         |
| `selfTerminating` | Ends the agent's process or boot (`node.reboot`, `boot.kexec`, `daemon.upgrade`). It completes when the node comes back with a new boot, never by being replayed.               |
| `attested`        | The result is signed with the node's key and verified before any state transition (wipes, diagnostics).                                                                         |
| `destructive`     | Must be signed by the control plane from v0.2.                                                                                                                                  |
| `timeout`         | How long the command may run after `not_after` before the sweeper fails it.                                                                                                     |
| `roles`           | Which agents may run it: `node`, `site`, `ramdisk`.                                                                                                                             |

The [command kinds reference](/docs/reference/command-kinds) lists every kind with its flags and payload message.

### Lifecycle [#lifecycle]

<Mermaid
  chart="stateDiagram-v2
  direction LR
  [*] --> pending
  pending --> dispatched: sent under a session epoch
  dispatched --> pending: BUSY, back off
  dispatched --> acked
  acked --> running
  running --> succeeded
  running --> failed
  dispatched --> rejected: refused before acceptance
  pending --> expired: past not_after
  pending --> cancelled"
/>

* A daemon checks a command before accepting it, in this order: `UNSUPPORTED` (another node, or an unknown or
  disallowed kind), `STALE_GENERATION` (its `min_generation` is ahead of the applied spec, after waiting until
  `not_after`), `EXPIRED` (past `not_after`), `BUSY` (no credits in the lane), and from v0.2 `BAD_SIGNATURE` for an
  unsigned destructive kind. Only `BUSY` is retried: the command goes back to `pending` with a back-off from 1 to 30
  seconds.
* The CommandSweeper expires `pending` commands past `not_after`, and fails in-flight ones past
  `not_after` plus the kind's `timeout` with reason `timeout`. Both wake the controller that owns the command.
* When a node reconnects with a new boot, in-flight self-terminating commands succeed; when it reconnects with a new
  journal, in-flight commands that are not retry-safe fail with `journal-lost` instead of running twice.

### Running commands through the API [#running-commands-through-the-api]

[`commands.run`](/docs/reference/api/commands#run) (`compute:nodes:exec` on the node) accepts an allowlist of kinds:
`diag.nvml`, `inventory.refresh`, `logs.collect` and `node.reboot`. Anything else is `UNSUPPORTED`. It takes an
`idempotencyKey`, scoped to your tenant and identity, and `notAfterSeconds` (default 600, from 10 to 3600):

```bash
better-compute api commands.run tenantId=$BETTER_COMPUTE_TENANT nodeId=node_… \
  kind=inventory.refresh payload:='{"sections": ["gpus"], "rescan": true}' idempotencyKey=refresh-1
better-compute api commands.get tenantId=$BETTER_COMPUTE_TENANT nodeId=node_… commandId=cmd_…
```

[`commands.cancel`](/docs/reference/api/commands#cancel) (v0.2, `compute:nodes:exec` on the node) cancels through the
same queue: a pending command becomes `cancelled` and is never sent, an in-flight one receives `Cancel` until the
daemon settles it, and a finished one is returned unchanged. See [node commands](/docs/guides/nodes/commands#cancelling)
and [delivery](/docs/daemon/delivery).

## When to use which [#when-to-use-which]

| You want to                                                          | Use                                                                               |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Give or take away devices, run or stop an instance, install a driver | Desired state: change the domain record and invalidate the node's spec            |
| Collect logs, re-read inventory, run a diagnostic now                | A command with a short `not_after`                                                |
| Reboot, wipe, reset a GPU                                            | A command; workflows send these as steps with `wf:<id>:<stepId>` idempotency keys |
| Upgrade the daemon across a pool                                     | Desired state (`daemon` section), rolled out in waves by the rollout module       |


# Domain model (/docs/guides/concepts/domain-model)

> Every entity Better Compute tracks, from regions and pools to atoms, bookings and commands, how they relate, and how each maps to a Better IAM resource type.



Every entity below is a TypeScript type in `@better-compute/core` and a table (or a column) in PostgreSQL. Records
carry epoch-millisecond timestamps, and every tenant-scoped record carries its `tenantId` or `ownerTenantId`: no
repository function reads an ambient tenant.

## How the entities relate [#how-the-entities-relate]

<Mermaid
  chart="erDiagram
  REGION ||--o{ SITE : contains
  SITE ||--o{ POOL : &#x22;schedules in&#x22;
  SITE ||--o{ NODE : hosts
  POOL |o--o{ NODE : groups
  POOL ||--o{ OFFERING : &#x22;offered as&#x22;
  NODE ||--o{ DEVICE : has
  DEVICE ||--o{ BOOKING : &#x22;atoms booked by&#x22;
  ALLOCATION ||--o{ BOOKING : holds
  RESERVATION ||--o{ ALLOCATION : &#x22;placed as&#x22;
  INSTANCE ||--o| ALLOCATION : &#x22;runs on&#x22;
  PROVIDER ||--o{ PROVIDER_ACCOUNT : &#x22;configured as&#x22;
  PROFILE ||--o{ WORKFLOW : &#x22;expanded into&#x22;
  WORKFLOW ||--o{ STEP : runs
  STEP |o--o| COMMAND : sends
  NODE ||--o{ COMMAND : receives
  NODE ||--|| NODESPEC : &#x22;desired state&#x22;
  TENANT ||--o{ PROJECT : owns"
/>

Ids read `<prefix>_<body>`, where a generated body is a 26-character ULID in lower-case Crockford base32, so ids of
one kind sort by creation time: `site_`, `pool_`, `node_`, `prj_`, `rsv_`, `alloc_`, `inst_`, `cmd_`, `wf_`, `img_`,
`prof_`, `jt_`, `pacct_`, `flv_`, `whk_`, `evt_` and others. Reservation and instance ids are derived from the
request instead of generated, as the section on reservations below explains.

## Regions, sites, pools and offerings [#regions-sites-pools-and-offerings]

A region is a geographic label in Better IAM's region-name grammar, stored on a site as
`iamRegion`.

A site is a boot and failure domain served by at least one `bcd site` agent. Its `kind` is
`shared` or `dedicated` (the platform's hardware) or `custom` (a tenant's own hardware). A site agent is itself a
`nodes` row with `role: 'site'`.

A pool is a scheduling and policy domain in one site, owned by one tenant:

| Field               | Values                                                                                                                                                                                                        |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `class`             | `metal-host`, `metal-dedicated`, `vps-host`, `provider`                                                                                                                                                       |
| `gpuModel`          | Exactly one GPU model family per pool, for example `H100`                                                                                                                                                     |
| `sharing`           | `exclusive` (default), `mig`, `timeslice`, `mps`                                                                                                                                                              |
| `agent`             | `bcd` (machines run the daemon) or `none` (reached only through a provider's API)                                                                                                                             |
| `policy`            | `acceptance` checks, `overcommit` ratios for scalars, `lostAfter`, `disruptionBudget` (default 0.05), `admission.budgetCheck`, `maxPending`, `hardwareProfile`, `sharing.replicas`, `migLayout`, `tenantRoot` |
| `providerAccountId` | For `provider` pools, the account that backs them                                                                                                                                                             |

An offering is a `pool_offers` row that makes a pool exist in another tenant's view,
with a quota per resource (`gpu`, `vcpu`, …), an optional `priority` and `includeDescendants`. A resource missing
from the quota grants zero. See [pools](/docs/guides/capacity/pools).

## Nodes [#nodes]

A node is a server running `bcd`, or a provider machine that runs it. Its state is not one
field but several independent axes, so each question has one answer:

| Axis             | Values                                                                                                                                                      | Changed by                               |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| `provisionState` | `adopting`, `adopted`, `enrolling`, `inspecting`, `commissioning`, `ready`, `deploying`, `deployed`, `releasing`, `rescue`, `failed`, `retiring`, `retired` | Workflows and the lifecycle controller   |
| `maintenance`    | true or false                                                                                                                                               | Failures, remediation, operators         |
| `cordoned`       | true or false                                                                                                                                               | `nodes.cordon`, `nodes.uncordon`, drains |
| `power`          | `on`, `off`, `unknown`                                                                                                                                      | BMC drivers                              |
| `connectivity`   | `connected`, `disconnected`, `unreachable`, `lost`                                                                                                          | Sessions and the liveness controller     |
| `conditions`     | Typed conditions such as `GpuCountMismatch`, `NodeUnreachable`, `ClockSkew`                                                                                 | Health, inventory, liveness              |

Beside the axes a node has a `role` (`unassigned`, `host`, `dedicated`, `site`), a `trust` (`platform`,
`tenant-owned`, `tenant-controlled`), an `adapter` naming how the control plane reaches it (`bcd/v1`,
`provider-api/v1`), per-concern driver bindings, a generation and
`observedGeneration`, and a hardware key: the SHA-256 of its normalized product UUID,
serial number, permanent MACs and GPU UUIDs, trusted only when at least two independent identifiers survive the
placeholder blocklist.

A node is schedulable only when all of these hold: `provisionState` is `adopted` or `deployed`, `role` is `host`,
it is neither in maintenance nor cordoned, `connectivity` is `connected`, its `noScheduleReasons` list is empty,
`observedGeneration` is at least 1, and its trust is `platform` (or `tenant-owned` in a pool its owner owns). See
[operating nodes](/docs/guides/nodes/operating-nodes).

## Devices and atoms [#devices-and-atoms]

A device is a unit with typed attributes, in the style of Kubernetes dynamic resource
allocation. Its `class` is `gpu`, `gpu.mig` (ids `<gpuUuid>/gi<g>/ci<c>`), `gpu.slot` (time-slice or MPS slots,
`<gpuUuid>/s<k>`), `nic.vf`, `disk` or `node`. Only atoms, devices with `bookable: true`, are
ever booked; a partitioned GPU's parent is not bookable. Device ids are stable: GPU UUID, disk WWN, MAC.

GPUs carry two names. `gpuProduct` is the normalized NVML name (`NVIDIA H100 80GB HBM3` becomes `H100-SXM5-80GB`);
`gpuModel` is the family (`H100`), mapped through a versioned `gpu-models.json`. An unknown name maps to `other` and
raises `GpuModelUnknown`. Pools, policies and meters use `gpuModel`; selectors may use either.

Inventory is the node's `NodeInventory v1` document, sectioned and hashed, with
append-only snapshots. Capacity is what the inventory makes bookable: allocatable,
allocated, reserved and available, per pool and resource. See [inventory and health](/docs/guides/nodes/inventory-and-health).

## Requests, reservations and bookings [#requests-reservations-and-bookings]

A reservation is a tenant's claim on a pool: `devices` (class, count, selector,
topology), an optional `start`, a `duration`, and a `binding`:

* `hold` (the default): CDI devices on the node, consumed by a workload manager there.
* `instance`: a VPS instance's allocation.
* `node`: a whole node deployed with the tenant's profile, in `tenantRoot` pools (v0.5).

A reservation starts as a draft whose id derives from the tenant, the caller's identity and
the idempotency key (`rsv_` plus the base32 SHA-256 of the three), so a retry finds the same row and Better IAM can
decide on the exact request before it exists. Its states are `draft`, `denied`, `awaiting-confirmation`, `pending`,
`placed`, `binding`, `active`, `releasing`, `released`, `failed`, `cancelled` and `expired`.

An allocation is a placement on one node (a gang has one per node), owned by a
reservation or an instance, listing its atoms and scalar amounts. A booking is one row
`(device_id, allocation_id, during)` in `device_bookings`, where `EXCLUDE USING gist (device_id WITH =, during WITH &&)`
allows one live booking per atom per instant. See [reservations](/docs/guides/capacity/reservations).

## Leases [#leases]

"Lease" is always qualified:

* An allocation lease is a renewable hold: a reservation made with `leaseSeconds`
  is released unless `reservations.renew` is called within every period.
* A liveness lease is a session's 40-second time to live.
* A shard, pool or site lease gives one worker or agent ownership with a fencing epoch:
  `shard:<n>`, `pool:<id>`, `site:<id>:boot`, `singleton:<job>`, `gateway:<id>`.

## Instances, images and profiles [#instances-images-and-profiles]

An instance is a VM, microVM or single-tenant container on an allocation, with a
`desiredState` (`running`, `stopped`, `deleted`) and an observed `state`. An image is a
signed, content-addressed artifact (`name@version`, `raw`, `qcow2` or `iso`, with a SHA-256), `private` or `public`,
and `verified` once its signature checks out against a signer allowlist. A profile is a
versioned provisioning template for `metal` or `vps` targets, with driver bindings, an image, user data, steps, host
configuration and acceptance; it is immutable once a workflow used it.

## Workflows, drivers and providers [#workflows-drivers-and-providers]

A workflow is a profile expanded for one target into persisted
steps. Each step runs on `control-plane`, `site`, `node` or `ramdisk`, and may
`await: 'rejoin'` for steps that end the agent's boot. A driver implements one or more
concerns (`power`, `management`, `boot`, `deploy`, `inspect`, `console`, `virtual-media`, `bios`, `raid`, `firmware`,
`config`, `runtime`). A provider is an external capacity API, configured as
provider accounts. See [workflows](/docs/guides/bare-metal/workflows) and
[drivers](/docs/extending/drivers).

## Commands and desired state [#commands-and-desired-state]

A command is a durable, idempotent one-shot message to one node, with a per-node `seq`,
`not_before`, `not_after`, an optional `min_generation`, and the flags of its kind.
A NodeSpec is the node's generation-numbered desired state. The difference between the two
is the subject of [desired state and commands](/docs/guides/concepts/desired-state-and-commands).

## Tenants, projects and admission [#tenants-projects-and-admission]

A tenant is a Better IAM `tenantId`: the root tenant is the platform, organizations are
customers. A project is an application-owned grouping inside a tenant (the `projects`
table). The admission context is the IAM decision context recorded with each
allowed mutation (identity, session kind, agent, delegation, request id) in `audit_events`, because Better IAM does
not audit allows. Asynchronous work acts on that record, never on a stored token.

## IAM resource types [#iam-resource-types]

Better Compute contributes these types to Better IAM's catalog, with the configurable prefix `compute-`. All of them
are application-owned: Better Compute resolves them from its own projection and never registers them in Better IAM.

| Type                       | What it is                            | Parent         | Relations                             | Attributes policies can read                                                      |
| -------------------------- | ------------------------------------- | -------------- | ------------------------------------- | --------------------------------------------------------------------------------- |
| `compute-fleet`            | The platform fleet                    | none           | none                                  | none                                                                              |
| `compute-tenant`           | A tenant's compute scope              | none           | none                                  | none                                                                              |
| `compute-project`          | A project                             | none           | `member`, `admin`                     | none                                                                              |
| `compute-site`             | A site                                | none           | none                                  | `region`, `iamRegion`, `kind`                                                     |
| `compute-pool`             | A pool                                | none           | `viewer`, `user`, `operator`, `admin` | `region`, `class`, `gpuModel`, `gpuMemoryGb`, `sharing`                           |
| `compute-node`             | A machine                             | `compute-pool` | `operator`                            | `region`, `gpuModel`, `gpuCount`, `lifecycle`, `trust`, `dedicatedTo`             |
| `compute-reservation`      | A GPU reservation, decided as a draft | `compute-pool` | none                                  | `region`, `gpuModel`, `gpuCount`, `durationHours`, `binding`, `projectId`, `spot` |
| `compute-instance`         | A VPS instance                        | `compute-pool` | none                                  | `vcpu`, `memoryGiB`, `gpus`, `imageId`, `imageVerified`, `projectId`              |
| `compute-image`            | An OS or ramdisk image                | none           | none                                  | `visibility`, `verified`                                                          |
| `compute-profile`          | A provisioning profile                | none           | none                                  | none                                                                              |
| `compute-provider-account` | An account with a capacity provider   | none           | none                                  | none                                                                              |

The first three are scope types. `compute-fleet` has the single id `global` and resolves only in the root tenant;
`compute-tenant/{t}` resolves only in tenant `t`; `compute-project/{id}` resolves in its owner tenant. Methods that
act on a whole scope, such as `pools.list` or `quotas.set`, are decided on one of them.

Because reservations and instances have `compute-pool` as their parent, a policy can combine pool access and
request limits in one decision: "users of this pool may reserve up to 8 H100s for at most 72 hours." See
[actions and resources](/docs/guides/iam/actions-and-resources) for every action.


# Concepts (/docs/guides/concepts)

> The ideas Better Compute is built on, from one Postgres truth and desired state to one IAM decision per mutation, and its fifteen design principles.



Better Compute answers three questions about a fleet of machines at any moment: what exists, what is free, and who
holds it. Everything else, from placement to provisioning to billing, is built on those answers. This section
explains the model behind them before the guides show how to use it.

## The shape of the system [#the-shape-of-the-system]

A node is a server running bcd, the daemon. It dials out to a
gateway and keeps one long-lived session open. The control plane never connects to it; it
writes what the node should look like, its NodeSpec, and queues
commands for one-shot actions. The gateway delivers both.

Nodes belong to a pool, a scheduling and policy domain in one site
with exactly one GPU model. A tenant asks a pool for devices with a reservation. Better
IAM decides whether the caller may ask, and the placer decides where the devices come from
by inserting bookings: rows that a database constraint makes exclusive per device and per
instant.

All of this state lives in PostgreSQL. Gateways, caches and in-memory projections can be rebuilt from it at any
time, and reconcilers keep the world converging toward it.

  - [Architecture](/docs/guides/concepts/architecture): Planes, components, process modes, and the four flows: join, reserve, bare metal and VPS.

  - [Domain model](/docs/guides/concepts/domain-model): Every entity, how they relate, and how each maps to a Better IAM resource type.

  - [Desired state and commands](/docs/guides/concepts/desired-state-and-commands): NodeSpec generations, level-triggered reconcilers, and when to use a command instead.

  - [Consistency](/docs/guides/concepts/consistency): One clock, exclusion constraints, the outbox and work queue, fencing, and why disconnected is not dead.

## Design principles [#design-principles]

The design follows fifteen principles. They explain most of the behavior you will see, including the behavior that
looks conservative at first, such as never releasing a reservation because a node went quiet.

| #   | Principle                                                  | What it means in practice                                                                                                                                                                                                                       |
| --- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| P1  | PostgreSQL is the truth                                    | Gateway memory, caches, projections and wake-up pokes can all be lost and rebuilt. Nothing is correct only because a process remembered it.                                                                                                     |
| P2  | Desired state plus generations, reconciled level-triggered | Standing facts (device assignments, instances, host configuration) are desired state with a generation number. Controllers compare desired and observed state and act on the difference, not on events. Commands are only for one-shot actions. |
| P3  | At-least-once delivery, made safe                          | Every message may arrive twice. Idempotency keys and fencing tokens (session epoch, lease epoch, generation, journal id) make the second arrival harmless.                                                                                      |
| P4  | Invariants live in the database                            | `EXCLUDE` constraints, `CHECK` constraints, guarded updates and partial unique indexes enforce the rules, whatever the application code does.                                                                                                   |
| P5  | Nodes only dial out                                        | A daemon opens no listening port. It connects to a gateway on 443, so nodes can sit behind NAT and firewalls.                                                                                                                                   |
| P6  | Disconnected is not dead                                   | Nothing is released because a node is unreachable, and bare metal is never declared lost automatically. A partition changes a condition, not ownership.                                                                                         |
| P7  | Fleet paths never call IAM                                 | Heartbeats, inventory, commands and reconcilers never call Better IAM. An admission makes exactly one IAM decision.                                                                                                                             |
| P8  | Telemetry stays out of the control path                    | Metrics go to a time-series database, never through PostgreSQL or the daemon's session.                                                                                                                                                         |
| P9  | One clock                                                  | SQL reads time only through `compute_now()`, and TypeScript business time comes from a `Clock` anchored to it. Gateways report ages, not timestamps.                                                                                            |
| P10 | Core never imports an implementation                       | Stores, the IAM port, modules and drivers are passed in. The two always-on modules, `inventory` and `capacity`, are the one exception.                                                                                                          |
| P11 | Fail closed                                                | An unknown command kind is `UNSUPPORTED`; a resolver miss is `NOT_FOUND` and a resolver timeout is a deny; a failed batch authorization fails the request; a method whose action is missing from the catalog fails `initialize()`.              |
| P12 | Everything is bounded                                      | Lanes, queues, payloads, the daemon's journal, spill buffers and metric cardinality all have limits.                                                                                                                                            |
| P13 | The daemon is dumb but capable                             | The daemon executes typed commands and converges typed desired state. Policy stays in the TypeScript control plane.                                                                                                                             |
| P14 | Relays cannot forge                                        | Gateways and site agents relay signed intent downward and node-signed results upward. Tokens and secrets cross them only sealed.                                                                                                                |
| P15 | Mirror Better IAM                                          | Conventions follow Better IAM (the factory, the operation envelope, the CLI framework, the packaging) unless there is a stated reason not to.                                                                                                   |

Where a principle turns into a mechanism, the concept pages explain it: P1, P3, P4, P6 and P9 in
[consistency](/docs/guides/concepts/consistency), P2 and P13 in
[desired state and commands](/docs/guides/concepts/desired-state-and-commands), P5, P7 and P14 in
[architecture](/docs/guides/concepts/architecture).

## Where Better IAM stops and Better Compute starts [#where-better-iam-stops-and-better-compute-starts]

Better Compute deliberately owns very little identity. It has no users, roles, API keys, policies or invoices:
[Better IAM](/docs/guides/iam) owns those. What Better Compute owns is the fleet and its machine identities:

| Better IAM owns                                     | Better Compute owns                                                      |
| --------------------------------------------------- | ------------------------------------------------------------------------ |
| People, agents, service accounts and their sessions | Nodes and their SPIFFE certificates (the machine PKI)                    |
| Tenants and the tenant tree                         | Sites, pools, offerings, quotas and projects inside those tenants        |
| Policies, roles and every allow or deny             | The method-to-action table, and one request to Better IAM per mutation   |
| The billing ledger and meters                       | Usage windows, computed from reserved time and sent to the ledger        |
| The audit chain for IAM events                      | `audit_events` rows with the admission context of every allowed mutation |

The line matters for security: a daemon never holds an IAM credential, and a person never holds a machine
certificate.

## Vocabulary [#vocabulary]

The guides use a small set of terms precisely. The ones that trip people up most:

* An atom is the only thing that is ever booked: a whole GPU, a MIG instance, or a time-slice
  slot. A partitioned GPU's parent is not bookable.
* A draft is a reservation or instance row written before the IAM decision, so Better IAM
  can decide on the exact request.
* A lease is always qualified: an allocation lease is a renewable hold on
  devices, a liveness lease says a session is alive, and a
  shard lease gives one worker ownership of a slice of reconciler work.
* Cordon stops new placements on a node; drain also waits for
  its current allocations to end; maintenance is a separate flag, set when
  provisioning fails or remediation gives up, that also keeps a node out of placement.

The [glossary](/docs/reference/glossary) defines every term.


# Audit (/docs/guides/events/audit)

> Every allowed mutation lands in audit_events with its admission context, sealed hourly by signed per-tenant RFC 6962 Merkle checkpoints you can verify and prove.



Better Compute keeps its own audit log, next to Better IAM's. Every mutation the API allows writes a row to
`audit_events` in the same transaction as the change, with the full admission
context: who acted, in which session, for which agent and delegation, which action on which resource, and
when it was decided. The audit module then seals those rows, hour by hour, into signed per-tenant Merkle
checkpoints. Anyone holding the audit key's pin can check that a tenant's trail was not edited, and a single event
can be proven to a third party with a small JSON file and no database access.

The two logs split the work:

| Log                      | Records                                                                        | Where                                     |
| ------------------------ | ------------------------------------------------------------------------------ | ----------------------------------------- |
| Better IAM's audit chain | denials and root overrides, as IAM decides them                                | Better IAM's database, per tenant         |
| `audit_events`           | allowed compute mutations with their admission context, joins, secret releases | compute's database, sealed by checkpoints |

## What is recorded [#what-is-recorded]

`audit_events` has one row per record:

| Column                         | Holds                                                                                    |
| ------------------------------ | ---------------------------------------------------------------------------------------- |
| `id`                           | `aud_…`                                                                                  |
| `tenant_id`                    | the tenant the record belongs to: the decision's tenant, or a node's owner               |
| `occurred_at`                  | `compute_now()` when written, which is the writing transaction's start time              |
| `action`                       | `compute:reservations:create`, or a system record such as `node.joined`                  |
| `resource_type`, `resource_id` | what it acted on, like `compute-reservation` and `rsv_…`                                 |
| `admission`                    | the admission context (JSON) for API mutations                                           |
| `details`                      | extra data (JSON) for system records                                                     |
| `txid`, `seq`                  | the writing transaction (`pg_current_xact_id()`) and a sequence: the commit-order cursor |

Rows are written by:

* **The API envelope**, for every allowed mutation, inside the handler's transaction. `admission` holds `tenantId`,
  `identityId`, `identityKind`, `sessionId`, `sessionKind`, `requestId`, `action`, `resource`, `decidedAt`, and
  `agentId` and `delegationId` when an agent acted in a delegated session. Reads are not recorded.
* **Composite calls** such as `fleet.apply`, one row per sub-call, under the sub-call's savepoint, so a failed
  sub-call leaves no row.
* **The join service**: `node.joined` or `node.reclaimed` on the node, with the attestor, join token, certificate
  serial, revoked serials, hardware key and `nodeKey` in `details`. `nodeKey` is `tpm` when the attestor proved the
  node key lives in the machine's TPM (a `tpm` join, cf24876; see
  [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)), else `unattested`.
* **The secret broker**: `secret.released` when a node receives a secret, with the secret reference as the
  resource and the node, request, certificate serial and version in `details`, never the value.

There are no foreign keys, so records outlive what they describe. The table is partitioned by UTC month on
`occurred_at` (`core/0018_partitioned_history`), and the retention pass never prunes it.

> **No API reads it yet.** 
  No API method returns audit rows today. Operators read `audit_events` in SQL and use the `better-compute audit`
  commands below. PLAN.md also plans command, step and ticket records in this table; the writers listed above are
  the ones in the code.

## Checkpoints [#checkpoints]

Better IAM's audit chain links each row to the one before it, so every writer serializes on the chain's head. Compute
does not: writers append freely, and a periodic pass seals what has committed.

<Mermaid
  chart="flowchart LR
  subgraph pass[&#x22;One checkpoint pass (hourly)&#x22;]
    R[&#x22;rows after the cursor, below xmin, in (txid, seq) order&#x22;]
    G[&#x22;group by tenant&#x22;]
    M[&#x22;RFC 6962 root per tenant&#x22;]
    S[&#x22;sign the body with the audit key&#x22;]
  end
  C1[&#x22;acme #41&#x22;] -->|&#x22;previous = sha256(body #41)&#x22;| C2[&#x22;acme #42&#x22;]
  R --> G --> M --> S --> C2"
/>

**Commit order, not clock order.** `occurred_at` is the start of the writing transaction, so a long transaction
could commit a row into an hour that was already sealed. Checkpoints therefore follow the `(txid, seq)` cursor,
reading only transactions older than the snapshot's xmin, the same rule the [outbox](/docs/guides/events#commit-order-and-the-xmin-cursor)
uses. No row can commit behind the cursor, however long its transaction ran. `occurred_at` is still hashed, as data,
in microseconds.

**One pass.** The pass locks the cursor row `audit` in `audit_checkpoint_cursor`, reads up to 50,000 rows after
it, groups them by tenant, and writes one checkpoint per tenant into `audit_checkpoints`, all in one transaction.
Then it moves the cursor to the last row read. A full pass runs again at once.

**One checkpoint** covers one tenant's rows within the pass's global range `(from, to]`. Its body is canonical
JSON (sorted keys, no whitespace), and those exact bytes are signed:

```json title="A checkpoint body"
{
  "from": "88100.4410",
  "previous": "3q2-7w…",
  "root": "Jc1tQ…",
  "sequence": 42,
  "size": 17,
  "tenantId": "acme",
  "to": "88213.4519",
  "v": 1
}
```

| Field        | Meaning                                                                             |
| ------------ | ----------------------------------------------------------------------------------- |
| `sequence`   | 1, 2, 3, … per tenant; a gap means a removed checkpoint                             |
| `from`, `to` | the global commit-cursor range, as `txid.seq`                                       |
| `size`       | the tenant's rows in the range                                                      |
| `root`       | the base64url RFC 6962 Merkle root of those rows, in commit order                   |
| `previous`   | the base64url SHA-256 of the previous checkpoint's body bytes; `null` for the first |

Each leaf is the RFC 6962 leaf hash, SHA-256 of `0x00` and the row's canonical JSON: `id`, `tenantId`,
`occurredAtUs` (microseconds since the epoch, as a string), `action`, `resourceType`, `resourceId`, `admission` and
`details`. The signature, `{ kid, sig, chain }`, is made for the purpose `audit-checkpoint` by the audit signing key,
a key that does nothing else. The row also stores `first_event_id`, `last_event_id` and the signed body text.

## Turn it on [#turn-it-on]

The audit key lives in the PKI bundle: `better-compute pki init` creates `auditSigning` beside the other keys, and
`pki show` prints its `auditSigningKeyPin` (`sha256/…`), which verifiers need. Add the module with the key:

```ts
import { auditModule } from 'better-compute/audit';
import { loadPkiBundle } from 'better-compute/cli';

const bundle = await loadPkiBundle('.better-compute/pki.json', process.env);

const compute = betterCompute({
  // …
  modules: [auditModule({ signer: bundle.auditSigning! })],
});
```

| Option   | Default  |                                                  |
| -------- | -------- | ------------------------------------------------ |
| `signer` | required | the bundle's `auditSigning` key                  |
| `every`  | `1h`     | how often the `audit.checkpoint` reconciler runs |
| `limit`  | 50,000   | rows per pass, all tenants together              |

The module adds migration `audit/0001_checkpoints` (`audit_checkpoints` and `audit_checkpoint_cursor`) and the
periodic `audit.checkpoint` reconciler, which a worker process runs. Its self-check warns `audit.checkpoints` in
`better-compute doctor` when rows older than three hours are still unsealed, which means no worker runs the job. The
example deployments add the module whenever the PKI bundle has an audit key.

To seal right away, for example before an export, run the same pass from the CLI. It repeats until nothing is left:

```bash
better-compute audit checkpoint     # needs the bundle's private key, unsealed with BETTER_COMPUTE_SECRET
# → { "passes": 1, "checkpoints": 3, "events": 212 }
```

## Verify a tenant's trail [#verify-a-tenants-trail]

```bash
better-compute audit verify --tenant acme
better-compute audit verify --tenant acme --pin sha256/… --query problems
```

`verify` recomputes the tenant's whole chain from the rows as they are now, in one read-only, repeatable-read
transaction, and exits `AUDIT_TAMPERED` after printing the report when anything is wrong. It checks signatures under
the pin from `--pin`, `BETTER_COMPUTE_AUDIT_PIN` or the PKI bundle's public half, so it needs no secret.

```json title="Report"
{
  "ok": false,
  "tenantId": "acme",
  "checkpoints": 42,
  "events": 3180,
  "unsealed": 9,
  "problems": [{ "code": "root", "sequence": 17, "message": "The rows in range do not hash to the sealed root" }]
}
```

| Problem     | Found when                                                                                    |
| ----------- | --------------------------------------------------------------------------------------------- |
| `gap`       | a checkpoint sequence is missing between two others                                           |
| `body`      | the stored body is not canonical, or does not match its row's tenant or range                 |
| `chain`     | `previous` is not the hash of the previous checkpoint's body                                  |
| `signature` | the signature does not verify under the audit key, or signs other bytes                       |
| `size`      | the number of rows in the range differs from the sealed `size`: a row was inserted or deleted |
| `root`      | the rows in the range do not hash to the sealed root: a row was edited, inserted or deleted   |
| `uncovered` | rows older than a threshold are not sealed yet (only when `uncoveredOlderThanMs` is passed)   |

`unsealed` counts the tenant's committed rows after its last checkpoint. The same check is available in code as
`verifyAuditCheckpoints(sql, { tenantId, trust: { pin }, uncoveredOlderThanMs? })` from `better-compute/audit`.

> **What the database alone cannot show.** 
  Rows written since the last checkpoint (up to an hour with the default cadence) are not sealed yet. Deleting the
  newest checkpoints together with their rows leaves no gap for `verify` to find, and whoever holds both database
  access and the audit key can rewrite everything. Keep the audit key off the database hosts where you can, and copy
  checkpoints, or at least each tenant's latest body and signature, somewhere the database's operators cannot
  change. A copy of an older checkpoint lets anyone confirm, later, that the chain still contains it.

## Prove one event [#prove-one-event]

An inclusion proof shows a third party, an auditor or the tenant, that one event is sealed in a signed checkpoint,
without giving them your database:

```bash
better-compute audit prove aud_01k… --tenant acme --out proof.json
better-compute audit check-proof proof.json --pin sha256/…
# → { "valid": true, "eventId": "aud_01k…", "tenantId": "acme", "sequence": 17 }
```

`prove` finds the checkpoint whose range covers the event and writes the event's leaf, its `index` in that
checkpoint, the audit path (base64url sibling hashes, bottom-up, per RFC 6962 section 2.1.1) and the signed
checkpoint body with its signature. It answers `NOT_FOUND` for an unknown event, or one that is not checkpointed yet.

`check-proof` needs only the file and the pin. It verifies the signature over the exact body bytes, that the parsed
body matches, that the event belongs to the checkpoint's tenant, and the audit path from the leaf to the signed root
(RFC 9162 section 2.1.3.2). It exits `INVALID_PROOF` otherwise. In code:

```ts
import { proveAuditEvent, verifyAuditProof } from 'better-compute/audit';

const proof = await proveAuditEvent(sql, { tenantId: 'acme', eventId: 'aud_01k…' });
const valid = verifyAuditProof(proof, { pin: 'sha256/…' }); // no database access
```

## Consistency proofs [#consistency-proofs]

`@better-compute/pki` also implements RFC 6962 consistency proofs: `merkleConsistencyProof(leaves, oldSize)` and
`verifyMerkleConsistency(oldSize, newSize, oldRoot, newRoot, proof)` show that a tree of `oldSize` leaves is a prefix
of a larger one, the append-only property of a transparency log.

The audit module does not use them today, because its trees do not grow: each checkpoint's tree covers only the rows
of its own range, and append-only history comes from the hash chain instead, with each body naming the hash of the
one before. There is no CLI command for consistency proofs. The primitives are there for tooling that builds a
single growing tree over a tenant's leaves.

## Copy it out with an audit sink [#copy-it-out-with-an-audit-sink]

An audit sink receives every committed `audit_events` row, in commit order, at least once: for a SIEM, or the copy
outside the database that the callout above asks for. Declare it with `defineSink({ kind: 'audit', id, write })` and
list it in `betterCompute({ sinks: { audit: [...] } })` or in a module's `extensions`. The built-in `audit-sinks`
module then runs a singleton reconciler every 5 seconds that hands each sink the rows after its own cursor
(`audit_sink_cursors`), 500 at a time, with a per-sink back-off on failure. Records carry `id` for deduplication.
The sink does not depend on the audit module: it copies rows whether or not they are checkpointed. See
[sinks](/docs/extending/sinks#audit-sinks).

## Archiving [#archiving]

`audit_events` is partitioned by month, and nothing deletes audit rows on a timer. Checkpoints are verified against
the rows the table holds, so archiving is a deliberate operator step in two parts:

1. `markAuditArchived(sql, { month: 'YYYY-MM', partition?, by? })` from `better-compute/audit` records the month in
   `audit_archives` (migration `audit/0002_archives`). It refuses the current or a future month with
   `INVALID_STATE`, and a month that still has rows no checkpoint sealed (run a checkpoint pass first). It is
   idempotent.
2. `detachHistoryPartition(tx, 'audit_events', 'audit_events_pYYYYMM')` from `@better-compute/postgres` takes the
   month out of every read at once, leaving it as a standalone table to export and drop.

`verify` then lists a checkpoint wholly inside archived months whose rows are gone in the report's `archived`
(`{ sequence, months, partial }`), with `archivedMonths` beside it: its body, chain and signature are still checked,
its root and size are not. While the rows are still attached it is verified as usual, so an edit still shows. A
checkpoint that straddles an archived month counts as archived (`partial: true`) only once its rows stop matching. A
month detached without being marked still shows `size` and `root` problems. Archive only months whose checkpoints
you keep, and keep the pin to verify them later. See [retention](/docs/operations/retention) and
[PKI](/docs/operations/pki).


# Events (/docs/guides/events)

> Every change writes a CloudEvents-style event to the outbox in its own transaction. Read it with events.list, stream it over SSE, or receive it by webhook.



A dashboard that shows reservations, a billing job that counts GPU hours, and an alerting service that pages someone
when a node goes missing all need to know what changed and in what order. Polling every list method is slow and
misses short-lived states. A callback fired from the request can be lost when the process dies right after the
commit, or sent for a change that was rolled back.

Better Compute writes an event for every change into the outbox, a table in the same
database, inside the same transaction as the change itself. The event exists if and only if the change committed.
Everything else (the history API, the live stream, webhooks, in-process subscribers and event sinks) reads that one
table.

## The outbox [#the-outbox]

Module code publishes through the context it is given, in the transaction it already holds:

```ts title="Inside a method handler or reconciler"
await context.events.publish(tx, {
  type: 'node.cordoned',
  tenantId: node.tenantId,
  subject: `node/${node.id}`,
  data: { reason: 'disk replacement' },
});
```

`publishMany(tx, events)` writes a batch in one statement, with `seq` in input order. The runtime checks every
event before anything is written:

| Field      | Rule                                                                                                  |
| ---------- | ----------------------------------------------------------------------------------------------------- |
| `type`     | `area.name` with at least one dot, at most 128 characters (`node.connected`, `node.provision.active`) |
| `tenantId` | Non-empty, at most 128 characters: the tenant that may read the event                                 |
| `subject`  | Non-empty, at most 512 characters: what the event is about, as `kind/id`                              |
| `data`     | Any JSON, or omitted (`null`). The whole payload is at most 64 KiB                                    |

A violation is `INVALID_INPUT`, and the transaction fails with it. The runtime adds an `id` (`evt_...`) and stamps
`time` from `compute_now()`, the database clock.

The table is small on purpose: `seq bigserial`, `txid xid8` (the writing transaction, from `pg_current_xact_id()`),
`topic` (the type), `payload jsonb` and `created_at`. It is created by the base schema (`core/0002_base`, with
`created_at` and the per-tenant index `outbox_tenant_cursor` added by `core/0009_runtime`).

### Commit order and the xmin cursor [#commit-order-and-the-xmin-cursor]

`seq` is drawn when a row is inserted, not when its transaction commits, so a reader that walks `seq` can hand out
42, move its cursor past it, and never see 41 when a slower transaction commits it a moment later.

<Mermaid
  chart="sequenceDiagram
  participant T1 as Transaction 1
  participant T2 as Transaction 2
  participant O as outbox
  participant R as Reader
  T1->>O: insert (seq 41)
  T2->>O: insert (seq 42)
  T2->>O: commit
  R->>O: read after the cursor
  Note over R,O: T1 is still open, so xmin is T1: neither row is returned
  T1->>O: commit
  R->>O: read after the cursor
  Note over R,O: both rows, in (txid, seq) order"
/>

Every outbox reader therefore takes only rows of transactions older than the snapshot's xmin
(`txid < pg_snapshot_xmin(pg_current_snapshot())`) and orders them by `(txid, seq)`. Nothing can commit behind a
cursor once it was handed out. The cost is latency: an event becomes visible when every transaction older than it
has finished, so one long transaction holds the stream back until it ends. See
[consistency](/docs/guides/concepts/consistency).

## Event shape [#event-shape]

Every reader returns the same CloudEvents-style object:

```json title="reservation.active"
{
  "id": "evt_01k5y3v0m6y8x7w5t4s3r2p1n0",
  "type": "reservation.active",
  "tenantId": "ten_01k5y2a9c3d4e5f6g7h8j9k0m1",
  "subject": "reservation/rsv_01k5y3a0b1c2d3e4f5g6h7j8k9",
  "time": "2026-09-24T10:15:02.114Z",
  "sequence": "88213.51120",
  "data": { "state": "active", "poolId": "pool_01k5y1z2x3w4v5t6s7r8q9p0n1" }
}
```

<TypeTable
  type="{
  id: { type: 'string', description: 'Unique event id (evt_...). Deduplicate on it: every consumer is at least once.', required: true },
  type: { type: 'string', description: 'The event type, such as reservation.active or node.disconnected.', required: true },
  tenantId: { type: 'string', description: 'The tenant the event belongs to. Every reader filters on it.', required: true },
  subject: { type: 'string', description: 'What it is about: reservation/rsv_..., node/node_..., instance/inst_...', required: true },
  time: { type: 'string', description: 'RFC 3339 in UTC, from compute_now() when the row was written.', required: true },
  sequence: { type: 'string', description: 'The outbox position (txid and seq) of this event. Use it to order and to resume.', required: true },
  data: { type: 'JSON | null', description: 'Type-specific details. Reservation events carry state and poolId, for example.' },
}"
/>

> **Two spellings of the same cursor.** 
  `events.list` and `nodes.events` write `sequence` and take `cursor` as `txid:seq` (a colon). The SSE stream and
  webhook bodies use `txid.seq` (a dot), and `Last-Event-ID` accepts only that form. Both name the same outbox
  position; replace the separator to move between them.

## Event types [#event-types]

The events module keeps a catalog of the families the control plane and its modules publish. Webhook subscriptions
are checked against it, and `webhooks.eventTypes` returns it. A test runs the simulator and fails when any module
writes a type outside the catalog.

| Family                                                                                   | Subject            | Module    | Types you will see                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------------------------------------------------------------------------------------- | ------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `site.*`                                                                                 | `site/{id}`        | server    | `site.created`, `site.updated`, `site.deleted`, `site.boot`                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `project.*`                                                                              | `project/{id}`     | server    | `project.created`, `project.archived`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `pool.*`                                                                                 | `pool/{id}`        | capacity  | `pool.created`, `pool.updated`, `pool.offered`, `pool.unoffered`, `pool.archived`                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `quota.updated`                                                                          | `pool/{id}`        | capacity  | `quota.updated`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `reservation.*`                                                                          | `reservation/{id}` | capacity  | `reservation.placed`, `pending`, `queued`, `binding`, `active`, `extended`, `preempting`, `preempted`, `repaired`, `failed`, `cancelled`, `expired`, `awaiting-confirmation`, `binding-rejected`, and one per state change                                                                                                                                                                                                                                                                                                           |
| `reservation.reassigned`                                                                 | `reservation/{id}` | capacity  | An offboarded identity's reservation moved to its successor: `{ reservationId, from, to, reason }`                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `node.*`                                                                                 | `node/{id}`        | server    | `node.joined`, `node.acceptance.started`, `node.acceptance.failed`, `node.adopted`, `node.connected`, `node.disconnected`, `node.unreachable`, `node.lost`, `node.status` (only when a report says something new; see below), `node.cordoned`, `node.uncordoned`, `node.draining`, `node.drained`, `node.labeled`, `node.tainted`, `node.adopting`, `node.deleted`, `node.command.queued`, `node.command.cancel-requested`, `node.report-failed`, `node.ssh-cert.issued`, `node.ssh-cert.revoked`                                    |
| `node.inventory.changed`, `node.gpu.*`, `node.mig.*`, `node.disk.*`, `node.driver.*`     | `node/{id}`        | inventory | `node.gpu.added`, `node.gpu.missing`, `node.mig.layout.changed`, `node.disk.replaced`, `node.driver.changed`; `node.mig.scheduled`, `node.mig.granted` and `node.mig.applied` from capacity's MIG layout gate; also `node.condition.changed` from the hygiene sweep and host pressure (within `node.*`)                                                                                                                                                                                                                              |
| `node.enrolled`, `node.pxe-mac.discovered`, `node.pxe-mac.confirmed`, `node.provision.*` | `node/{id}`        | metal     | `node.provision.<state>` for each provisioning transition; `node.pxe-mac.discovered` when PXE discovery found a candidate MAC for a machine enrolled without MACs (`bound` says whether a trusted pool bound it at once); `node.pxe-mac.confirmed` when an operator bound a MAC with `metal.confirmMac`                                                                                                                                                                                                                              |
| `command.settled`                                                                        | `command/{id}`     | gateway   | A command reached a terminal state                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `workflow.*`                                                                             | `workflow/{id}`    | provision | `workflow.created`, `started`, `paused`, `resumed`, `retried`, `cancelled`, and one per state                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `instance.*`                                                                             | `instance/{id}`    | vps       | `instance.created`, `scheduled`, `provisioning`, `starting`, `stopping`, `resized`, `deleting`, `deleted`, `failed`, `lost`, `recovered`, `recreating`, `orphaned`                                                                                                                                                                                                                                                                                                                                                                   |
| `sandbox.*`                                                                              | `sandbox/{id}`     | vps       | `sandbox.created`, `ready`, `extended`, `expired`, `deleted`, `exec.issued`; `sandbox.snapshot.ready` / `failed` when a snapshot settles; see [sandbox events](/docs/guides/vps/sandboxes#events)                                                                                                                                                                                                                                                                                                                                    |
| `fleet.*`                                                                                | `fleet/{id}`       | fleet     | `fleet.drifted`, `fleet.converged`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `rollout.*`                                                                              | `rollout/{id}`     | rollout   | `rollout.started`, `paused`, `resumed`, `aborted`, `completed`                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `tunnel.*`                                                                               | `tunnel/{id}`      | server    | `tunnel.issued`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `security.*`                                                                             | varies             | server    | `security.join-token.created`, `security.join-token.revoked`, `security.site-token.created`, `security.site-token.revoked`, `security.join.refused`, `security.renew.refused` (0e42018; see [refused renewals](/docs/operations/security#refused-renewals)), `security.certificate-revoked`, `security.node.identity-revoked`, `security.node.duplicate-session` and `security.node.quarantined` (from the gateway), `security.secret.put`, `security.secret.deleted`, `security.secret.refused`, `security.workload.refused` (v0.5) |
| `webhook.ping`                                                                           | `webhook/{id}`     | events    | A test delivery from `webhooks.ping`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

The catalog names the subject each family is declared with, but code is the final word: `tunnel.issued`, for
example, is published with the instance or node it opens as its subject. Modules you write publish their own
families; register them with `eventsModule({ extraEvents })` so webhooks can subscribe to them (see
[webhooks](/docs/guides/events/webhooks)).

`node.status` is published when a node's status report says something new on its session, not for every report.
The gateway compares reports without their gauges (host pressure, volume storage), which move on every converge
pass, so `node.status` does not carry every gauge update: read gauges from [metrics](/docs/operations/observability),
and conditions from `node.condition.changed`. A reconnected session publishes its first report again.

## Read history with events.list [#read-history-with-eventslist]

`events.list` returns a tenant's committed events after a cursor, oldest first. It is decided as
`compute:events:read` on `compute-tenant/{tenantId}` ([reference](/docs/reference/api/events#list)).

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant whose events to read.', required: true },
  type: { type: 'string', description: 'A type prefix, not a pattern: node. or reservation. or security.join (lower case, at most 64 characters).' },
  limit: { type: 'number', description: 'Events per page, at most 500.', default: '100' },
  cursor: { type: 'string', description: 'An event sequence (txid:seq). The page starts after it.' },
}"
/>

The result is `{ items, cursor? }`. `cursor` is present only when more rows follow; to keep tailing, remember the
last item's `sequence` and pass it as the next `cursor`.

```ts title="Read every reservation event of a tenant"
let cursor: string | undefined;
do {
  const page = await client.events.list({ tenantId, type: 'reservation.', limit: 500, ...(cursor ? { cursor } : {}) });
  for (const event of page.items) await handle(event);
  cursor = page.cursor;
} while (cursor);
```

```bash
better-compute api events.list tenantId=$TENANT type=node. limit:=20 --format table
```

`nodes.events` is the same read narrowed to one node's subject, decided as `compute:nodes:read` on the node, with
at most 200 events per page ([reference](/docs/reference/api/nodes#events)).

## Ways to consume events [#ways-to-consume-events]

| Consumer                                  | How it reads                      | Position kept by                    | Use it for                                   |
| ----------------------------------------- | --------------------------------- | ----------------------------------- | -------------------------------------------- |
| `events.list`                             | Pull, one page per call           | The caller                          | History, backfill, reports                   |
| [`GET /watch`](/docs/guides/events/watch) | Server-Sent Events                | The client (`Last-Event-ID`)        | Browsers, consoles, scripts that follow live |
| [Webhooks](/docs/guides/events/webhooks)  | Signed HTTPS POSTs                | The events module (`event_cursors`) | Other services of the tenant                 |
| `compute.events.subscribe`                | In process                        | Memory, per process                 | Reacting inside your own server              |
| [Event sinks](/docs/extending/sinks)      | `sinks.events` on `betterCompute` | `outbox_cursors` (`sink:<id>`)      | Warehouses, queues, platform-wide feeds      |

`compute.events.subscribe(pattern, handler)` calls `handler` for each committed event whose type matches (`*` matches
any run of characters, so `reservation.*` and `*` work) and returns the unsubscribe function. The first subscriber
starts the process's cursor at the head, so it sees events from that moment on, and the `event-subscribers` job of
every started process feeds it. A handler that throws is logged and does not stop the others. The position lives in
memory: a restarted process starts at the head again, so use a sink when you need every event.

Event sinks run in the `sinks` worker role, each under its own lease `singleton:outbox-dispatch:<sink id>` and in
its own loop, so different workers can deliver different sinks. Each sink has its own durable cursor, so a failing
sink backs off (1 s, doubling up to 5 minutes) without holding the others back, and a
new lease holder resumes where the last one saved. A failing batch is bisected down to one event, and an event the
sink refuses on every attempt while its neighbour goes through is dead-lettered to `outbox_dead_letters`; see
[poison events](/docs/extending/sinks#poison-events).

Every consumer is at least once: an event can arrive twice after a crash or a retry. Deduplicate on `id`, and order
by `sequence` (compare txid first, then seq, as numbers).

## Retention [#retention]

The outbox is not an archive. `retentionModule()` from `@better-compute/postgres` prunes it with the periodic
`core.retention` reconciler. Deployments built by the CLI (`serve`, `dev`, `migrate`) add it automatically unless your
configuration registers a `retention` module itself.

* A row is deleted only when every durable cursor of the `outbox` stream in `outbox_cursors` has passed it, and it is
  older than `minAgeSeconds` (default 3600).
* A pass runs every 5 minutes (`every`) and deletes in batches of 5,000 (`batch`), at most 20 statements per pass
  (`maxRounds`), so a backlog drains over several passes without long locks.
* A stream with no durable cursor is never pruned. Today only event sinks save outbox cursors, so a deployment
  without sinks keeps its whole outbox.
* The webhook fan-out cursor, SSE clients and `events.list` callers do not hold retention back. A client that
  resumes from a pruned position continues with the oldest row still there, without an error.

See [retention](/docs/operations/retention) for the other tables the pass cleans.

## In this section [#in-this-section]

  - [Webhooks](/docs/guides/events/webhooks): Signed deliveries to a tenant's HTTPS endpoints, with retries, dead letters and redelivery.

  - [Watch](/docs/guides/events/watch): GET /watch: live events over Server-Sent Events, resumable with Last-Event-ID.

  - [Audit](/docs/guides/events/audit): Admission contexts in audit\_events, sealed by signed per-tenant Merkle checkpoints.


# Watch (/docs/guides/events/watch)

> Stream a tenant's events live over Server-Sent Events with handleWatch, resume from any cursor with Last-Event-ID, filter by type, and authorize every stream.



`handleWatch` streams one tenant's events as [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html),
read straight from the outbox. A client opens one long `GET`, receives each committed event
as it becomes visible, and, when the connection drops, reconnects with the id of the last event it saw and continues
exactly after it: no gap, no duplicate from the server's side.

It is the right tool for anything that follows events while it runs: the console's live stores, a dashboard, a
script that waits for a reservation to become active. For a service that must see every event even while it is
down, use [webhooks](/docs/guides/events/webhooks), which keep the position on the server.

## Mount the endpoint [#mount-the-endpoint]

`handleWatch(request, options)` is a Fetch-style handler from `better-compute/events`. The compute server does not
mount it: the host that serves the API does, next to it, and supplies the authorization. That keeps the long-lived
stream out of the JSON API's envelope and lets each host choose how streams are authenticated.

  **serve (host export):**

    ```ts title="better-compute.config.ts"
    import { defineHost, watchAuthorizer } from 'better-compute/cli';
    import { createWatchRegistry, handleWatch } from 'better-compute/events';

    export const host = defineHost(({ compute }) => {
      const authorize = watchAuthorizer(compute); // decided exactly like events.list
      const streams = createWatchRegistry(); // so IAM revocations close open streams at once
      return {
        iam: { bridge, streams: [streams] }, // bridge: your createComputeIamBridge()
        http: {
          routes: [
            {
              path: '/watch',
              method: 'GET',
              handler: (request: Request) =>
                handleWatch(request, { sql: compute.context.sql, authorize, registry: streams }),
            },
          ],
        },
      };
    });
    ```

    `better-compute serve` answers the host export's routes beside the API, so this serves `GET /watch` on the API's
    listener. `watchAuthorizer(compute, { cookie? })` decides the stream through the server's own envelope: it calls
    `events.list` for one event as the caller (authentication, the tenant pre-check, one `compute:events:read` decision
    on `compute-tenant/{t}`, and the caches), with `?tenantId=` defaulting to the caller's tenant. It returns the caller's
    subject and, when the IAM port has `authorizeSession`, a re-authorizer, so the stream can be closed or re-checked
    later. Without `cookie: true` it accepts only the `Authorization` header.
  
  **Next.js route:**

    ```ts title="app/api/compute/watch/route.ts"
    import { ComputeError, resourceType } from 'better-compute/core';
    import { handleWatch } from 'better-compute/events';
    import { compute, origin } from '@/lib/compute';

    export const runtime = 'nodejs';
    export const dynamic = 'force-dynamic';

    export async function GET(request: Request): Promise<Response> {
      return handleWatch(request, {
        sql: compute.context.sql,
        authorize: async (watch) => {
          // Cookie-authenticated streams need a CSRF guard: require the custom header, check the origin.
          if (watch.headers.get('x-better-compute') !== '1')
            throw new ComputeError('CSRF_REJECTED', 'Send X-Better-Compute: 1');
          const site = watch.headers.get('sec-fetch-site');
          const from = watch.headers.get('origin');
          if ((site && site !== 'same-origin') || (from && from !== origin))
            throw new ComputeError('UNTRUSTED_ORIGIN', 'Origin is not trusted');
          const credential = { headers: watch.headers };
          const caller = await compute.context.iam.authenticate(credential);
          const tenantId = new URL(watch.url).searchParams.get('tenantId') ?? caller.tenantId;
          await compute.context.iam.require(credential, {
            tenantId,
            action: 'compute:events:read',
            resource: { type: resourceType('tenant'), id: tenantId },
          });
          return { tenantId };
        },
      });
    }
    ```

    This is how the console serves `/api/compute/watch`.
  
  **Any Fetch server:**

    ```ts
    import { handleWatch } from 'better-compute/events';

    const watch = (request: Request) =>
      handleWatch(request, { sql: compute.context.sql, authorize: authorizeWatch });

    // Bun, Deno, Hono, or anything that maps a Request to a Response:
    if (new URL(request.url).pathname === '/watch') return watch(request);
    ```
  
<TypeTable
  type="{
  sql: { type: 'SqlExecutor', description: &#x22;The control plane's database executor: compute.context.sql.&#x22;, required: true },
  authorize: {
    type: '(request: Request) => Promise<WatchGrant>',
    description: 'Authenticates and authorizes the request and returns { tenantId, subject?, reauthorize? }: the one tenant whose events it may read, who asked, and how to re-check them. Throw a ComputeError to refuse.',
    required: true,
  },
  registry: { type: 'WatchRegistry', description: 'createWatchRegistry(): where open streams register, so a revocation can close them at once. Share one per process.' },
  reauthorizeMs: { type: 'number', description: 'How often the grant\'s reauthorize runs.', default: '60000' },
  iamOutageMs: { type: 'number', description: 'How long reauthorize may keep failing (IAM unreachable) before the stream closes.', default: '120000' },
  pollMs: { type: 'number', description: 'How often to look for new events when the last read found none.', default: '1000' },
  heartbeatMs: { type: 'number', description: 'How often to send a comment line that keeps proxies from closing an idle stream.', default: '15000' },
  batch: { type: 'number', description: 'Events per outbox read.', default: '200' },
  retryMs: { type: 'number', description: 'The reconnect delay sent to the client as retry:.', default: '2000' },
  maxDurationMs: { type: 'number', description: 'Close the stream after this long; clients reconnect with Last-Event-ID.', default: '3600000 (1 h)' },
}"
/>

## Authorization [#authorization]

`authorize` is the whole security boundary of the stream, so decide it the way the API decides `events.list`:
`compute:events:read` on `compute-tenant/{tenantId}`. The stream then carries only rows whose `tenantId` is the one
`authorize` returned; the filter is part of the outbox query, not applied after it.

* Authenticate with the same credentials the API accepts (a bearer token, a session JWT, or on the same origin the
  IAM cookie) by passing the request's headers to `compute.context.iam.authenticate`.
* Default to the caller's own tenant. Accepting `?tenantId=` lets a platform administrator follow another tenant,
  and the `require` call decides whether they may.
* When the stream accepts a cookie, add a CSRF guard like the Next.js tab above. `GET` requests carry no custom
  header by default, and browsers omit `Origin` on same-origin `GET`s, so the console requires
  `X-Better-Compute: 1`, which a cross-origin page cannot send without a preflight nothing approves.

> **Authenticating is not enough.** 
  An `authorize` that only authenticates lets any member of a tenant follow all of that tenant's events. Decide
  `compute:events:read` as above, or use `watchAuthorizer`, as the example deployments (`examples/shared/compose.ts`)
  do. Without a grant, a member gets 403; a caller without a token, 401.

### Revocation [#revocation]

A stream is authorized when it opens, so the grant says how to end it early:

* **`reauthorize`** runs every `reauthorizeMs` (60 s). `false` closes the stream; a throw counts as IAM unreachable,
  and after `iamOutageMs` (2 minutes) of that the stream closes.
* **`registry`** lets a revocation close matching streams at once: `closeWhere({ agentId | delegationId |
  sessionId | identityId | tenantId + anyAgent }, reason)`. A tenant alone is refused, since it would close too
  much. With `host.iam: { bridge }` and the registry in `host.streams` (or `host.iam.streams`), `serve` installs
  handlers that close an agent's or
  delegation's streams on `delegation:revoke` and `agent:suspend`, every agent's in the tenant on
  `agent:suspend-all`, and an identity's on its offboarding or session revocation (the client reconnects with
  `Last-Event-ID` and is decided again). `dev --sim` passes `host.streams` to its embedded IAM's handlers too, so the
  examples' `/watch` streams close on revocation in development as well.

A closed stream ends with `event: error` carrying `{ "code": "ACCESS_DENIED", "message": <reason> }`, and events read during the
revocation are not sent. `tests/e2e/access.test.ts` checks that a revoked delegation and a member's sign-out close
`/watch` within 5 seconds.

A refusal, or any error before the stream starts, is a JSON error with the code's status, like the API's:
`{ "error": { "code": "ACCESS_DENIED", "message": "…" } }` with 403, `INVALID_INPUT` with 400 for a bad cursor or
type list, and `METHOD_NOT_ALLOWED` with 405 for anything but `GET`.

## The stream [#the-stream]

```text
GET /watch?types=reservation.*,node.connected
Accept: text/event-stream
Authorization: Bearer biam_…
Last-Event-ID: 88213.51119

retry: 2000

id: 88213.51120
event: reservation.active
data: {"id":"evt_01k5y3v0m6y8x7w5t4s3r2p1n0","type":"reservation.active","tenantId":"acme","subject":"reservation/rsv_01k5y3a0b1c2d3e4f5g6h7j8k9","time":"2026-09-24T10:15:02.114Z","sequence":"88213.51120","data":{"state":"active","poolId":"pool_01k5y1z2x3w4v5t6s7r8q9p0n1"}}

: keep-alive 88213.51187
```

The response is `text/event-stream` with `Cache-Control: no-cache, no-transform` and `X-Accel-Buffering: no`, so
nginx-style proxies do not buffer it. It opens with `retry:`, then sends one block per event:

| Field   | Value                                                                                   |
| ------- | --------------------------------------------------------------------------------------- |
| `id`    | the event's outbox position, `txid.seq`, equal to its `sequence`                        |
| `event` | the event type, so `EventSource.addEventListener('reservation.active', …)` works        |
| `data`  | the whole event as one line of JSON, the [event shape](/docs/guides/events#event-shape) |

A comment line `: keep-alive <cursor>` goes out every `heartbeatMs`. If reading the outbox fails mid-stream, the
server sends `event: error` with `data: {"message": …}` and closes; the client reconnects and resumes. After
`maxDurationMs` the server closes the stream on purpose, which also makes clients re-authenticate at least once an
hour.

## Cursors and resume [#cursors-and-resume]

The stream's cursor is the outbox position `<txid>.<seq>`, with a dot. It is every event's `sequence` and every
SSE `id:`.

| Request                             | Starts after                                                                                  |
| ----------------------------------- | --------------------------------------------------------------------------------------------- |
| no cursor                           | the current tail: only events committed from now on; history is not replayed                  |
| `Last-Event-ID: 88213.51119` header | that event                                                                                    |
| `?lastEventId=88213.51119`          | that event (for clients that cannot set the header, such as a first `EventSource` connection) |
| `?lastEventId=0.0`                  | the start of the outbox: everything retention has kept                                        |

The header wins when both are sent. A cursor that is not two groups of 1 to 20 digits joined by a dot is
`INVALID_INPUT`. Note that `events.list` writes the same position with a colon (`88213:51119`); replace the colon
with a dot to move from history to the live stream.

The outbox is read with the xmin rule: only transactions older than the snapshot's xmin, in `(txid, seq)` order. No
event can therefore commit behind a cursor the stream has already handed out, which is what makes resuming exact.
The cost is that an event becomes visible only when every older transaction has finished; see
[commit order](/docs/guides/events#commit-order-and-the-xmin-cursor). A cursor older than what
[retention](/docs/operations/retention) has pruned continues with the oldest row still there, without an error.

To backfill and then follow without a gap, page history with `events.list`, then open the stream with the last
item's `sequence`, colon replaced by a dot, as `Last-Event-ID`.

## Filters [#filters]

`?types=` takes 1 to 20 comma-separated patterns of up to 128 characters, with the same matching as webhooks: `*`
(the default), a family such as `reservation.*` or `node.gpu.*`, or an exact type. An empty list is
`INVALID_INPUT`. Unlike webhooks, patterns are not checked against the event catalog, so a typo matches nothing:
take family names from the [event reference](/docs/reference/events).

Filtering happens after the tenant filter, and the cursor still advances past events that do not match, so a
filtered stream resumes correctly too.

## Consume it [#consume-it]

**The typed client: `$watch`.** `client.$watch({ types?, tenantId?, lastEventId?, signal?, path?, maxBackoffMs? })` from
`@better-compute/client` is an async iterable over the stream. It sends the client's token and headers, yields each
event with its cursor as `sequence`, reconnects with `Last-Event-ID` after the server's `retry:` delay (or with
back-off, up to 30 seconds by default, after failures), stops on `break` or its `signal`, and throws
`UNAUTHENTICATED` on a 401 and `ACCESS_DENIED` on a 403. It reads `{basePath}/watch` by default
(`/api/compute/watch`); pass `path: '/watch'` for a stream mounted at the root, as in the `serve` tab above.
`tenantId` (e2fe6f8) sends `?tenantId=`, which the host's `watchAuthorizer` decides; without it the stream is the
caller's own tenant.

```ts
for await (const event of client.$watch({ types: ['reservation.*'], lastEventId: savedCursor, path: '/watch' })) {
  handle(event);
  savedCursor = event.sequence;
}
```

See [the TypeScript client](/docs/guides/clients/typescript-client#watching-events) for its options.

**The CLI: `events watch`.** `better-compute events watch --url … [--tenant T] [--types …] [--since CURSOR] [--count N]` follows the same stream with `$watch` and prints one terminal-safe JSON line per event; its `--path`
defaults to `/watch` (the console app serves `/api/compute/watch`). `--since` takes the stream's own `<txid>.<seq>`
cursor, the `sequence` of a printed line (`--since 184467.12`, 86706b4), and also the colon form of `events.list`
pages (`184467:12`), the same outbox position, which it sends as the dotted one (867c837); `events list --cursor`
takes only the colon form. See [follow events live](/docs/guides/clients/cli#follow-events-live).

**Browsers and Node.js: console-kit.** `watchEvents` from `@better-compute/console-kit` reads the stream over
`fetch` with callbacks instead of a loop, and can also send an `Authorization` header, which `EventSource` cannot. It keeps the last `id`, reconnects
after the server's `retry:` delay when the server closes the stream, backs off (doubling, up to 60 seconds) after
failures, and sends `Last-Event-ID` on every reconnect.

```ts
import { watchEvents } from '@better-compute/console-kit';

const watcher = watchEvents({
  url: 'https://compute.example.com/watch',
  types: ['reservation.*'],
  lastEventId: savedCursor, // optional: resume across restarts
  headers: async () => ({ authorization: `Bearer ${await getToken()}` }), // fresh per connection
  onEvent: (event) => {
    handle(event);
    savedCursor = event.sequence as string;
  },
  onStatus: (status, detail) => console.log(status, detail?.retryInMs),
});
```

`startLiveConsole(client, { tenantId, watchUrl, headers, permissions })` goes further: it loads the tenant's
stores and routes every event to them. See [console](/docs/guides/clients/console#live-updates).

**The console** uses `watchEvents` against its own `/api/compute/watch` with the IAM cookie and
`X-Better-Compute: 1`, shows a "live" badge while the stream is open, and falls back to refreshing loaded stores
every 20 seconds while it is not.

**`EventSource`** works when the stream accepts a cookie and needs no custom header, or when you put the credential
somewhere your `authorize` reads other than a header. It reconnects with `Last-Event-ID` by itself.

**curl** is enough to look at a stream:

```bash
curl -N https://compute.example.com/watch?types=node.* \
  -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" -H "Accept: text/event-stream"
```

## Delivery guarantees [#delivery-guarantees]

* **At least once across reconnects.** The server never skips or repeats an event after the cursor it was given.
  A client that crashes between handling an event and saving its cursor sees it again, so deduplicate on `id`.
* **In commit order**, within one stream.
* **One tenant per stream.** A platform view of several tenants opens several streams, or uses an
  [event sink](/docs/extending/sinks).
* **No back-pressure on the database.** Each open stream polls the outbox every `pollMs` while idle and reads in
  batches of `batch` while behind. Many idle streams mean many small queries; put the watch route on processes
  sized for it.


# Webhooks (/docs/guides/events/webhooks)

> Subscribe a tenant's HTTPS endpoint to event types, verify the HMAC signature on each delivery, and handle retries, dead letters, redelivery and secret rotation.



A webhook is a tenant's subscription: an HTTPS URL and a list of event type patterns. Every committed event of that
tenant whose type matches is POSTed to the URL as JSON, signed with a secret only the tenant and Better Compute
know. Delivery is at least once, with retries over about two days, a dead-letter state you can inspect, and
redelivery on request.

Webhooks come from the events module. Add it to your deployment with a secret of at least 32 characters, shared by
every process:

```ts
import { eventsModule } from 'better-compute/events';

const compute = betterCompute({
  // …
  modules: [eventsModule({ secret: process.env.BETTER_COMPUTE_SECRET! })],
});
```

The module adds the `webhooks` API group, the tables `webhooks`, `webhook_deliveries` and `event_cursors`
(migrations `events/0001_webhooks` and `events/0002_delivery_retention`), and three reconcilers that a worker process
runs: `events.fanout` every second, `events.deliver` every two seconds and `events.prune` every hour. Without a
worker running reconcilers (the `controllers` role of `better-compute serve`), webhooks are stored but nothing is
sent.

## How a delivery happens [#how-a-delivery-happens]

<Mermaid
  chart="sequenceDiagram
  participant T as Change transaction
  participant O as outbox
  participant F as events.fanout (1 s)
  participant Q as webhook_deliveries
  participant D as events.deliver (2 s)
  participant R as Your endpoint
  T->>O: event row, same transaction as the change
  F->>O: read after the fan-out cursor (xmin rule)
  F->>Q: one pending row per matching webhook, cursor advanced in the same transaction
  D->>Q: claim due rows (FOR UPDATE SKIP LOCKED)
  D->>R: POST signed JSON (10 s timeout)
  alt 2xx
    D->>Q: delivered
  else error, timeout or non-2xx
    D->>Q: pending again after backoff, or dead
  end"
/>

**Fan-out.** `events.fanout` reads the outbox after its cursor in `event_cursors`, with the
same xmin rule as every outbox reader, and queues one delivery for each active webhook of the event's tenant whose
patterns match and that existed when the event was written. The queue insert and the cursor move commit together,
and `(webhook_id, event_id)` is unique, so an event is queued at most once per webhook. The cursor is anchored at the
outbox tail when the control plane starts: history from before the first start is never replayed, and a new webhook
receives events from its creation on.

**Delivery.** `events.deliver` claims up to 50 due deliveries per pass and leases each for the timeout plus 30
seconds, so a crashed worker's claims come back. Within a pass, one webhook's deliveries go out one at a time in the
order they were queued; different webhooks are served in parallel. Each claim counts as an attempt. A delivery of a
paused or deleted webhook is cancelled instead of sent.

## Create a webhook [#create-a-webhook]

  **TypeScript:**

    ```ts
    const { webhook, secret } = await client.webhooks.create({
      tenantId: 'acme',
      url: 'https://hooks.example.com/compute',
      events: ['reservation.*', 'node.disconnected', 'node.lost'],
      description: 'Scheduler sync',
    });
    // Store `secret` now (whsec_...): it is returned once and never again.
    ```
  
  **CLI:**

    ```bash
    better-compute api webhooks.create tenantId=acme url=https://hooks.example.com/compute \
      'events[]=reservation.*' 'events[]=node.disconnected' description="Scheduler sync"
    ```
  
  **HTTP:**

    ```bash
    curl -X POST https://compute.example.com/api/compute/webhooks/create \
      -H "Authorization: Bearer $BETTER_COMPUTE_TOKEN" \
      -H "Content-Type: application/json" -H "X-Better-Compute: 1" \
      -d '{"tenantId":"acme","url":"https://hooks.example.com/compute","events":["reservation.*"]}'
    # → { "data": { "webhook": { "id": "whk_…", … }, "secret": "whsec_…" } }
    ```
  
<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant whose events to deliver.', required: true },
  url: { type: 'string', description: 'An https URL at a public address, without credentials. The fragment is dropped.', required: true },
  events: { type: 'string[]', description: '1 to 20 patterns: *, a family like reservation.* or node.gpu.*, or an exact type. At most 128 characters each; duplicates are removed.', required: true },
  description: { type: 'string', description: 'Up to 256 characters.' },
}"
/>

A tenant can have 20 webhooks (`maxPerTenant`); the 21st is `QUOTA_EXCEEDED`. Every webhooks method is decided as
`compute:webhooks:manage` on `compute-tenant/{tenantId}`.

`webhooks.create` takes no idempotency key, so retrying a create whose answer was lost makes a second webhook with
its own secret, and both receive every matching event. Before retrying, check
[`webhooks.list`](/docs/reference/api/webhooks#list) and [delete](#test-pause-and-delete) the one you do not keep.

### Choosing events [#choosing-events]

A pattern is `*` (everything), a family ending in `.*`, or one exact type. `reservation.*` matches
`reservation.placed` and `reservation.active`; `node.*` also matches `node.gpu.missing`. Patterns are lower case
letters, digits and hyphens between dots.

`create` and `update` check every pattern against the event catalog and refuse one that nothing can match, such as
`nodes.*`, with `INVALID_INPUT` and the offenders in `details.unknown`, instead of letting a typo silently match
nothing. `webhooks.eventTypes` returns the catalog: each family's `type`, `subject`, `module` and `description`.
The [event reference](/docs/reference/events) lists the same catalog, and the [events](/docs/guides/events#event-types)
page lists the individual types you will see in each family.

Modules and plugins that publish their own families register them with `eventsModule({ extraEvents: [...] })` so
tenants can subscribe; `extraEvents: false` turns the check off and accepts any well-formed pattern.

### Webhook URLs [#webhook-urls]

Webhook URLs are tenant input, and the workers that send deliveries sit inside your platform network, so a URL
pointing at a metadata service, a BMC or a database would be a server-side request forgery. Two checks prevent it:

* **When saved**, the URL must be `https`, carry no user name or password, and not be an IP literal of a non-public
  address, `localhost`, `*.localhost` or `*.internal`.
* **At every delivery**, the host is resolved again and the delivery fails if any address it resolves to is
  loopback, private, link-local, carrier-grade NAT, multicast, a documentation range or otherwise non-public (IPv4
  and IPv6, including IPv4-mapped addresses). This catches DNS names that change after the webhook was saved.

For local development only, `eventsModule({ allowInsecure: true })` permits `http:` and `allowPrivate: true`
permits private and loopback destinations.

## Receive and verify [#receive-and-verify]

Each delivery is a `POST` with the event as its JSON body:

```json title="Body"
{
  "id": "evt_01k5y3v0m6y8x7w5t4s3r2p1n0",
  "type": "reservation.active",
  "tenantId": "acme",
  "subject": "reservation/rsv_01k5y3a0b1c2d3e4f5g6h7j8k9",
  "time": "2026-09-24T10:15:02.114Z",
  "sequence": "88213.51120",
  "data": { "state": "active", "poolId": "pool_01k5y1z2x3w4v5t6s7r8q9p0n1" }
}
```

The body is stored when the delivery is queued, so every attempt, and every redelivery, sends the same bytes.
`sequence` is the outbox position in the `txid.seq` form; it is empty for a `webhook.ping`.

| Header                       | Value                                                             |
| ---------------------------- | ----------------------------------------------------------------- |
| `Content-Type`               | `application/json`                                                |
| `User-Agent`                 | `better-compute-webhooks/1`                                       |
| `X-Better-Compute-Event`     | the event type                                                    |
| `X-Better-Compute-Delivery`  | the delivery id (`dlv_…`), the same on every retry and redelivery |
| `X-Better-Compute-Webhook`   | the webhook id (`whk_…`)                                          |
| `X-Better-Compute-Timestamp` | the signing time, in epoch seconds                                |
| `X-Better-Compute-Signature` | `v1=<hex>`, or several comma-separated during a secret rotation   |

The signature is the scheme Better IAM uses for its own webhooks: an HMAC-SHA256, keyed with the webhook's secret,
over the timestamp, a dot and the raw body, `${timestamp}.${body}`, as lower-case hex. Verify it before you trust
anything in the body, and check that the timestamp is recent so an old delivery cannot be replayed.

```ts title="app/hooks/compute/route.ts"
import { verifyWebhookSignature } from 'better-compute/events';

export async function POST(request: Request): Promise<Response> {
  const body = await request.text(); // the raw bytes: never re-serialize before verifying
  const valid = verifyWebhookSignature({
    secret: process.env.COMPUTE_WEBHOOK_SECRET!,
    signature: request.headers.get('x-better-compute-signature') ?? '',
    timestamp: request.headers.get('x-better-compute-timestamp') ?? '',
    body,
  });
  if (!valid) return new Response('invalid signature', { status: 401 });

  const delivery = request.headers.get('x-better-compute-delivery')!;
  if (await alreadyHandled(delivery)) return new Response(null, { status: 204 });
  const event = JSON.parse(body);
  await enqueue(event); // do the work elsewhere: answer within 10 seconds
  await markHandled(delivery);
  return new Response(null, { status: 204 });
}
```

`verifyWebhookSignature({ secret, signature, timestamp, body, toleranceSeconds?, now? })` returns true when any
`v1=` value in the header matches, compared in constant time, and the timestamp is within `toleranceSeconds` of now
(300 by default, in either direction). In another language, compute the same HMAC and compare it with each `v1=`
part:

```bash
printf '%s.%s' "$TIMESTAMP" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex
```

**Answer quickly, and idempotently.** Any 2xx response marks the delivery `delivered`; its body is ignored.
Anything else, a redirect (redirects are not followed), a connection error or no answer within 10 seconds is a
failed attempt. Because delivery is at least once, the same event can arrive twice, after a timeout that your
endpoint in fact handled or after a redelivery. Deduplicate on `X-Better-Compute-Delivery` (one per webhook and
event) or on the event `id`, and order by `sequence`, not by arrival: a delivery that failed is retried later
without holding back the ones queued after it.

## Retries and dead letters [#retries-and-dead-letters]

A failed attempt is retried after a delay that grows with each attempt. The eighth failed attempt makes the delivery
`dead`.

| After attempt | 1     | 2     | 3      | 4   | 5   | 6    | 7    | 8    |
| ------------- | ----- | ----- | ------ | --- | --- | ---- | ---- | ---- |
| Next try in   | 1 min | 5 min | 30 min | 2 h | 6 h | 12 h | 24 h | dead |

That is about 44 hours and 36 minutes from the first attempt to the last. `eventsModule({ delivery })` changes the
policy for a deployment: `maxAttempts` (8), `backoffSeconds` (the table above; the last value repeats), `timeoutMs`
(10000) and `limit` (50 deliveries per pass). Each failure records `lastStatus` (the HTTP status, when there was
one) and `lastError`.

| State       | Meaning                                                          |
| ----------- | ---------------------------------------------------------------- |
| `pending`   | queued or waiting for its next attempt (`nextAttemptAt`)         |
| `delivered` | an attempt got a 2xx; `deliveredAt` is set                       |
| `dead`      | every attempt failed                                             |
| `cancelled` | the webhook was paused or deleted while the delivery was pending |

Inspect a webhook's history, newest first, and put deliveries back in the queue once the endpoint is fixed:

```ts
const { items } = await client.webhooks.deliveries({ tenantId: 'acme', webhookId, state: 'dead', limit: 100 });
for (const delivery of items)
  await client.webhooks.redeliver({ tenantId: 'acme', deliveryId: delivery.id });
```

`redeliver` works on a `dead`, `cancelled` or `delivered` delivery: it becomes `pending`, due now, with its attempt
count reset to zero, and keeps its id and body. It refuses with `NOT_FOUND` when the delivery does not exist in the
tenant or its webhook is paused. `deliveries` returns at most 200 items (50 by default).

`better-compute doctor` reports `events.webhook-dead-letters` while any delivery is dead, and
`events.webhook-backlog` when more than 1,000 deliveries are over five minutes late, which usually means no worker
runs `events.deliver`.

### How long history is kept [#how-long-history-is-kept]

`events.prune` deletes settled delivery history once it is old enough, measured from when the delivery was queued:
`delivered` and `cancelled` deliveries after 30 days, `dead` ones after 90, so there is time to fix an endpoint and
redeliver. It deletes 5,000 rows per run and runs again a second later while a batch comes back full. `pending`
deliveries are never deleted, whatever their age.

```ts
eventsModule({ secret, retention: { deliveredDays: 14, deadDays: 60 } }); // retention: false keeps everything
```

A period under one day is `INVALID_CONFIG` when the module is defined. `pruneWebhookDeliveries(sql, retention?,
limit?)` runs one batch, for hosts that schedule pruning themselves. See [retention](/docs/operations/retention).

## Test, pause and delete [#test-pause-and-delete]

* **`ping`** queues a `webhook.ping` event to one webhook, due now, whatever its patterns. Its data is
  `{ webhookId }`. The result is the queued delivery, so you can follow it with `deliveries`.
* **`update`** changes `url`, `events`, `description` or `active`, with the same checks as
  `create`. Setting `active: false` pauses the webhook and cancels its pending deliveries; events that happen while
  it is paused are not queued for it. Setting `active: true` resumes it from new events on; redeliver cancelled
  deliveries you still want.
* **`delete`** removes the webhook and its whole delivery history.

## Rotate the secret [#rotate-the-secret]

`rotateSecret` returns a new secret, once. With `keepPreviousSeconds` (up to 86400, one day), the old secret keeps
signing too until then, so each delivery carries two signatures, the new one first. Receivers that accept either
secret during the window switch without dropping a delivery:

```ts
const { secret, webhook } = await client.webhooks.rotateSecret({
  tenantId: 'acme',
  webhookId,
  keepPreviousSeconds: 3600,
});
// webhook.previousSecretExpiresAt: when the old secret stops signing
```

```ts title="Receiver during a rotation"
const valid = [process.env.COMPUTE_WEBHOOK_SECRET!, process.env.COMPUTE_WEBHOOK_SECRET_OLD]
  .filter((secret): secret is string => Boolean(secret))
  .some((secret) => verifyWebhookSignature({ secret, signature, timestamp, body }));
```

Without `keepPreviousSeconds` the old secret stops at once.

**At rest**, signing secrets are sealed with AES-256-GCM under a key derived with HKDF-SHA256 from the deployment
secret and bound to the webhook id, so a sealed value copied to another row does not open. When you rotate the
deployment secret itself, pass the old one in `eventsModule({ secret, previousSecrets: [old] })` so existing rows
still open.

## The webhooks API [#the-webhooks-api]

| Method                                                      | Input                                                                 | Returns                                      |
| ----------------------------------------------------------- | --------------------------------------------------------------------- | -------------------------------------------- |
| [`create`](/docs/reference/api/webhooks#create)             | `tenantId`, `url`, `events`, `description?`                           | `{ webhook, secret }`                        |
| [`list`](/docs/reference/api/webhooks#list)                 | `tenantId`                                                            | `{ items: Webhook[] }`, oldest first         |
| [`get`](/docs/reference/api/webhooks#get)                   | `tenantId`, `webhookId`                                               | `Webhook`                                    |
| [`update`](/docs/reference/api/webhooks#update)             | `tenantId`, `webhookId`, `url?`, `events?`, `description?`, `active?` | `Webhook`                                    |
| [`rotateSecret`](/docs/reference/api/webhooks#rotatesecret) | `tenantId`, `webhookId`, `keepPreviousSeconds?` (0 to 86400)          | `{ webhook, secret }`                        |
| [`delete`](/docs/reference/api/webhooks#delete)             | `tenantId`, `webhookId`                                               | `{ deleted: true }`                          |
| [`ping`](/docs/reference/api/webhooks#ping)                 | `tenantId`, `webhookId`                                               | `{ delivery }`                               |
| [`deliveries`](/docs/reference/api/webhooks#deliveries)     | `tenantId`, `webhookId`, `state?`, `limit?` (1 to 200)                | `{ items: WebhookDelivery[] }`, newest first |
| [`redeliver`](/docs/reference/api/webhooks#redeliver)       | `tenantId`, `deliveryId`                                              | `{ delivery }`                               |
| [`eventTypes`](/docs/reference/api/webhooks#eventtypes)     | `tenantId`                                                            | `{ items: EventFamily[] }`                   |

All ten are decided as `compute:webhooks:manage` on `compute-tenant/{tenantId}`. A webhook of another tenant is
`NOT_FOUND`.

<TypeTable
  type="{
  id: { type: 'string', description: 'whk_ followed by a sortable id.' },
  tenantId: { type: 'string', description: 'The owning tenant.' },
  url: { type: 'string', description: 'The normalized URL.' },
  description: { type: 'string', description: 'When set.' },
  events: { type: 'string[]', description: 'The patterns.' },
  active: { type: 'boolean', description: 'false while paused.' },
  createdAt: { type: 'number', description: 'Epoch milliseconds. Only events written after it are delivered.' },
  updatedAt: { type: 'number', description: 'Epoch milliseconds.' },
  previousSecretExpiresAt: { type: 'number', description: 'While a rotated-out secret still signs.' },
  version: { type: 'number', description: 'Raised by every update and rotation.' },
}"
/>

A `WebhookDelivery` has `id` (`dlv_…`), `webhookId`, `tenantId`, `eventId`, `eventType`, `state`, `attempts`,
`nextAttemptAt`, `createdAt`, and `lastStatus`, `lastError` and `deliveredAt` when they apply. Times are epoch
milliseconds.

## Webhooks or watch [#webhooks-or-watch]

Webhooks suit another service of the tenant that must see every event even while it is down: the position lives in
Better Compute, and failed deliveries wait for it. For a browser, a console or a script that follows events while it
runs, [watch](/docs/guides/events/watch) streams them over one connection and resumes from a cursor the client
keeps. For a platform-wide feed into a warehouse or a queue, use an [event sink](/docs/extending/sinks).


# Actions and resources (/docs/guides/iam/actions-and-resources)

> The compute:* actions and compute-* resource types Better Compute adds to Better IAM, the method-to-action table, list filtering and cross-tenant refusal.



Every protected method asks Better IAM one question: may this caller perform this **action** on this
**resource** in this tenant? Actions are written `compute:<area>:<verb>`, such as
`compute:reservations:create`. Resources are written `<type>/<id>`, such as `compute-pool/{poolId}`. Both come
from compute's catalog, which the bridge installs into every Better IAM host through `bridge.plugin()`, and they
are the names you write [policies](/docs/guides/iam/policies) against.

## Resource types [#resource-types]

All of compute's resource types are app-owned: nothing is ever registered in Better IAM. When IAM needs a
resource, it calls compute's resolver, which answers from the memory-resident
authz projection. The prefix is `compute-` by default; `typePrefix` changes it
(it must match `^[a-z][a-z0-9-]{0,31}$`) and must be the same in `betterCompute()` and `createComputeIamBridge()`.

| Type                       | What it is                                                                                                                                                                                                                             | Parent         | Relations                             | Declared attributes                                                                              |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `compute-fleet`            | The platform fleet. A scope type whose only id is `global`.                                                                                                                                                                            |                |                                       |                                                                                                  |
| `compute-tenant`           | A tenant's compute scope. A scope type whose id is the tenant id.                                                                                                                                                                      |                |                                       |                                                                                                  |
| `compute-project`          | A project                                                                                                                                                                                                    |                | `member`, `admin`                     |                                                                                                  |
| `compute-site`             | A boot and failure domain                                                                                                                                                                                                              |                |                                       | `region`, `iamRegion`, `kind`                                                                    |
| `compute-pool`             | A scheduling and policy domain                                                                                                                                                                                                         |                | `viewer`, `user`, `operator`, `admin` | `region`, `class`, `gpuModel`, `gpuMemoryGb`, `sharing`                                          |
| `compute-node`             | A machine                                                                                                                                                                                                                              | `compute-pool` | `operator`                            | `region`, `gpuModel`, `gpuCount`, `lifecycle`, `trust`, `dedicatedTo`                            |
| `compute-reservation`      | A GPU reservation, decided while it is still a draft                                                                                                                                                           | `compute-pool` |                                       | `region`, `gpuModel`, `gpuCount`, `durationHours`, `binding`, `projectId`, `spot`                |
| `compute-instance`         | A VPS instance                                                                                                                                                                                                                         | `compute-pool` |                                       | `vcpu`, `memoryGiB`, `gpus`, `imageId`, `imageVerified`, `projectId`                             |
| `compute-image`            | An OS or ramdisk image                                                                                                                                                                                                                 |                |                                       | `visibility`, `verified`                                                                         |
| `compute-profile`          | A provisioning profile                                                                                                                                                                                                                 |                |                                       |                                                                                                  |
| `compute-provider-account` | An account with an external capacity provider                                                                                                                                                                                          |                |                                       |                                                                                                  |
| `compute-sandbox`          | A [sandbox](/docs/guides/vps/sandboxes) (v0.5): a VPS instance with a template and a lifetime, decided while it is still a draft                                                                                                       | `compute-pool` |                                       | `templateId`, `vcpu`, `memoryGiB`, `gpus`, `ttlSeconds`, `warm`, and `fromSnapshot` on a restore |
| `compute-sandbox-template` | A tenant's sandbox template (v0.5): pool, flavor, image, runtime, TTL bounds, warm pool                                                                                                                                                |                |                                       |                                                                                                  |
| `compute-sandbox-snapshot` | A sandbox's snapshot (v0.5): disk and memory, tenant-private, owned by its sandbox's owner whoever took it; may outlive its sandbox                                                                                                    |                |                                       | `sandboxId`, `templateId`, `includeMemory`                                                       |
| `compute-sandbox-volume`   | A tenant's persistent [sandbox volume](/docs/guides/vps/sandboxes#volumes) (v1.1, agent memory): a disk on one host of a pool that its sandboxes attach across runs, owned by whoever created it; decided before it exists on `create` | `compute-pool` |                                       | `sizeGiB` (pending on `resize`)                                                                  |

Besides a row's attributes, every resolved resource carries `ownerId` when the row has an owner, and the prefixed
`parentType` and `parentId` of its parent. Better IAM turns the parent into `resource.parentRelations`, so a
policy on a reservation can test the caller's relations on its pool. The attributes each row actually carries
today are listed under [Resource attributes](/docs/guides/iam/policies).

### Scope types [#scope-types]

Three types exist only as scopes, so a decision about "everything in this tenant" or "the whole fleet" has a
resource to name:

* `compute-fleet/global` resolves only in the root tenant. Fleet-wide methods are decided in the root tenant,
  whatever `tenantId` their input names.
* `compute-tenant/{t}` resolves only when its id is the decision's tenant.
* `compute-project/{id}` resolves in the project's owner tenant.

### How the resolver answers [#how-the-resolver-answers]

The resolver runs inside Better IAM's locked decision transaction, so a hit must do no I/O:

* It echoes `{ tenantId, type, id }` exactly. Better IAM refuses anything else as `RESOURCE_MISMATCH`.
* A row missing from memory costs one primary-key read, bounded to 20 ms. A row that does not exist is
  `NOT_FOUND`, which Better IAM does not audit and which aborts a whole `authorizeMany` batch.
* A lookup that times out, or a projection that is not loaded yet, denies (`ACCESS_DENIED`).
* Types without compute's prefix go to the `hostFallback` you passed to `bridge.resolveResource()`, or fail with
  `RESOURCE_RESOLVER_REQUIRED`.

Each process loads the projection at start and then tails `authz_changes` about once a second. Staleness can
cause a spurious denial, never a wrong mutation, because compute re-validates inside its own transaction.

Admissions never decide on a stale copy. After an admission's first transaction commits the draft (or the changed
row, as in `reservations.extend`), the envelope calls the optional `IamPort.refreshResource(tenantId, resource)`
before `iam.require`. The embedded port from `bridge.embedded()` implements it with `AuthzProjection.refresh(type,
id, tenantId)`, which re-reads that one row; on a miss or a failed read it drops the row, so the decision takes the
bounded miss path above and fails closed. Before this, `reservations.extend` decided on the row cached at creation,
so a `BoundedExtend`-style policy capped at 72 hours could be passed by one extra hour within the tail interval.
`tests/server-example-policies.test.ts` runs the example policies against real drafts to hold this.

In remote mode the IAM host decides on its replica's copy, so pass `refresh: { url, secret, sql? }` to
`bridge.remote()` (or `remoteIam()`, which it wraps):
the port then posts a signed `{ tenantId, type, id, lsn }` to the IAM host's refresh handler
(`computeIamHost().refreshHandler`), which waits at most 2 seconds until its replica has replayed compute's WAL
position from after transaction 1 and re-reads the row. Any failure fails the admission with `UNAVAILABLE` (fail
closed). Without `refresh` the remote port has no `refreshResource`, remote admissions can decide on a row up to the
replica lag plus 1 second old, and `better-compute doctor` warns `iam.refresh`.

The remote port forwards only identity and client-context headers with the caller's credential (`authorization`,
`cookie` when there is no bearer, `origin`, `user-agent`, `x-request-id`), never the client's own
`x-forwarded-for`, `x-real-ip` or `forwarded` (e7092bf, security review 14 L3): behind a proxy that appends to them,
a caller could choose the `request.sourceIp` its policy conditions see. The address IAM sees is
`Credential.sourceIp`, which the host verified and the port sends as `x-forwarded-for`. `compute.nodeHandler` sets it
from the socket's peer address; a fetch host passes it as `compute.handler(request, { peer: { address } })`
(d269738), from the socket or from the hop a proxy it trusts reported. Without it IAM sees no source address. See
[hosting the API](/docs/operations/deployment#the-caller-address).

## Actions [#actions]

The core catalog declares 56 actions. Most areas have a `read` action and a `manage`, `create` or finer verb.
Where an action appears on a scope type as well as on an item type, the scope form gates list and create calls
and the item form gates calls on one resource:

| Action                                                                                                                                                                                                                         | Resource types that carry it                                                                                                             |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `compute:fleet:admin`                                                                                                                                                                                                          | `compute-fleet`                                                                                                                          |
| `compute:sites:read`                                                                                                                                                                                                           | `compute-tenant`, `compute-site`                                                                                                         |
| `compute:sites:manage`                                                                                                                                                                                                         | `compute-fleet`, `compute-tenant`, `compute-site`                                                                                        |
| `compute:pools:read`                                                                                                                                                                                                           | `compute-tenant`, `compute-pool`                                                                                                         |
| `compute:pools:create`                                                                                                                                                                                                         | `compute-fleet`, `compute-tenant`                                                                                                        |
| `compute:pools:manage`                                                                                                                                                                                                         | `compute-pool`                                                                                                                           |
| `compute:pools:share`                                                                                                                                                                                                          | `compute-pool`                                                                                                                           |
| `compute:capacity:read`                                                                                                                                                                                                        | `compute-tenant`, `compute-pool`                                                                                                         |
| `compute:quotas:read`                                                                                                                                                                                                          | `compute-tenant`                                                                                                                         |
| `compute:quotas:manage`                                                                                                                                                                                                        | `compute-fleet`                                                                                                                          |
| `compute:join-tokens:create`, `compute:join-tokens:read`, `compute:join-tokens:revoke`                                                                                                                                         | `compute-pool`                                                                                                                           |
| `compute:flavors:read`, `compute:flavors:manage`                                                                                                                                                                               | `compute-pool`                                                                                                                           |
| `compute:nodes:read`                                                                                                                                                                                                           | `compute-tenant`, `compute-node`                                                                                                         |
| `compute:nodes:provision`                                                                                                                                                                                                      | `compute-pool`, `compute-node`                                                                                                           |
| `compute:nodes:update`, `compute:nodes:cordon`, `compute:nodes:drain`, `compute:nodes:power`, `compute:nodes:exec`, `compute:nodes:console`, `compute:nodes:ssh-root`, `compute:nodes:delete`, `compute:nodes:revoke-identity` | `compute-node`                                                                                                                           |
| `compute:workflows:read`, `compute:workflows:manage`                                                                                                                                                                           | `compute-node`                                                                                                                           |
| `compute:reservations:read`                                                                                                                                                                                                    | `compute-tenant`, `compute-reservation`                                                                                                  |
| `compute:reservations:create`, `compute:reservations:extend`, `compute:reservations:release`                                                                                                                                   | `compute-reservation`                                                                                                                    |
| `compute:instances:read`                                                                                                                                                                                                       | `compute-tenant`, `compute-instance`                                                                                                     |
| `compute:instances:create`, `compute:instances:update`, `compute:instances:exec`, `compute:instances:delete`                                                                                                                   | `compute-instance`                                                                                                                       |
| `compute:projects:read`, `compute:projects:manage`                                                                                                                                                                             | `compute-tenant`, `compute-project`                                                                                                      |
| `compute:usage:read`                                                                                                                                                                                                           | `compute-tenant`, `compute-project`, `compute-fleet` (the cross-tenant platform view and dead letters; root tenant only)                 |
| `compute:usage:manage`                                                                                                                                                                                                         | `compute-fleet` (retry dead letters and void pending windows; root tenant only)                                                          |
| `compute:events:read`, `compute:webhooks:manage`                                                                                                                                                                               | `compute-tenant`                                                                                                                         |
| `compute:secrets:manage`                                                                                                                                                                                                       | `compute-fleet`, `compute-tenant`                                                                                                        |
| `compute:drivers:read`                                                                                                                                                                                                         | `compute-fleet`, `compute-tenant`                                                                                                        |
| `compute:billing:read`, `compute:billing:manage`                                                                                                                                                                               | `compute-fleet` (the platform's meters and prices; root tenant only)                                                                     |
| `compute:profiles:read`, `compute:profiles:manage`                                                                                                                                                                             | `compute-tenant`, `compute-profile`                                                                                                      |
| `compute:images:read`                                                                                                                                                                                                          | `compute-tenant`, `compute-image`                                                                                                        |
| `compute:images:publish`                                                                                                                                                                                                       | `compute-fleet`, `compute-tenant`, `compute-image`                                                                                       |
| `compute:providers:read`                                                                                                                                                                                                       | `compute-tenant`, `compute-provider-account`                                                                                             |
| `compute:providers:manage`                                                                                                                                                                                                     | `compute-fleet`, `compute-tenant`, `compute-provider-account`                                                                            |
| `compute:sandboxes:read`, `compute:sandboxes:manage`                                                                                                                                                                           | `compute-tenant`, `compute-sandbox-template`, `compute-sandbox-volume` (`read` also on `compute-sandbox` and `compute-sandbox-snapshot`) |
| `compute:sandboxes:create`, `compute:sandboxes:update`, `compute:sandboxes:exec`, `compute:sandboxes:delete`                                                                                                                   | `compute-sandbox` (`delete` also on `compute-sandbox-snapshot`)                                                                          |
| `compute:batches:read`, `compute:batches:create`, `compute:batches:approve`                                                                                                                                                    | `compute-tenant` (v1.0 destructive-batch quorum)                                                                                         |

### Adding actions [#adding-actions]

Modules and plugins extend the catalog with a `catalog` fragment, which `createComputeIamBridge()` merges into the
types above. The built-in modules add none: the core catalog covers all of their methods. The merge enforces
Better IAM's limits and fails with `INVALID_CONFIG` on a breach:

* An action matches `^compute:[a-z][a-z0-9-]*(:[a-z][a-z0-9-]*)+$`. A plugin's actions must start with
  `compute:<pluginId>:`.
* A type has at most 64 attributes and 32 relations, and its name at most 64 characters. Attribute types are
  `string`, `number` or `boolean`.
* Two fragments may not disagree on a type's parent or on an attribute's type. A parent must be in the catalog.

```ts title="acme-autoinstall.ts"
import { definePlugin } from 'better-compute/core';

export const acmeAutoinstall = () =>
  definePlugin({
    id: 'acme-autoinstall',
    catalog: {
      // Merged into compute-node; plugin actions live under compute:<id>:.
      resourceTypes: { node: { actions: ['compute:acme-autoinstall:render'] } },
    },
    // api(context) { ...methods that declare action 'compute:acme-autoinstall:render' }
  });
```

Pass the plugin to both `betterCompute({ plugins })` and `createComputeIamBridge({ plugins })`, so the API and
every IAM host agree. `initialize()` checks the method registry against the merged catalog: a method whose action
is missing (`uncatalogued`) or that disagrees with its method-table row (`mismatch`) fails with `INVALID_CONFIG`,
and a non-plugin method with no row (`undeclared`) is a `catalog.undeclared` warning in `better-compute doctor`.
See [Modules](/docs/extending/modules) and [Methods](/docs/extending/methods).

## Method-to-action table [#method-to-action-table]

This table is rendered from the live method registry. The normative source is PLAN.md §9.2, kept as `apiSurface`
in `@better-compute/core`: `scripts/authz-matrix.mjs` generates one test per row, and
`tests/iam-bridge-matrix.test.ts` checks every action and type pair against a real Better IAM carrying the
bridge, allowing each only when granted. It decides through compute's own `IamPort`: `bridge.embedded` under
`pnpm test`, and `bridge.remote` over the IAM's HTTP API under `pnpm test:iam-remote`.

* **Decided on** is the resource type of the one decision. Its id comes from the input: the pool for
  `compute-pool`, the tenant for `compute-tenant`, `global` for `compute-fleet`.
* **or** means the method decides on one of two resources depending on its input. For example, `sites.create`
  decides on `compute-fleet` for `shared` and `dedicated` sites and on `compute-tenant` for a tenant's `custom`
  site, and `usage.list` decides on `compute-project` when the input names a project.
* **+ items** marks a list that follows the [list recipe](#list-methods-and-item-filtering).
* **planned** marks a method this release does not serve yet; it arrives in the phase shown.

### Platform and tenancy [#platform-and-tenancy]

| Method                                                        | Action                    | Decided on                          | Phase |
| ------------------------------------------------------------- | ------------------------- | ----------------------------------- | ----- |
| [`system.whoami`](/docs/reference/api/system#whoami)          | none (public)             | nothing                             | v0.1  |
| [`system.version`](/docs/reference/api/system#version)        | none (public)             | nothing                             | v0.1  |
| [`system.selfCheck`](/docs/reference/api/system#selfcheck)    | `compute:fleet:admin`     | `compute-fleet`                     | v0.1  |
| [`projects.create`](/docs/reference/api/projects#create)      | `compute:projects:manage` | `compute-tenant`                    | v0.1  |
| [`projects.get`](/docs/reference/api/projects#get)            | `compute:projects:read`   | `compute-project`                   | v0.1  |
| [`projects.list`](/docs/reference/api/projects#list)          | `compute:projects:read`   | `compute-tenant` + items            | v0.1  |
| [`projects.archive`](/docs/reference/api/projects#archive)    | `compute:projects:manage` | `compute-project`                   | v0.2  |
| [`sites.list`](/docs/reference/api/sites#list)                | `compute:sites:read`      | `compute-tenant` + items            | v0.1  |
| [`sites.get`](/docs/reference/api/sites#get)                  | `compute:sites:read`      | `compute-site`                      | v0.1  |
| [`sites.create`](/docs/reference/api/sites#create)            | `compute:sites:manage`    | `compute-fleet` or `compute-tenant` | v0.1  |
| [`sites.update`](/docs/reference/api/sites#update)            | `compute:sites:manage`    | `compute-site`                      | v0.2  |
| [`sites.delete`](/docs/reference/api/sites#delete)            | `compute:sites:manage`    | `compute-site`                      | v0.2  |
| [`siteTokens.create`](/docs/reference/api/site-tokens#create) | `compute:sites:manage`    | `compute-site`                      | v0.2  |
| [`siteTokens.revoke`](/docs/reference/api/site-tokens#revoke) | `compute:sites:manage`    | `compute-site`                      | v0.2  |
| [`secrets.put`](/docs/reference/api/secrets#put)              | `compute:secrets:manage`  | `compute-tenant` or `compute-fleet` | v0.2  |
| [`secrets.delete`](/docs/reference/api/secrets#delete)        | `compute:secrets:manage`  | `compute-tenant` or `compute-fleet` | v0.2  |
| [`drivers.list`](/docs/reference/api/drivers#list)            | `compute:drivers:read`    | `compute-tenant`                    | v0.2  |

### Nodes and access [#nodes-and-access]

| Method                                                              | Action                                                                               | Decided on                                                      | Phase |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------- | ----- |
| [`joinTokens.create`](/docs/reference/api/join-tokens#create)       | `compute:join-tokens:create`                                                         | `compute-pool`                                                  | v0.1  |
| [`joinTokens.list`](/docs/reference/api/join-tokens#list)           | `compute:join-tokens:read`                                                           | `compute-pool`                                                  | v0.1  |
| [`joinTokens.revoke`](/docs/reference/api/join-tokens#revoke)       | `compute:join-tokens:revoke`                                                         | `compute-pool`                                                  | v0.1  |
| [`joinRequests.list`](/docs/reference/api/join-requests#list)       | `compute:join-tokens:read`                                                           | `compute-pool`                                                  | v0.2  |
| [`joinRequests.approve`](/docs/reference/api/join-requests#approve) | `compute:join-tokens:create`                                                         | `compute-pool`                                                  | v0.2  |
| [`joinRequests.reject`](/docs/reference/api/join-requests#reject)   | `compute:join-tokens:create`                                                         | `compute-pool`                                                  | v0.2  |
| [`nodes.list`](/docs/reference/api/nodes#list)                      | `compute:nodes:read`                                                                 | `compute-tenant` + items                                        | v0.1  |
| [`nodes.get`](/docs/reference/api/nodes#get)                        | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.1  |
| [`nodes.events`](/docs/reference/api/nodes#events)                  | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.1  |
| [`nodes.cordon`](/docs/reference/api/nodes#cordon)                  | `compute:nodes:cordon`                                                               | `compute-node`                                                  | v0.1  |
| [`nodes.uncordon`](/docs/reference/api/nodes#uncordon)              | `compute:nodes:cordon`                                                               | `compute-node`                                                  | v0.1  |
| [`nodes.drain`](/docs/reference/api/nodes#drain)                    | `compute:nodes:drain`                                                                | `compute-node`                                                  | v0.1  |
| [`nodes.label`](/docs/reference/api/nodes#label)                    | `compute:nodes:update`                                                               | `compute-node`                                                  | v0.1  |
| [`nodes.taint`](/docs/reference/api/nodes#taint)                    | `compute:nodes:update`                                                               | `compute-node`                                                  | v0.1  |
| [`nodes.adopt`](/docs/reference/api/nodes#adopt)                    | `compute:nodes:provision`                                                            | `compute-node`                                                  | v0.1  |
| [`nodes.revokeIdentity`](/docs/reference/api/nodes#revokeidentity)  | `compute:nodes:revoke-identity`                                                      | `compute-node`                                                  | v0.1  |
| [`nodes.delete`](/docs/reference/api/nodes#delete)                  | `compute:nodes:delete`                                                               | `compute-node`                                                  | v0.2  |
| [`nodes.issueSshCert`](/docs/reference/api/nodes#issuesshcert)      | `compute:nodes:ssh-root`                                                             | `compute-node`                                                  | v0.3  |
| [`nodes.revokeSshCert`](/docs/reference/api/nodes#revokesshcert)    | `compute:nodes:ssh-root`                                                             | `compute-node`                                                  | v0.3  |
| [`nodes.listSshCerts`](/docs/reference/api/nodes#listsshcerts)      | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.3  |
| [`inventory.get`](/docs/reference/api/inventory#get)                | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.1  |
| [`inventory.history`](/docs/reference/api/inventory#history)        | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.2  |
| [`inventory.diff`](/docs/reference/api/inventory#diff)              | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.2  |
| [`commands.run`](/docs/reference/api/commands#run)                  | `compute:nodes:exec`                                                                 | `compute-node`                                                  | v0.1  |
| [`commands.get`](/docs/reference/api/commands#get)                  | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.1  |
| [`commands.list`](/docs/reference/api/commands#list)                | `compute:nodes:read`                                                                 | `compute-node`                                                  | v0.1  |
| [`commands.cancel`](/docs/reference/api/commands#cancel)            | `compute:nodes:exec`                                                                 | `compute-node`                                                  | v0.2  |
| `commandBatches.create`                                             | `compute:batches:create`                                                             | `compute-tenant`                                                | v1.0  |
| `commandBatches.get`, `list`, `scope`                               | `compute:batches:read`                                                               | `compute-tenant`                                                | v1.0  |
| `commandBatches.approve`, `cancel`                                  | `compute:batches:approve` (on top of the approver signatures, never instead of them) | `compute-tenant`                                                | v1.0  |
| [`tunnels.open`](/docs/reference/api/tunnels#open)                  | `compute:instances:exec`                                                             | `compute-instance` or `compute:nodes:console` on `compute-node` | v0.3  |

### Capacity [#capacity]

| Method                                                             | Action                         | Decided on                          | Phase |
| ------------------------------------------------------------------ | ------------------------------ | ----------------------------------- | ----- |
| [`pools.list`](/docs/reference/api/pools#list)                     | `compute:pools:read`           | `compute-tenant` + items            | v0.1  |
| [`pools.get`](/docs/reference/api/pools#get)                       | `compute:pools:read`           | `compute-pool`                      | v0.1  |
| [`pools.create`](/docs/reference/api/pools#create)                 | `compute:pools:create`         | `compute-fleet` or `compute-tenant` | v0.1  |
| [`pools.capacity`](/docs/reference/api/pools#capacity)             | `compute:capacity:read`        | `compute-pool`                      | v0.1  |
| [`pools.offer`](/docs/reference/api/pools#offer)                   | `compute:pools:share`          | `compute-pool`                      | v0.1  |
| [`pools.unoffer`](/docs/reference/api/pools#unoffer)               | `compute:pools:share`          | `compute-pool`                      | v0.1  |
| [`pools.update`](/docs/reference/api/pools#update)                 | `compute:pools:manage`         | `compute-pool`                      | v0.2  |
| [`pools.archive`](/docs/reference/api/pools#archive)               | `compute:pools:manage`         | `compute-pool`                      | v0.2  |
| [`capacity.summary`](/docs/reference/api/capacity#summary)         | `compute:capacity:read`        | `compute-tenant`                    | v0.1  |
| [`capacity.byPool`](/docs/reference/api/capacity#bypool)           | `compute:capacity:read`        | `compute-pool`                      | v0.2  |
| [`capacity.preview`](/docs/reference/api/capacity#preview)         | `compute:capacity:read`        | `compute-pool`                      | v0.2  |
| [`capacity.timeline`](/docs/reference/api/capacity#timeline)       | `compute:capacity:read`        | `compute-pool`                      | v0.2  |
| [`reservations.create`](/docs/reference/api/reservations#create)   | `compute:reservations:create`  | `compute-reservation` (the draft)   | v0.1  |
| [`reservations.get`](/docs/reference/api/reservations#get)         | `compute:reservations:read`    | `compute-reservation`               | v0.1  |
| [`reservations.list`](/docs/reference/api/reservations#list)       | `compute:reservations:read`    | `compute-tenant` + items            | v0.1  |
| [`reservations.release`](/docs/reference/api/reservations#release) | `compute:reservations:release` | `compute-reservation`               | v0.1  |
| [`reservations.cancel`](/docs/reference/api/reservations#cancel)   | `compute:reservations:release` | `compute-reservation`               | v0.2  |
| [`reservations.extend`](/docs/reference/api/reservations#extend)   | `compute:reservations:extend`  | `compute-reservation`               | v0.1  |
| [`reservations.renew`](/docs/reference/api/reservations#renew)     | `compute:reservations:extend`  | `compute-reservation`               | v0.1  |
| [`quotas.get`](/docs/reference/api/quotas#get)                     | `compute:quotas:read`          | `compute-tenant`                    | v0.1  |
| [`quotas.set`](/docs/reference/api/quotas#set)                     | `compute:quotas:manage`        | `compute-fleet`                     | v0.2  |

### Usage and events [#usage-and-events]

| Method                                                               | Action                    | Decided on                            | Phase               |
| -------------------------------------------------------------------- | ------------------------- | ------------------------------------- | ------------------- |
| [`usage.list`](/docs/reference/api/usage#list)                       | `compute:usage:read`      | `compute-tenant` or `compute-project` | v0.1                |
| [`usage.export`](/docs/reference/api/usage#export)                   | `compute:usage:read`      | `compute-tenant` or `compute-project` | v0.2                |
| [`events.list`](/docs/reference/api/events#list)                     | `compute:events:read`     | `compute-tenant`                      | v0.1                |
| [`webhooks.create`](/docs/reference/api/webhooks#create)             | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.list`](/docs/reference/api/webhooks#list)                 | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.get`](/docs/reference/api/webhooks#get)                   | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.update`](/docs/reference/api/webhooks#update)             | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.rotateSecret`](/docs/reference/api/webhooks#rotatesecret) | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.delete`](/docs/reference/api/webhooks#delete)             | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.ping`](/docs/reference/api/webhooks#ping)                 | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.deliveries`](/docs/reference/api/webhooks#deliveries)     | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.redeliver`](/docs/reference/api/webhooks#redeliver)       | `compute:webhooks:manage` | `compute-tenant`                      | v0.3                |
| [`webhooks.eventTypes`](/docs/reference/api/webhooks#eventtypes)     | `compute:webhooks:manage` | `compute-tenant`                      | served, no §9.2 row |

### Provisioning and bare metal [#provisioning-and-bare-metal]

| Method                                                       | Action                     | Decided on                                     | Phase |
| ------------------------------------------------------------ | -------------------------- | ---------------------------------------------- | ----- |
| [`profiles.create`](/docs/reference/api/profiles#create)     | `compute:profiles:manage`  | `compute-tenant`                               | v0.2  |
| [`profiles.get`](/docs/reference/api/profiles#get)           | `compute:profiles:read`    | `compute-profile`                              | v0.2  |
| [`profiles.validate`](/docs/reference/api/profiles#validate) | `compute:profiles:read`    | `compute-profile`                              | v0.2  |
| [`profiles.render`](/docs/reference/api/profiles#render)     | `compute:profiles:read`    | `compute-profile`                              | v0.2  |
| [`profiles.list`](/docs/reference/api/profiles#list)         | `compute:profiles:read`    | `compute-tenant` + items                       | v0.2  |
| [`profiles.update`](/docs/reference/api/profiles#update)     | `compute:profiles:manage`  | `compute-profile`                              | v0.2  |
| [`profiles.delete`](/docs/reference/api/profiles#delete)     | `compute:profiles:manage`  | `compute-profile`                              | v0.2  |
| [`images.register`](/docs/reference/api/images#register)     | `compute:images:publish`   | `compute-tenant` or `compute-fleet`            | v0.2  |
| [`images.get`](/docs/reference/api/images#get)               | `compute:images:read`      | `compute-image`                                | v0.2  |
| [`images.list`](/docs/reference/api/images#list)             | `compute:images:read`      | `compute-tenant` + items                       | v0.2  |
| [`images.promote`](/docs/reference/api/images#promote)       | `compute:images:publish`   | `compute-image`                                | v0.2  |
| [`workflows.get`](/docs/reference/api/workflows#get)         | `compute:workflows:read`   | `compute-node`                                 | v0.2  |
| [`workflows.list`](/docs/reference/api/workflows#list)       | `compute:workflows:read`   | `compute-node`                                 | v0.2  |
| [`workflows.abort`](/docs/reference/api/workflows#abort)     | `compute:workflows:manage` | `compute-node`                                 | v0.2  |
| [`workflows.retry`](/docs/reference/api/workflows#retry)     | `compute:workflows:manage` | `compute-node`                                 | v0.2  |
| [`workflows.pause`](/docs/reference/api/workflows#pause)     | `compute:workflows:manage` | `compute-node`                                 | v0.2  |
| [`workflows.resume`](/docs/reference/api/workflows#resume)   | `compute:workflows:manage` | `compute-node`                                 | v0.2  |
| [`metal.enroll`](/docs/reference/api/metal#enroll)           | `compute:nodes:provision`  | `compute-pool`                                 | v0.2  |
| [`metal.manage`](/docs/reference/api/metal#manage)           | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.inspect`](/docs/reference/api/metal#inspect)         | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.provide`](/docs/reference/api/metal#provide)         | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.deploy`](/docs/reference/api/metal#deploy)           | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.undeploy`](/docs/reference/api/metal#undeploy)       | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.rescue`](/docs/reference/api/metal#rescue)           | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.unrescue`](/docs/reference/api/metal#unrescue)       | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.abort`](/docs/reference/api/metal#abort)             | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.recover`](/docs/reference/api/metal#recover)         | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.retire`](/docs/reference/api/metal#retire)           | `compute:nodes:provision`  | `compute-node`                                 | v0.2  |
| [`metal.power`](/docs/reference/api/metal#power)             | `compute:nodes:power`      | `compute-node`                                 | v0.2  |
| [`metal.bulk`](/docs/reference/api/metal#bulk)               | `compute:nodes:provision`  | `compute-pool`, then each node                 | v1.0  |
| [`metal.discovered`](/docs/reference/api/metal#discovered)   | `compute:sites:read`       | `compute-site` (the site's owning tenant only) | v1.0  |
| [`metal.confirmMac`](/docs/reference/api/metal#confirmmac)   | `compute:nodes:provision`  | `compute-node`                                 | v1.0  |

### VPS [#vps]

| Method                                                                                | Action                                                                                                                                                                                                                                                                       | Decided on                                                                                                                                                            | Phase |
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| [`instances.create`](/docs/reference/api/instances#create)                            | `compute:instances:create`, and `compute:secrets:manage` when `userData` is a `secret://` reference                                                                                                                                                                          | `compute-instance` (the draft); the secrets decision on `compute-tenant`, plus `compute-fleet` in the root tenant                                                     | v0.3  |
| [`instances.list`](/docs/reference/api/instances#list)                                | `compute:instances:read`                                                                                                                                                                                                                                                     | `compute-tenant` + items                                                                                                                                              | v0.3  |
| [`instances.get`](/docs/reference/api/instances#get)                                  | `compute:instances:read`                                                                                                                                                                                                                                                     | `compute-instance`                                                                                                                                                    | v0.3  |
| [`instances.start`](/docs/reference/api/instances#start)                              | `compute:instances:update`                                                                                                                                                                                                                                                   | `compute-instance`                                                                                                                                                    | v0.3  |
| [`instances.stop`](/docs/reference/api/instances#stop)                                | `compute:instances:update`                                                                                                                                                                                                                                                   | `compute-instance`                                                                                                                                                    | v0.3  |
| [`instances.resize`](/docs/reference/api/instances#resize)                            | `compute:instances:update`                                                                                                                                                                                                                                                   | `compute-instance`                                                                                                                                                    | v0.3  |
| [`instances.delete`](/docs/reference/api/instances#delete)                            | `compute:instances:delete`                                                                                                                                                                                                                                                   | `compute-instance`                                                                                                                                                    | v0.3  |
| [`flavors.list`](/docs/reference/api/flavors#list)                                    | `compute:flavors:read`                                                                                                                                                                                                                                                       | `compute-pool`                                                                                                                                                        | v0.3  |
| [`flavors.create`](/docs/reference/api/flavors#create)                                | `compute:flavors:manage`                                                                                                                                                                                                                                                     | `compute-pool`                                                                                                                                                        | v0.3  |
| [`flavors.update`](/docs/reference/api/flavors#update)                                | `compute:flavors:manage`                                                                                                                                                                                                                                                     | `compute-pool`                                                                                                                                                        | v0.3  |
| [`flavors.delete`](/docs/reference/api/flavors#delete)                                | `compute:flavors:manage`                                                                                                                                                                                                                                                     | `compute-pool`                                                                                                                                                        | v0.3  |
| [`providerAccounts.create`](/docs/reference/api/provider-accounts#create)             | `compute:providers:manage`                                                                                                                                                                                                                                                   | `compute-fleet` or `compute-tenant`                                                                                                                                   | v0.3  |
| [`providerAccounts.list`](/docs/reference/api/provider-accounts#list)                 | `compute:providers:read`                                                                                                                                                                                                                                                     | `compute-tenant` + items                                                                                                                                              | v0.3  |
| [`providerAccounts.get`](/docs/reference/api/provider-accounts#get)                   | `compute:providers:read`                                                                                                                                                                                                                                                     | `compute-provider-account`                                                                                                                                            | v0.3  |
| [`providerAccounts.offers`](/docs/reference/api/provider-accounts#offers)             | `compute:providers:read`                                                                                                                                                                                                                                                     | `compute-provider-account`                                                                                                                                            | v0.3  |
| [`providerAccounts.update`](/docs/reference/api/provider-accounts#update)             | `compute:providers:manage`                                                                                                                                                                                                                                                   | `compute-provider-account`                                                                                                                                            | v0.3  |
| [`providerAccounts.delete`](/docs/reference/api/provider-accounts#delete)             | `compute:providers:manage`                                                                                                                                                                                                                                                   | `compute-provider-account`                                                                                                                                            | v0.3  |
| [`sandboxes.create`](/docs/reference/api/sandboxes#create)                            | `compute:sandboxes:create`                                                                                                                                                                                                                                                   | `compute-sandbox` (the draft)                                                                                                                                         | v0.5  |
| [`sandboxes.list`](/docs/reference/api/sandboxes#list)                                | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-tenant` + items                                                                                                                                              | v0.5  |
| [`sandboxes.get`](/docs/reference/api/sandboxes#get)                                  | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-sandbox`                                                                                                                                                     | v0.5  |
| [`sandboxes.extend`](/docs/reference/api/sandboxes#extend)                            | `compute:sandboxes:update`                                                                                                                                                                                                                                                   | `compute-sandbox`                                                                                                                                                     | v0.5  |
| [`sandboxes.snapshot`](/docs/reference/api/sandboxes#snapshot)                        | `compute:sandboxes:update`                                                                                                                                                                                                                                                   | `compute-sandbox`                                                                                                                                                     | v0.5  |
| [`sandboxes.exec`](/docs/reference/api/sandboxes#exec) (hosted by the tunnels module) | `compute:sandboxes:exec`                                                                                                                                                                                                                                                     | `compute-sandbox`                                                                                                                                                     | v0.5  |
| [`sandboxes.delete`](/docs/reference/api/sandboxes#delete)                            | `compute:sandboxes:delete`                                                                                                                                                                                                                                                   | `compute-sandbox`                                                                                                                                                     | v0.5  |
| [`sandboxTemplates.create`](/docs/reference/api/sandbox-templates#create)             | `compute:sandboxes:manage`, and `compute:secrets:manage` when it names secrets; for each volume it names, `compute:sandboxes:manage` (read-write mount) or `compute:sandboxes:read` (read-only)                                                                              | `compute-tenant` (the secrets decision also on `compute-fleet` in the root tenant; the volume decisions on each `compute-sandbox-volume`, 77bf9f5)                    | v0.5  |
| [`sandboxTemplates.list`](/docs/reference/api/sandbox-templates#list)                 | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-tenant` + items                                                                                                                                              | v0.5  |
| [`sandboxTemplates.get`](/docs/reference/api/sandbox-templates#get)                   | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-sandbox-template`                                                                                                                                            | v0.5  |
| [`sandboxTemplates.update`](/docs/reference/api/sandbox-templates#update)             | `compute:sandboxes:manage`, and `compute:secrets:manage` when the resulting template names secrets and the update changes them or what the guest runs; the per-volume decisions for a new `volumes` list, or for the current one when the update changes what the guest runs | `compute-sandbox-template` (the secrets decision on `compute-tenant`, plus `compute-fleet` in the root tenant; the volume decisions on each `compute-sandbox-volume`) | v0.5  |
| [`sandboxTemplates.delete`](/docs/reference/api/sandbox-templates#delete)             | `compute:sandboxes:manage`                                                                                                                                                                                                                                                   | `compute-sandbox-template`                                                                                                                                            | v0.5  |
| [`sandboxSnapshots.list`](/docs/reference/api/sandbox-snapshots#list)                 | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-tenant` + items                                                                                                                                              | v0.5  |
| [`sandboxSnapshots.get`](/docs/reference/api/sandbox-snapshots#get)                   | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-sandbox-snapshot`                                                                                                                                            | v0.5  |
| [`sandboxSnapshots.delete`](/docs/reference/api/sandbox-snapshots#delete)             | `compute:sandboxes:delete`                                                                                                                                                                                                                                                   | `compute-sandbox-snapshot`                                                                                                                                            | v0.5  |
| [`sandboxVolumes.create`](/docs/reference/api/sandbox-volumes#create)                 | `compute:sandboxes:manage`, with the requested `sizeGiB`                                                                                                                                                                                                                     | `compute-sandbox-volume` (the volume it would create, under its pool; 6cb30e9, was `compute-tenant`)                                                                  | v1.0  |
| [`sandboxVolumes.list`](/docs/reference/api/sandbox-volumes#list)                     | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-tenant` (`scope` `mine` by default, or `all`)                                                                                                                | v1.0  |
| [`sandboxVolumes.get`](/docs/reference/api/sandbox-volumes#get)                       | `compute:sandboxes:read`                                                                                                                                                                                                                                                     | `compute-sandbox-volume`                                                                                                                                              | v1.0  |
| [`sandboxVolumes.resize`](/docs/reference/api/sandbox-volumes#resize)                 | `compute:sandboxes:manage`, with the new `sizeGiB` pending                                                                                                                                                                                                                   | `compute-sandbox-volume`                                                                                                                                              | v1.0  |
| [`sandboxVolumes.delete`](/docs/reference/api/sandbox-volumes#delete)                 | `compute:sandboxes:manage`                                                                                                                                                                                                                                                   | `compute-sandbox-volume`                                                                                                                                              | v1.0  |

### Fleet and rollouts [#fleet-and-rollouts]

| Method                                                       | Action                | Decided on      | Phase |
| ------------------------------------------------------------ | --------------------- | --------------- | ----- |
| [`fleet.export`](/docs/reference/api/fleet#export)           | `compute:fleet:admin` | `compute-fleet` | v0.2  |
| [`fleet.plan`](/docs/reference/api/fleet#plan)               | `compute:fleet:admin` | `compute-fleet` | v0.2  |
| [`fleet.apply`](/docs/reference/api/fleet#apply)             | `compute:fleet:admin` | `compute-fleet` | v0.2  |
| [`fleet.status`](/docs/reference/api/fleet#status)           | `compute:fleet:admin` | `compute-fleet` | v0.2  |
| [`releases.register`](/docs/reference/api/releases#register) | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`releases.list`](/docs/reference/api/releases#list)         | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.create`](/docs/reference/api/rollouts#create)     | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.list`](/docs/reference/api/rollouts#list)         | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.get`](/docs/reference/api/rollouts#get)           | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.pause`](/docs/reference/api/rollouts#pause)       | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.resume`](/docs/reference/api/rollouts#resume)     | `compute:fleet:admin` | `compute-fleet` | v0.4  |
| [`rollouts.abort`](/docs/reference/api/rollouts#abort)       | `compute:fleet:admin` | `compute-fleet` | v0.4  |

## How a call is decided [#how-a-call-is-decided]

The envelope in `@better-compute/server` runs the same steps for every method, over HTTP
(`POST /api/compute/<group>/<method>`), the typed client and `compute.api.<group>.<method>(credential, input)`:

<Mermaid
  chart="flowchart TD
  Call[&#x22;Call&#x22;] --> Public{&#x22;Public method?&#x22;}
  Public -- yes --> Run[&#x22;Validate and run&#x22;]
  Public -- no --> Cred{&#x22;Credential presented?&#x22;}
  Cred -- no --> Unauth[&#x22;UNAUTHENTICATED, no IAM call&#x22;]
  Cred -- yes --> Validate[&#x22;Validate input&#x22;]
  Validate --> Authn[&#x22;Authenticate (cached 30 s)&#x22;]
  Authn --> Pre{&#x22;Caller in the tenant, or root?&#x22;}
  Pre -- no --> Denied[&#x22;ACCESS_DENIED, no database access&#x22;]
  Pre -- yes --> Kind{&#x22;Admission?&#x22;}
  Kind -- yes --> Prepare[&#x22;Transaction 1: prepare the draft&#x22;]
  Prepare --> Require[&#x22;One require&#x22;]
  Kind -- no --> Decide[&#x22;One require (mutation) or authorize (read)&#x22;]
  Require --> Handler[&#x22;Handler in a transaction, audit row&#x22;]
  Decide --> Handler"
/>

1. Public methods (`system.whoami`, `system.version`) validate their input and run.
2. A call without a credential is `UNAUTHENTICATED`, with no IAM call.
3. The input is validated (`INVALID_INPUT`) before any IAM call.
4. `IamPort.authenticate` identifies the caller. The answer is cached for 30 seconds.
5. The tenant pre-check below runs.
6. Methods with an admission (request-as-resource creates, `reservations.extend`, and creates that take an
   idempotency key) run `prepare` in a first transaction: the idempotency lookup and the draft and projection rows,
   never an IAM call. A replay by the same caller returns the stored result without a decision. Otherwise the one
   decision is `require` on the resource `prepare` returned.
7. Every other mutation makes one `require`, never cached. A read makes one `authorize`, whose allows are cached
   for 10 seconds per session.
8. The handler runs in one transaction, read-only for reads. An allowed mutation also writes an `audit_events` row
   with its admission context in that transaction.

IAM is never called inside a compute transaction. Denials are opaque: Better IAM reports every denial as
`ACCESS_DENIED`, including a host without the plugin answering `UNKNOWN_ACTION`, and compute adds nothing to it.

## Cross-tenant refusal [#cross-tenant-refusal]

A caller whose tenant is neither the input's `tenantId` nor the root tenant gets `ACCESS_DENIED` before compute
reads or writes its database and before any IAM decision. For fleet-wide methods, the decision tenant is the
root tenant, so only root-tenant callers pass.

The check was added because Better IAM 0.1 resolved the resource before it checked the tenant: without it, an
outsider probing ids got `NOT_FOUND` for ids that do not exist and `ACCESS_DENIED` for ids that do, an existence
oracle, and every probe appended a denial to the victim tenant's audit chain. Better IAM 0.2, which Better Compute
now requires, checks the tenant first itself: a foreign tenant is denied (`TENANT_MISMATCH`, masked as
`ACCESS_DENIED`) before any resolver runs, absent and present ids get the same answer, and the denial is audited on
the caller's own tenant with the target in `targetTenantId` (pinned by `tests/iam-contract.test.ts`). Compute keeps
its own check anyway: the answer is the same in every case, and an outsider costs no database access and no IAM
decision.
`tests/server-admission.test.ts` asserts that such a call makes one IAM call (`authenticate`) and no database
access at all.

Root-tenant callers continue to IAM, which allows them into another tenant only through `ROOT_OVERRIDE`: a
root-tenant person with root authority, in a user session with MFA. Better IAM audits every override. Inside your
own tenant, an id that is not in your view resolves to `NOT_FOUND`, which reveals nothing beyond the tenant.

## List methods and item filtering [#list-methods-and-item-filtering]

The methods marked **+ items** return only the rows the caller may read, without one IAM round trip per row:

1. **Coarse gate.** The envelope's one decision on the scope, `compute-tenant/{t}` or `compute-project/{id}`.
2. **SQL pre-filter.** The query is scoped to the tenant and to rows the projection holds in that tenant's view.
3. **Owner-scoped kinds.** `reservations.list` and `instances.list` take `scope: 'mine' | 'all'`, and `mine` is
   the default: the coarse decision plus `owner_identity_id = caller` in SQL, with no per-item checks.
4. **Everything else**, including `scope: 'all'`, `pools.list`, `nodes.list`, `sites.list`, `projects.list`,
   `profiles.list`, `images.list` and `providerAccounts.list`: after the page is read, every item is decided with
   the method's action on its own resource. Allows already in the read-decision cache are reused; the rest go in
   `authorizeMany` batches of at most 50.

```ts
// The caller's own reservations: one decision on compute-tenant, no per-item checks.
const mine = await compute.api.reservations.list(credential, { tenantId });

// Every reservation the caller may read in the tenant: per-item checks in batches of 50.
const all = await compute.api.reservations.list(credential, { tenantId, scope: 'all' });
```

A batch never degrades into a partial list. When the resolver does not know an item yet (`NOT_FOUND` aborts the
whole batch), compute waits 100 ms for the projection to catch up, reloads the page and retries once. Any other
batch error fails the request, and an answer whose length does not match the batch is `INTERNAL_ERROR`.

Every denied item costs an audit write: Better IAM appends a denial to the tenant's audit chain, under its global
lock. Write policies so that list pages mostly allow, and prefer `scope: 'mine'` where it answers the question.

## Caches [#caches]

| Cache           | Key                                                                | Lifetime | Holds        |
| --------------- | ------------------------------------------------------------------ | -------- | ------------ |
| Authentications | sha256 of the token, the `Authorization` header and the IAM cookie | 30 s     | The `Caller` |
| Read decisions  | Session, tenant, action, type and id                               | 10 s     | Allows only  |

Both caches hold at most 10 000 entries, least recently used first out. Mutations are never cached. The server
exposes invalidation as `compute.caches`: `dropIdentity(identityId)`, `dropSession(sessionId)`,
`dropDecisions()`, `clear()` and `stats()`. Every drop also discards answers that were in flight when it
happened, so a revocation cannot be overwritten by an older allow. Wire these to IAM events as shown in
[IAM integration](/docs/guides/iam).


# Billing and usage (/docs/guides/iam/billing-and-usage)

> How the root tenant bills its sub-tenants for compute. Better Compute meters every resource it issues (hosted and node capacity, provider VMs, reservation grants, networks, disks, snapshots and provider pass-through costs) into hourly usage windows; Better IAM prices them, applies budgets and closes the invoices.



Better IAM is compute's billing ledger. Compute **meters**: it measures what each tenant held and for how long and
records it on Better IAM meters. Better IAM **rates and invoices**: it prices the usage, applies negotiated prices,
terms, credits and budgets, and closes each month into a statement. Compute never prices anything itself; the one
amount it computes is the reported cost of a provider pass-through charge.

The root (platform) tenant owns the meters and the prices, and each organization directly under it is a billing
account. That lets root bill its sub-tenants for the compute it issues to them: hosted and node capacity, cloud VMs
it creates or attaches for them, reservation capacity it commits to them, their networks, disks, public addresses
and snapshots, and provider costs such as egress at cost times a multiplier.

The basis is **reserved time**: a resource bills from the moment it is held until it is released, whether or not it
was busy. Usage leaves compute as hourly usage windows, each with an idempotency
key, so every retry is safe.

<Mermaid
  chart="flowchart LR
  Alloc[&#x22;Allocations&#x22;] -- &#x22;capacity.usage, vps.usage&#x22; --> Windows[(&#x22;usage_windows&#x22;)]
  Intervals[&#x22;Provider VM intervals&#x22;] -- &#x22;vps.provider-usage&#x22; --> Windows
  Grants[&#x22;Reservation grants&#x22;] -- &#x22;vps.grants&#x22; --> Windows
  Nets[&#x22;Networks, snapshots&#x22;] -- &#x22;vps.ancillary-usage&#x22; --> Windows
  Export[&#x22;Provider billing export&#x22;] -- &#x22;vps.costs&#x22; --> Costs[(&#x22;provider_costs&#x22;)]
  Costs -- &#x22;pass-through rules&#x22; --> Windows
  Windows -- &#x22;iam.usage (1 h)&#x22; --> Ledger[&#x22;Better IAM billing&#x22;]
  Windows -- &#x22;what still fails&#x22; --> Dead[(&#x22;usage_dead_letters&#x22;)]
  Costs -. &#x22;billing.margins, billing.leak&#x22; .-> Root[&#x22;Root&#x22;]"
/>

Every source writes the same windows and one flush, `iam.usage`, records them all, with the same dead letters and
outcomes.

## Set it up [#set-it-up]

1. **Give the IAM port a root billing credential.** The `billing` methods, the meter sync and the tenant
   estimates call Better IAM's billing API on the root tenant as a root service account. Embedded:
   `bridge.embedded(iam, { rootTenantId, serviceCredential })`, with an account that holds `iam:billing:manage` and
   `iam:billing:read` on the root tenant. Remote: `bridge.remote(...)` with its `serviceToken`. Without one the
   `billing` methods answer `UNSUPPORTED` and name this fix, estimates are absent, and the iam module's doctor warns
   `iam.billing-port`.
2. **Pick the deployment currency.** Better IAM bills in one currency per deployment. Name it once, in
   `vps({ billing: { currency } })` (ISO 4217, default `USD`): price proposals, tenant estimates, provider costs
   and margins all use it.
3. **Create the meters** (`billing.syncMeters`, or `better-compute iam setup --meters`), then **set prices**
   (`billing.setPrices`). See [Meters](#meters) and [Prices](#prices).
4. **Run the workers.** `iam.usage` (the iam module) flushes hourly; the vps module runs `vps.provider-usage`,
   `vps.grants`, `vps.ancillary-usage` (every 5 minutes), `vps.costs` (every 6 hours) and `vps.suspension` (every
   minute).

```ts title="compute.ts"
import { bridge } from 'better-compute/iam';
import { vps } from 'better-compute/vps';
import { gcp } from '@better-compute/provider-gcp';

const iamPort = bridge.embedded(iam, {
  rootTenantId,
  serviceCredential: { token: rootBillingKey }, // iam:billing:manage + iam:billing:read on the root tenant
  runIamJobs: { closeGraceHours: 6 }, // the default; the same value as vps's below
});

vps({
  providers: [gcp],
  billing: { currency: 'USD', gpuModels: ['H100', 'A100', 'L4'], closeGraceHours: 6 },
  costs: { every: '6h', lookbackDays: 3 },
});
```

`iam.jobs.close-period` closes each month only once the next one is `closeGraceHours` old (see
[Period close](#period-close)). Set the IAM jobs' and vps's `closeGraceHours` to the same value.

## Meters [#meters]

Pricing in Better IAM is per meter and cannot vary by tag, so a distinct price needs a distinct meter. Compute
defines one catalog, `meterCatalog`, on the root tenant. Every meter is a `rate-card` meter counted per hour:

| Meter                                                                                     | What accrues it                                                                                                                                  |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `gpu.<model>.hours`, lower case, such as `gpu.h100.hours`                                 | GPUs booked by a reservation, and the GPUs of a hosted instance's flavor. Fractional atoms count as fractions of a GPU.                          |
| `gpu.<model>.spot.hours`                                                                  | GPUs booked by a spot reservation (v0.5)                                                                                                         |
| `node.metal.hours`                                                                        | A whole-node lease (`binding: 'node'`, v0.5): one per lease hour, on top of its GPUs                                                             |
| `vps.vcpu.hours`, `vps.memory.gib.hours`                                                  | A hosted instance's vCPUs, and its memory in GiB                                                                                                 |
| `vps.volume.gib.hours`                                                                    | A hosted volume's GiB                                                                                                                            |
| `vps.sandbox.warm`                                                                        | A warm [sandbox](/docs/guides/vps/sandboxes#warm-pools) instance's vCPUs until a sandbox claims it, billed to the tenant with no identity (v0.5) |
| `vm.vcpu.hours`, `vm.memory.gib.hours`, `vm.gpu.<model>[.spot].hours`                     | A running provider VM's vCPUs, memory and GPUs                                                                                                   |
| `vm.instance.hours`                                                                       | One per running provider VM, for a per-VM fee                                                                                                    |
| `vm.reserved.vcpu.hours`, `vm.reserved.memory.gib.hours`, `vm.reserved.gpu.<model>.hours` | The same, for a VM admitted against a provider reservation outside any grant                                                                     |
| `vm.grant.slot.hours`                                                                     | Every slot of a reservation grant, every hour of its term, used or not                                                                           |
| `vm.grant.usage.hours`                                                                    | Informational: one per running VM inside a grant (price it at 0)                                                                                 |
| `vm.managed.hours`                                                                        | One per running VM in a `platform-fee` pool: the platform's management fee                                                                       |
| `network.vpc.hours`                                                                       | One per provisioned tenant network                                                                                                               |
| `vps.share.gib.hours`                                                                     | A shared [provider volume](/docs/guides/vps/provider-volumes#shared-nfs-volumes)'s GiB (a network file share, such as DigitalOcean NFS)          |
| `ip.public.hours`                                                                         | One per running provider VM with a public address                                                                                                |
| `disk.<type>.gib.hours`                                                                   | A provider VM's disks in GiB, running or stopped: `pd-standard`, `pd-balanced`, `pd-ssd`, `hyperdisk-balanced`, `local-ssd`                      |
| `snapshot.gib.hours`                                                                      | A sandbox snapshot's size in GiB                                                                                                                 |
| `filesystem.<kind>.gib.hours`                                                             | A shared filesystem's billed size in GiB: `nfs` (EFS, its metered size) and `lustre` (FSx for Lustre, its provisioned capacity)                  |
| `flavor.<name>.hours`                                                                     | One per running VM of a flavor root gave a whole-shape meter (see [Flavor meters](#flavor-meters))                                               |

A fractional atom bills its share of a GPU: a MIG instance its compute slices over 7 (`3g.40gb` is 3/7), a
time-slice slot one over the number of slots, and an MPS slot its units over 100. See
[Fractional GPUs](/docs/guides/capacity/fractional-gpus).

**GPU models.** Every known GPU family (`H100`, `H200`, `B200`, `GB200`, `A100`, `L40S`, `L4`, `RTX-PRO-6000`,
`GH200`, `L40`, `A40`, `A30`, `A10`, `RTX-6000-ADA`, `RTX-A6000`, `RTX-5090`, `RTX-5080` and `RTX-4090`) always gets
`gpu.<family>.hours` and `gpu.<family>.spot.hours`, and `gpu.other.hours` covers a pool without a known model. The
provider-VM and reserved GPU meters exist only for the models you name: `vps({ billing: { gpuModels } })` and the
IAM bridge (`createComputeIamBridge({ meterCatalog: { gpuModels } })`) or `iam setup --gpu-models` (default
`H100`, `H200`, `B200`, `A100`, `L4` and `L40S`, plus `other`, always added). Name the same models in both, or
`billing.meters` reports the difference as missing. A provider's GPU string is named by its catalog family
(`nvidia-h100-80gb` is `h100`); a model the list does not name bills `vm.gpu.other.hours`.

**Reported meters** are the exception: a [pass-through](#pass-through) charge carries its own cost, so its meter is a
`reported` meter you create on the root tenant yourself (Better IAM's `billing.createMeter` with `pricing:
'reported'`, or the IAM port's `ensureMeters`). The catalog holds none.

Better IAM allows 100 meters per tenant, so the catalog refuses a longer GPU model list or too many flavor meters.

### Create the meters [#create-the-meters]

Meters live in the root tenant, and Better IAM resolves them for every tenant below it. Root creates the missing
ones through the API:

```ts title="meters.ts"
const { catalog, missing } = await compute.api.billing.meters(rootCredential, {});
const { created, existing } = await compute.api.billing.syncMeters(rootCredential, {});
```

[`billing.meters`](/docs/reference/api/billing#meters) lists the catalog and the keys the root tenant lacks;
[`billing.syncMeters`](/docs/reference/api/billing#syncmeters) creates them and never changes a meter that exists.
Both are decided as `compute:billing:*` on `compute-fleet`, which resolves only in the root tenant, so a tenant
administrator is refused however broad its own grants.

The CLI does the same with a root credential, so you can run it again after adding a model:

```bash
BETTER_IAM_TOKEN=biam_... better-compute iam setup --audience https://compute.example.com \
  --meters --iam-url https://iam.example.com --root-tenant ten_root --gpu-models H100,A100
```

* `--meters` needs `--iam-url` (or `BETTER_IAM_URL`) and `--root-tenant` (or `BETTER_COMPUTE_ROOT_TENANT`), and
  `BETTER_IAM_TOKEN` must hold a root session token or API key. It calls Better IAM through the optional peer
  `@better-iam/client`. The result lists `meters.created` and `meters.existing`.
* If compute's IAM port holds a root billing credential, the iam module also defines the catalog when it starts
  (best effort: a refusal is logged as a warning).

The same steps are exported for code:

```ts title="setup.ts"
import { meterCatalog, setupIamMeters } from 'better-compute/iam';

const { created, existing } = await setupIamMeters(
  iam.api,
  { credential: rootCredential, tenantId: rootTenantId },
  meterCatalog({ gpuModels: ['H100', 'A100'] }),
);
```

> **meterCatalog, not computeMeters.** 
  `meterCatalog` is the full list and what `billing.syncMeters`, `iam setup --meters` and the iam module create.
  `computeMeters` is the older, smaller list (one `gpu.<model>.hours` per model, `node.metal.hours`,
  `vps.vcpu.hours`, `vps.memory.gib.hours` and `vps.sandbox.warm`); it stays exported, and creates no spot, `other`,
  volume or `vm.*` meter. Better IAM refuses usage on an unknown meter with `NOT_FOUND`, so windows for a meter you
  never created are dead-lettered (`meter-missing`). Meters that already exist are never changed: they keep their
  unit and the prices you set, and only missing ones are created.

## Prices [#prices]

Prices are Better IAM `PriceSpec`s (`per-unit`, `graduated`, `volume` or `package`) in micros of the deployment
currency, stored on the root tenant's rate card. Root manages them through compute:

```ts title="prices.ts"
const { applied, failed } = await compute.api.billing.setPrices(rootCredential, {
  prices: [
    // the list price
    {
      meter: 'vm.gpu.h100.hours',
      effectiveFrom: '2026-11',
      spec: { model: 'per-unit', unitAmountMicros: 2_900_000 },
    },
    // a negotiated price for one customer (and its subtree)
    {
      meter: 'vm.gpu.h100.hours',
      targetTenantId: 'ten_acme',
      effectiveFrom: '2026-11',
      spec: { model: 'per-unit', unitAmountMicros: 2_500_000 },
    },
    {
      meter: 'vm.grant.usage.hours',
      effectiveFrom: '2026-11',
      spec: { model: 'per-unit', unitAmountMicros: 0 },
    },
  ],
});
```

* [`billing.setPrices`](/docs/reference/api/billing#setprices) takes up to 100 prices. Each meter must be in the
  catalog or be a flavor meter, else the whole request is `INVALID_INPUT`. A price Better IAM refuses lands in
  `failed` and does not stop the others.
* [`billing.prices`](/docs/reference/api/billing#prices) reads the rate card, newest first; the list price is the
  entry whose `targetTenantId` is the root tenant.
* Price by resource (the `vm.vcpu`, `vm.memory` and `vm.gpu` meters plus `vm.instance.hours`) or by whole machine
  shape ([flavor meters](#flavor-meters)), per region if you need to.
* Discounts, minimum commitments, tax, credits and invoice items stay in Better IAM (`billing.setTerms`,
  credits, `billing.createInvoiceItem`, credit notes). They are root-only there and need a root administrator's
  user session with MFA, so compute never calls them.

### Price proposals [#price-proposals]

[`billing.proposePrices`](/docs/reference/api/billing#proposeprices) reads a platform provider account's list
prices (for Compute Engine, the Cloud Billing Catalog), adds `marginPct` and returns `per-unit` specs root can
publish. It sets nothing: root reviews the proposals and passes the ones it wants to `setPrices`.

```ts title="propose.ts"
const { proposals, unpriced, currency } = await compute.api.billing.proposePrices(rootCredential, {
  accountId,
  marginPct: 25,
  region: 'us-central1',
  effectiveFrom: '2026-11',
});
await compute.api.billing.setPrices(rootCredential, {
  prices: proposals.map(({ meter, spec }) => ({ meter, effectiveFrom: '2026-11', spec })),
});
```

`vm.vcpu.hours` and `vm.memory.gib.hours` take the region's cheapest general-purpose machine family; GPU, disk and
address meters take their SKU; flavor meters are composed from their family's SKUs. Instance, reserved, grant,
managed, network and snapshot meters are always `unpriced`: they have no provider list price. Egress is not
proposed at all: it is billed at its real cost by [pass-through](#pass-through).

## Provider VMs [#provider-vms]

A cloud VM in a provider pool, created or attached, is metered from **lifecycle intervals**: `running` and
`stopped` spans of the instance, written in the same transaction as the state change that compute observed (the
reconciler's poll, at most a minute behind while steady, or the API action that made it). The database refuses two
open intervals for one instance. Every 5 minutes, `vps.provider-usage` turns the ended hours of every interval into
`instance` windows, once each hour is `vps({ billing: { meterGraceMs } })` old (default 5 minutes, from 0 to an hour,
else `INVALID_CONFIG`), so a change stamped inside the hour has committed.

* **Edges.** Billing starts at the first observed `running` and ends at the observed stop, delete or detach. A
  requested stop keeps billing until the provider reports the VM stopped. A VM that is `lost` stops billing and
  starts a new interval when it is seen again; the lost time is not billed. An `error` closes the interval.
* **Shape.** An interval captures what the VM is billed on when it opens: the flavor or attached shape, its GPUs by
  catalog model, whether it is reserved or granted, its flavor meter, its public address and its disks. A resize,
  a grant change, or a public address appearing or disappearing closes the interval and opens a new one.
* **Attribution.** Windows go to the instance's tenant and owner (`identityId`), whoever issued the VM, with tags
  `region`, `pool`, `project`, `provider`, `gpuModel`, `flavor` and `source: instance`. Instance ids are never tags:
  they stay in the window's key (`cmp:instance:<id>:<windowStartMs>:<meter>`) and in `usage.platform`.

### Billing modes [#billing-modes]

A provider pool's `policy.billing.mode` decides what its VMs record:

| Mode           | A running VM records                                                  | A stopped VM records |
| -------------- | --------------------------------------------------------------------- | -------------------- |
| `metered`      | Its compute lines, plus `ip.public.hours` and `disk.<type>.gib.hours` | Its disk lines       |
| `platform-fee` | `vm.managed.hours` only                                               | Nothing              |
| `none`         | Nothing                                                               | Nothing              |

The default is `metered` for root-owned pools and every hosted pool, and `platform-fee` for a provider pool on a
tenant's own cloud account. Only the platform sets `policy.billing` on
[`pools.create`](/docs/reference/api/pools#create) and `pools.update`: the explicit mode outranks the default, so
any other caller gets `ACCESS_DENIED`. The mode is read when an interval opens.

The compute lines of a `metered` VM, per hour:

* **Composed:** `vm.vcpu.hours` × vCPUs, `vm.memory.gib.hours` × GiB, `vm.gpu.<model>.hours` × GPUs (`.spot` for a
  Spot VM), and `vm.instance.hours` × 1.
* **Reserved:** a VM admitted against a provider reservation outside any grant uses `vm.reserved.*` instead (never
  `.spot`); `vm.instance.hours` stays as it is.
* **Granted:** a VM inside a [reservation grant](#reservation-grants) records only `vm.grant.usage.hours` × 1. The
  grant's slot-hours pay for it, so it records no compute meter.
* **Flavor meter:** `flavor.<name>.hours` × 1 replaces the composed lines.

The address and disk lines apply in `metered` mode only, and to granted and flavor-meter VMs too: a grant covers the
VM, not its disks or address.

### Attached VMs and BYO accounts [#attached-vms-and-byo-accounts]

* **Attach.** An attached VM bills its tenant from its first observed `running` after it was attached, never
  before. [`instances.attach`](/docs/reference/api/instances#attach) takes `billing: 'metered' | 'none'`, default
  the pool's mode. `none` opens no interval, so nothing of the VM is metered, running or stopped; only the platform
  may pass it (`ACCESS_DENIED` otherwise). An attach to tenant A, a detach and an attach to tenant B give two
  intervals with the right tenants.
* **BYO.** A tenant that runs VMs on its own cloud account already pays its provider for them, so its pools default
  to `platform-fee`: at most `vm.managed.hours`, never a `vm.*` compute, address or disk meter, and its networks
  bill nothing. Its provider costs are its own: they are attributed to it but never passed through and never count
  toward the platform's margins.

### Flavor meters [#flavor-meters]

Root can price a machine shape as one line: `billing.meter` on a flavor (`flavors.create` or `flavors.update`,
root only), such as `flavor.a3-highgpu-8g.us-central1.hours`. It matches `^flavor.[a-z0-9.-]{1,56}.hours$`, is
created on the root tenant when the IAM port can define meters, and is included in `billing.meters` and
`billing.syncMeters`. An instance copies the meter when it is created, so a later change to the flavor never changes
how an existing instance bills. Because Better IAM prices cannot vary by tag, flavor meters are also how you charge
a different price per region.

## Reservation grants [#reservation-grants]

A [reservation grant](/docs/reference/api/reservation-grants) commits slots of a provider capacity reservation (a
GCE reservation on a root account, bound to a root pool) to one tenant for a term. Root creates and ends grants;
the grantee pays for the committed slots whether it uses them or not, and gets priority on them.

```ts title="grant.ts"
const grant = await compute.api.reservationGrants.create(rootCredential, {
  tenantId: rootTenantId,
  accountId,
  reservationId: 'us-central1-a/h100-pool',
  grantTenantId: 'ten_acme',
  slots: 2,
  startsAt: Date.parse('2026-11-01T00:00:00Z'),
  endsAt: Date.parse('2026-12-01T00:00:00Z'),
  idempotencyKey: 'acme-nov-h100',
});
```

* **Slot-hours.** `vps.grants` bills the grantee `vm.grant.slot.hours` × `slots` for every hour of the term that
  has ended (an hour the term only partly covers bills that fraction), tagged with the reservation's region, the
  pool, the provider and `source: grant`. Billing stops at `endsAt`.
* **No double billing.** The grantee's VMs inside the grant record only `vm.grant.usage.hours`. Beyond the grant, a
  VM takes an ordinary free slot of the reservation and bills `vm.reserved.*`.
* **Admission.** The grantee's creates fill its grant first. Nobody else can take a granted slot, even an idle one
  or one a stopped grant VM holds: others share `total − in use − unused granted slots`. Grants over one term never
  exceed the reservation's `total`; the database decides concurrent grants under the reservation's row lock
  (`CAPACITY_UNAVAILABLE` with `{ total, granted }`).
* **Adoption and renewals.** A grant that starts while its tenant already runs VMs on the reservation outside any
  grant takes them in, oldest first, up to its slots, and their intervals reopen as granted. When a grant ends,
  another live grant of the tenant on the same reservation (a renewal) takes its VMs; otherwise they bill
  `vm.reserved.*` from then on. VMs attached with `billing: 'none'` are never taken in.
* **Ending.** `reservationGrants.end` sets `endsAt` to now: the hours already in the term stay billed. A grant that
  has not started is `cancelled`. Deleting a tenant ends its active grants and cancels its scheduled ones.
* **A reservation that goes or shrinks.** When the provider stops reporting the reservation, its active grants end
  then and its scheduled ones are cancelled (`reason: 'reservation-gone'` on the events), and its VMs bill on-demand
  from then on. When its `total` drops below the slots its live grants hold, `reservation-grant.capacity-short`
  tells root once per change; the grants keep billing until root ends one.
* **Accounts.** `providerAccounts.unbindReservation` and `providerAccounts.delete` refuse with `INVALID_STATE`
  (`details.grantIds`) while a grant on the account is scheduled or active, and `delete` also until `vps.grants`
  has metered every committed slot-hour of the ended ones.
* **Reads.** `reservationGrants.get` and `list` are tenant reads (`compute:quotas:read` on the tenant): a tenant
  sees its own grants, and root lists every tenant's with `all: true`.

Grants are root contracts and are not budget-checked. Each change publishes `reservation-grant.created`, `active`,
`ended` or `cancelled` in the grantee's tenant.

## Networks, disks and snapshots [#networks-disks-and-snapshots]

The `vps.ancillary-usage` reconciler (every 5 minutes) meters what a tenant holds beside its VMs; addresses and
disks are lines of the VM's own intervals.

* **Networks.** `network.vpc.hours` × 1 per tenant network from its first `ready` (`provisioned_at`) until it is
  deleted, billed to the network's tenant. A network being deleted or in `error` still exists at the provider and
  still bills; one never provisioned bills nothing. Firewall rules and subnets are free. Networks in a `none` or
  `platform-fee` pool bill nothing.
* **Public addresses.** `ip.public.hours` × 1 while a `metered` VM runs with a public address.
* **Disks.** `disk.<type>.gib.hours` × GiB per disk of a `metered` VM, while it runs and while it is stopped (the
  provider bills stopped disks too). The disks come from the provider's machine record (Compute Engine reports each
  disk and its type); otherwise one boot disk of the flavor's `diskGiB`, of `hyperdisk-balanced` for the machine
  families that take only Hyperdisk, else `pd-balanced`. A type with no meter of its own is billed as
  `pd-balanced` and the instance shows a `DiskTypeUnknown` condition.
* **Snapshots.** `snapshot.gib.hours` × `sizeBytes / 2^30` per sandbox snapshot from `ready` until it is deleted,
  billed to the snapshot's tenant, unless its pool's mode is `none` or `platform-fee`.
* **Shared filesystems.** `filesystem.<kind>.gib.hours` × the size the provider last reported (EFS's metered
  size, FSx for Lustre's provisioned capacity; else the `sizeGiB` asked for) from the filesystem's first `ready`
  until it is deleted, billed to its tenant unless its network's pool's mode is `none` or `platform-fee`.

## Provider costs [#provider-costs]

The `vps.costs` worker reads what the providers billed the platform, for pass-through charges, margins and leak
detection. It runs for every account whose provider can read costs (`ProviderImpl.costs`): for Compute Engine,
an account with `billingExport` set (see [Google Compute Engine](/docs/guides/vps/providers#google-compute-engine)).

* **Reads.** Every `vps({ costs: { every } })` (default `6h`), each account's last `lookbackDays` (default 3, at
  most 31) are read outside any transaction and stored in `provider_costs` in batches of 500, keyed by the
  provider's stable row id. A run stays within `passBudgetMs` (default 50 s, from 5 s to 55 s) and continues in a
  follow-up run. A rate limit or an outage backs the account off (5 minutes without a retry hint).
* **Attribution,** in order: the row's `better-compute-instance` label (the instance's tenant), its
  `better-compute-tenant` label, then, on a tenant's own (BYO) account, that tenant. Anything else is
  `unattributed` and publishes `billing.unattributed`. A label on a tenant's own account can never move its cost to
  another tenant.
* **Late and corrected rows.** Re-exported rows are idempotent. A corrected row replaces the stored one and
  publishes `billing.cost-corrected` (`passedThrough` says whether it was already billed); a passed-through row is
  never billed again, so root issues a credit note in Better IAM where the correction warrants one.
* **Currency.** A row in another currency than `billing.currency` is stored as received, flagged
  `CurrencyMismatch`, and left out of pass-through and margins.
* **Events.** `billing.cost-ingested` once per pass per account, with totals.

### Pass-through [#pass-through]

Costs compute cannot meter itself, such as egress and provider snapshots, are billed at cost times a multiplier
through `billing.passThrough` rules on a **platform** provider account
([`providerAccounts.create`](/docs/reference/api/provider-accounts#create) and `update`):

```ts title="pass-through.ts"
await compute.api.providerAccounts.update(rootCredential, {
  tenantId: rootTenantId,
  accountId,
  config,
  billing: {
    passThrough: [
      { skuPattern: 'egress', meter: 'network.egress.gib', multiplier: 1.2 },
      { skuPattern: 'Storage PD Snapshot', meter: 'storage.snapshot.gib', multiplier: 1.1 },
    ],
  },
});
```

* **Rules.** At most 50 rules of `{ skuPattern, meter, multiplier }`. `skuPattern` is a case-insensitive substring
  of the SKU id or description, and the first matching rule wins. `meter` must be a `reported` meter on the root
  tenant (`INVALID_INPUT` otherwise). `multiplier` is 1 to 10. Rules are allowed on root accounts only.
* **What passes.** A regular-usage row attributed to a tenant other than the platform, in the deployment currency,
  with a positive net cost (cost plus credits), becomes one usage event on the rule's meter: `quantity` the row's
  usage, `cost` = net cost × `multiplier`, tags `region`, `provider`, `pool` and `source: cost`, key
  `costx:<accountId>:<rowId>`. It is written as a `cost` usage window and recorded by the normal flush, once.
* **Never billed twice.** A pattern that matches a SKU family compute meters itself is `INVALID_CONFIG`
  (`details.reason: 'meteredSku'`), and a row of such a SKU never passes, whatever the rules. For Compute Engine
  that is core, RAM, GPU and instance SKUs, persistent disk and Hyperdisk capacity, local SSD and external IP
  charges. A row of an instance billed `none`, or in a `none` or `platform-fee` pool, never passes either.
* **Decided at ingest.** Whether a row may pass is stored with it when it is read (`provider_costs.passable`), and
  each pass reads only those rows not passed yet, in batches, stopping between batches once the run is out of
  time. Changing an account's rules decides it again for its rows within the cost lookback.

### Margins and leak detection [#margins-and-leak-detection]

[`billing.margins`](/docs/reference/api/billing#margins) (root only) returns revenue minus provider cost over a range
of at most 400 days, grouped by `tenant`, `pool`, `account` or `meter`:

```ts title="margins.ts"
const margins = await compute.api.billing.margins(rootCredential, {
  from: Date.parse('2026-10-01T00:00:00Z'),
  to: Date.parse('2026-11-01T00:00:00Z'),
  groupBy: 'tenant',
});
// { rows, unattributedCostMicros, currency, from, to }
// rows: [{ key: 'ten_acme', revenueMicros, costMicros, marginMicros, marginPct }]
```

* **Revenue** is what Better IAM billed (`IamPort.spend`, by whole billing periods); without it, revenue and
  margins are `null`. **Cost** is cost plus credits of the platform's own accounts only: a tenant's own account is
  its cost, not the platform's.
* Revenue is rated usage **before contract terms**: list or negotiated prices, before the `setTerms` discount
  Better IAM applies when it closes the period. A discounted tenant's margin is overstated by its discount.
* The root tenant's own usage is never revenue (Better IAM never invoices the platform). By tenant, root's row keeps
  its cost with `revenueMicros: 0`. By pool or meter, spend cannot be scoped to root alone, so those groups include
  root's internal usage at its rated price.
* Provider compute cost has no meter dimension, so with `groupBy: 'meter'` it shows under `(none)`. Read the meter
  view for pass-through margins and the tenant or pool view for compute.

At the end of every `vps.costs` pass over a platform account, each instance's provider-billed compute is compared
with its metered running time per UTC day. The provider hours are the **distinct hour slots** the provider billed
compute for the instance, not the summed quantities (a provider bills cores and memory as separate rows). When they
exceed the metered hours by more than `vps({ costs: { leakToleranceHours } })` (default 1), `billing.leak` is
published, at most once per instance and day, so a metering bug surfaces before an invoice goes out. Instances
billed `none` and tenants' own accounts are not checked.

## Estimates [#estimates]

Tenants see an hourly price before they create something. On a provider pool, each
[`flavors.list`](/docs/reference/api/flavors#list) item carries `estimate`, and
[`instances.estimate`](/docs/reference/api/instances#estimate) answers the same for one flavor before a create:

```ts title="estimate.ts"
const { estimate } = await compute.api.instances.estimate(credential, {
  tenantId,
  poolId,
  flavor: 'a3-highgpu-8g',
});
// { perHourMicros: 98_320_000, currency: 'USD',
//   lines: [{ meter: 'vm.vcpu.hours', quantity: 208, amountMicros }, …] }
```

* The lines are the meters an instance of the flavor would record per hour in that pool's mode, each priced by
  Better IAM (`IamPort.quote`) with the tenant's effective price, so a negotiated price shows. A line with no price
  has `amountMicros: null`, and then `perHourMicros` is `null` rather than a partial sum.
* `instances.estimate` is a read of the pool's flavors (`compute:flavors:read` on the pool): no draft, no quota, no
  provider call and no audit row.
* The estimate is for an on-demand VM: Spot, grants and the VM's disks and addresses are not in it. Estimates are
  cached five minutes per tenant and pool, and are absent when the IAM port cannot quote.

## Usage windows [#usage-windows]

The `capacity.usage` job (every 5 minutes; `capacity: { usageEvery }` changes it) and the `vps.usage` job meter
allocations, and the vps reconcilers above meter their sources, into `usage_windows`, partitioned by month:

```sql title="usage_windows"
CREATE TABLE usage_windows (
  idempotency_key text NOT NULL,        -- see the key forms below
  allocation_id text REFERENCES allocations,  -- allocation windows only
  source_kind text NOT NULL DEFAULT 'allocation',  -- allocation, instance, grant, network, snapshot, cost
  source_id text,                       -- the instance, grant, network, snapshot or cost row
  tenant_id text NOT NULL,
  identity_id text,                     -- NULL once orphaned (tenant-attributed), and for tenant-billed sources
  meter text NOT NULL,
  quantity numeric NOT NULL CHECK (quantity >= 0),
  cost_micros bigint,                   -- pass-through: the reported cost
  window_start timestamptz NOT NULL,
  window_end timestamptz NOT NULL,
  tags jsonb NOT NULL DEFAULT '{}',     -- region, pool, project, provider, gpuModel, flavor, source, agent
  recorded_at timestamptz,
  outcome text CHECK (outcome IN ('recorded', 'duplicate', 'reattributed', 'dead-lettered')),
  claimed_at timestamptz,               -- a flush is sending it
  voided_at timestamptz, void_reason text,
  CHECK (window_end > window_start),
  PRIMARY KEY (idempotency_key, window_start)
) PARTITION BY RANGE (window_start);
```

* Windows are aligned to the clock hour. The quantity is the meter's rate per hour times the fraction of the
  hour used, rounded to six decimals: 4 GPUs held from 10:15 bill 3 GPU hours for the 10:00 window. A span from
  23:30 on the last day of a month to 00:30 gives two windows, one in each month.
* A window is written only when it is final: its hour has closed, or billing ended inside it. It never changes
  afterwards, and every pass is safe to repeat. Each source keeps a watermark and never derives an hour twice.
* **Keys.** An allocation window is `cmp:<allocationId>:<windowStartMs>` (an allocation that bills a second meter
  in the same window, a node lease's GPUs, appends `:<meter>`), unchanged from earlier versions. Every other source
  is `cmp:<kind>:<sourceId>:<windowStartMs>:<meter>`, and a pass-through window `costx:<accountId>:<rowId>`.
* Tags carry only bounded values: `region`, `pool`, `project`, `provider`, `gpuModel`, `flavor`, `source` and, for
  delegated work, `agent`. Never a reservation, allocation, node, instance, network or grant id.

A node lease bills `node.metal.hours` plus only the GPUs that were healthy at the start of each window. The
others are booked only to keep other tenants off the node, and the reservation shows an `UnhealthyAtoms`
condition for them.

### Attribution [#attribution]

Each allocation and instance window names the identity it bills: the reservation's or instance's owner. Grants,
networks, snapshots and pass-through costs bill the tenant with no identity. For work an AI agent does on a
person's behalf, the identity is the person, and the window carries the tag `agent=<agentId>`. For an agent using
its own key, the owner is the agent itself. In Better IAM's usage reports, agent-key usage therefore appears under
`groupBy: 'agent'`, while delegated usage appears only under `groupBy: 'tag:agent'`, because Better IAM's
`UsageInput` has no `agentId` for delegated work (upstream change U9 would add it).

## The usage sink [#the-usage-sink]

The `iam.usage` reconciler of the `iam` module runs every hour. Each pass claims up to 1000 ended, unrecorded,
unvoided windows of every source (it sets `claimed_at` in a short statement of its own), sends them through
`IamPort.recordUsage` (a pass-through window with its `cost`), and stores each outcome on its window:

| Outcome         | Meaning                                                                                            |
| --------------- | -------------------------------------------------------------------------------------------------- |
| `recorded`      | Better IAM recorded the event                                                                      |
| `duplicate`     | The idempotency key was recorded before; nothing changed                                           |
| `reattributed`  | The identity no longer exists: the event was recorded under its successor, or for the tenant alone |
| `dead-lettered` | The event still failed on its own; it is stored in `usage_dead_letters`                            |

A claim lasts 10 minutes: a pass that died leaves claims the next pass takes over, and a resend is a `duplicate`
because Better IAM knows the keys. A pass whose send fails keeps its claims too (Better IAM may have recorded part
of it before the error arrived), so `usage.void` cannot void a window that may be billed; the windows are sent
again once the claims expire. A pass only ever clears its own claim, never a newer one.

Embedded mode sends batches of 200 events per `billing.recordMany` (`usageBatchSize` on `bridge.embedded()` takes 1
to 1000), which bounds how long one call holds Better IAM's lock.
Remote mode sends at most 100 per HTTP call, as the root service account.

### Poison isolation [#poison-isolation]

Better IAM records a `recordMany` batch in one transaction, so a single bad event would fail its whole batch on
every retry. The sink isolates it:

<Mermaid
  chart="flowchart TD
  Batch[&#x22;recordMany(batch)&#x22;] --> Ok{&#x22;Succeeded?&#x22;}
  Ok -- yes --> Stored[&#x22;recorded or duplicate&#x22;]
  Ok -- &#x22;transient error&#x22; --> Retry[&#x22;Rethrow: claims kept, resent after 10 minutes&#x22;]
  Ok -- &#x22;event-specific error&#x22; --> Size{&#x22;One event left?&#x22;}
  Size -- no --> Split[&#x22;Split in half, record each half&#x22;]
  Size -- yes --> Identity{&#x22;Identity not found?&#x22;}
  Identity -- yes --> Successor[&#x22;Re-record under the successor, or tenant-attributed&#x22;]
  Identity -- no --> Dead[&#x22;dead-lettered&#x22;]"
/>

* Event-specific errors are `NOT_FOUND`, `CONFLICT`, `BILLING_PERIOD_CLOSED`, `INVALID_INPUT`,
  `INVALID_TRANSITION` and `METER_ARCHIVED`. Anything else, such as Better IAM being down or a lock timeout, is
  transient: the whole call rejects, the windows stay unrecorded and claimed, and the first pass after the claim
  expires (10 minutes) sends them again with the same keys. So is Better IAM's `NOT_FOUND` "Endpoint not found": it
  means the route is wrong (a misconfigured base path, a proxy), every event would fail alike, and dead-lettering
  them all would lose the usage.
* An identity deleted or moved before the flush (`NOT_FOUND` "Identity not found") is re-recorded under the
  successor named by Better IAM's `identity:offboard` event, which the IAM inbox stores in `iam_successors` and
  follows for up to 8 hops. Without a successor, the event is recorded with no identity and the tag
  `attribution=orphaned`, and the window's `identity_id` is cleared.
* `CONFLICT` means an idempotency key was reused on another meter, which points to a compute bug.
  `BILLING_PERIOD_CLOSED` means the period was closed before the window arrived. Both are dead-lettered.

## Corrections [#corrections]

Nothing re-sends a dead letter automatically: its window is marked `dead-lettered` and the sink moves on. A closed
period (`BILLING_PERIOD_CLOSED`) is dead-lettered once and never retried by a later pass.

```sql title="usage_dead_letters"
CREATE TABLE usage_dead_letters (
  idempotency_key text PRIMARY KEY,
  tenant_id text NOT NULL,
  event jsonb NOT NULL,          -- the UsageEvent as last sent
  error_code text NOT NULL,
  error_message text NOT NULL,
  attempts int NOT NULL DEFAULT 1 CHECK (attempts > 0),
  first_failed_at timestamptz NOT NULL DEFAULT compute_now(),
  last_failed_at timestamptz NOT NULL DEFAULT compute_now(),
  resolved_at timestamptz
);
```

Root works the letters, and corrects usage before it is invoiced, through the `usage` methods. All are decided on
`compute-fleet/global`, so a tenant administrator is refused:

* [`usage.deadLetters({ reason?, cursor?, limit? })`](/docs/reference/api/usage#deadletters)
  (`compute:usage:read`) lists the open dead letters, each with `idempotencyKey`, `tenantId`, `meter`,
  `quantity`, `windowStart`, `reason` and `error`. The `reason` comes from Better IAM's error:
  `meter-missing` (`NOT_FOUND` naming a meter), `period-closed` (`BILLING_PERIOD_CLOSED`), `identity`
  (`NOT_FOUND` for an identity or tenant), `invalid` (`INVALID_INPUT`, `INVALID_TRANSITION`) or `other`.
* [`usage.retry({ idempotencyKeys | reason })`](/docs/reference/api/usage#retry) (`compute:usage:manage`) clears
  the outcome of those windows, so the next flush sends them again. A pass that records one resolves its letter; a
  pass that fails again counts another attempt. Fix the cause first (`billing.syncMeters` for `meter-missing`).
  `period-closed` letters are refused with `INVALID_STATE`, because retrying would only fail again: the usage needs
  a root invoice item in Better IAM (`billing.createInvoiceItem`). A key whose window retention already pruned is
  `NOT_FOUND`: record that usage in Better IAM by hand.
* [`usage.void({ idempotencyKeys, reason })`](/docs/reference/api/usage#void) (`compute:usage:manage`) sets
  `voided_at` and `void_reason` on windows no pass has sent (outcome `NULL`), so no pass ever sends them, and on
  dead-lettered windows (Better IAM refused them, so nothing was billed). Voiding any window resolves its open dead
  letters. The call is all or nothing.
  * A window Better IAM recorded (`recorded`, `duplicate`, `reattributed`) gives `INVALID_STATE` ("issue a credit
    note in Better IAM"): recorded usage is corrected in Better IAM by credit note, never by rewriting compute's
    history.
  * A window a pass has claimed within the last 10 minutes is being sent: `INVALID_STATE` ("being sent; retry
    shortly").
  * A voided window leaves the backlog counters and the tenants' `usage.list` and `usage.export`, and retention
    prunes it like a recorded one. Tenants see `voided: true` and the time with `includeVoided`, never root's
    reason.

## Period close [#period-close]

Better IAM closes calendar months. Compute makes sure the last hour of a month lands in it:

* Every metering reconciler writes only windows whose hour has ended, `iam.usage` flushes every hour, and provider
  usage waits `meterGraceMs` (5 minutes) after each hour. A window that straddles midnight on the last day is split
  at the hour, so it is in the right month.
* **Close at least 6 hours after the month ends.** Better IAM's `billing.closePeriod()` closes the previous month
  whenever it runs, and a window that arrives after it is refused (`BILLING_PERIOD_CLOSED`). With `runIamJobs`, the
  iam module runs `iam.jobs.close-period` every hour, and it calls `closePeriod()` only once the current UTC month
  is at least `runIamJobs: { closeGraceHours }` old (default 6, from 0 to 72). Closing a closed month again is
  harmless. A host that closes months itself leaves the job out of `runIamJobs.only` and waits as long.
* **Late pass-through rows.** A provider's billing export can land a cost row days after its hour. When the row's
  usage month has already closed (now is at least `vps({ billing: { closeGraceHours } })` hours, default 6, after the
  month ends), its pass-through window is written at the first hour of the current UTC month instead: a late charge
  on the next statement, with the same `costx:` key, while `provider_costs.usage_start` keeps the real hour. Set
  vps's `closeGraceHours` to the IAM jobs' value. A pass-through window that is still refused as `period-closed` has
  its amount as `cost` in `usage.deadLetters`, for a root invoice item.
* Before closing, check the iam module's doctor: `iam.usage-backlog` is an error while windows of a past month are
  still unrecorded. After closing, `usage.deadLetters({ reason: 'period-closed' })` lists what missed the month, for
  a root invoice item in the next one.

## Read usage [#read-usage]

`usage.list` returns a tenant's windows as the tenant can read them, with each window's outcome. Windows of every
source appear, including the instance, grant, network, snapshot and pass-through windows of its cloud VMs:

```ts title="usage.ts"
const page = await compute.api.usage.list(credential, {
  tenantId,
  projectId, // optional: decides compute:usage:read on the project instead of the tenant
  meter: 'vm.gpu.h100.hours',
  from: Date.parse('2026-09-01T00:00:00Z'), // windows that end after this
  to: Date.parse('2026-10-01T00:00:00Z'), // windows that start before this
  limit: 500, // 100 by default
});
// page.items[0]: { idempotencyKey, allocationId?, tenantId, identityId?, meter, quantity,
//                  windowStart, windowEnd, tags, recordedAt?, outcome? }
```

It decides `compute:usage:read` on `compute-tenant/{t}`, or on `compute-project/{id}` when the input names a
project. Prices and invoices come from Better IAM's own billing API.

### Platform view [#platform-view]

`usage.platform` is root's cross-tenant view: the windows of every tenant totalled by any of `tenant`, `pool`,
`meter`, `source` and `provider`, with the windows per outcome. It decides `compute:usage:read` on
`compute-fleet/global`. See [`usage.platform`](/docs/reference/api/usage#platform).

```ts title="platform.ts"
const view = await compute.api.usage.platform(rootCredential, {
  from: Date.parse('2026-09-01T00:00:00Z'),
  to: Date.parse('2026-10-01T00:00:00Z'), // at most 93 days after from
  groupBy: ['tenant', 'meter'],
});
// { format: 'json', rows: [{ keys: { tenant, meter }, quantity, windows, outcomes: { recorded: 3, pending: 1 } }],
//   count, voided, truncated }
```

### Export usage [#export-usage]

[`usage.export`](/docs/reference/api/usage#export) (v0.2) returns every window of a range in one response, for a
billing export, with the same decision as `usage.list`:

```ts title="export.ts"
const exported = await compute.api.usage.export(credential, {
  tenantId,
  projectId, // optional, as for usage.list
  meter: 'gpu.h100.hours', // optional
  from: Date.parse('2026-09-01T00:00:00Z'),
  to: Date.parse('2026-10-01T00:00:00Z'), // after from, at most 93 days later
  format: 'csv', // or 'jsonl'
});
// { format, data, count, totals: { 'gpu.h100.hours': 812 }, voided, truncated }
```

* `csv` (the default) is RFC 4180 CSV with a header row: `idempotencyKey`, `allocationId`, `tenantId`,
  `identityId`, `meter`, `quantity`, `windowStart`, `windowEnd`, `tags`, `recordedAt`, `outcome`, times as ISO
  8601 and `tags` as JSON. `jsonl` is one JSON window per line. A cell a spreadsheet would read as a formula
  (starting with `=`, `+`, `-`, `@`, a tab or a carriage return) is prefixed with `'`, here and in
  `usage.platform`'s CSV.
* `totals` sums `quantity` per meter over the exported windows; voided windows are counted apart in `voided`.
* At most 50,000 windows, ordered by start: `truncated: true` means more matched, so narrow the range or the
  meter.

## Budgets [#budgets]

PLAN.md stops spending in three layers, and all three work today:

1. **In the decision.** The `StopWhenOverBudget` statement of the [gpu-user policy](/docs/guides/iam/policies)
   denies new reservations, extensions and instances once `principal.spendExceeded` is true, meaning an enforced
   Better IAM budget covering the caller is spent. It costs nothing extra: Better IAM evaluates it inside the one
   decision. It also names `compute:sandboxes:create` and `compute:sandboxes:update`, so new
   [sandboxes](/docs/guides/vps/sandboxes) and sandbox extensions stop too.
2. **Per pool, before admission.** Every path that creates something billable runs capacity's
   `budgetAdmissionCheck` as its
   [`MethodAdmission.check`](/docs/extending/methods#admission-the-request-as-a-resource), after the IAM decision
   with no transaction open: [`reservations.create`](/docs/reference/api/reservations#create),
   [`reservations.extend`](/docs/reference/api/reservations#extend),
   [`instances.create`](/docs/reference/api/instances#create),
   [`instances.attach`](/docs/reference/api/instances#attach) (a billed attach: not `billing: 'none'`, not a
   `none` pool), [`networks.create`](/docs/reference/api/networks#create) (on a `metered` pool),
   [`sandboxes.create`](/docs/reference/api/sandboxes#create) (on the template's pool) and
   [`sandboxes.extend`](/docs/reference/api/sandboxes#extend) (on the sandbox's pool). It calls Better IAM's
   `billing.check` through `IamPort.billingCheck({ tenantId, identityId }, credential)` when the pool's policy sets
   `admission.budgetCheck: 'meter'`, or when the tenant's offer has `onBudgetExceeded: 'stop-new'`, `'drain'` or
   `'suspend'`; otherwise it costs nothing. A spent budget refuses with
   [`SPEND_LIMIT_REACHED`](/docs/reference/errors#spend_limit_reached) (402), with the budget's `budgetId` and
   `name` in `details`. A root caller issuing into a tenant is checked against that **tenant's** budgets, not
   root's. A refused create keeps its draft, so the same idempotency key is decided again once the budget's window
   resets; a refused extend clears the pending change. In remote mode the check runs as the caller, because Better
   IAM's HTTP `billing.check` reports the credential's own spend in its own tenant. Reservation grants are root
   contracts and are not budget-checked.
3. **On a budget alert.** Better IAM's `billing:budget-alert` event reaches the IAM inbox's `budget` reaction.
   `budgetAlertHandler` reads the alert (`budgetAlertOf`: budget, `actual` or `forecast`, threshold, spent,
   amount, window) and acts only on an exceeded budget, an `actual` alert at 100% or more. `serve`'s default
   handler then applies capacity's `applyBudgetPolicy` to every offer to the tenant with an `onBudgetExceeded`.

Root sets an offer's policy over the API: [`pools.offer`](/docs/reference/api/pools#offer) takes
`onBudgetExceeded` and `allowedRegions`, and [`pools.updateOffer`](/docs/reference/api/pools#updateoffer) changes
them, or suspends and lifts the offer by hand:

| Policy     | On a spent budget                                                                                                                                         |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `notify`   | Publishes `quota.budget-exceeded`                                                                                                                         |
| `stop-new` | Also refuses new work on the pool through the per-pool check                                                                                              |
| `drain`    | Also cancels the tenant's reservations on the pool that have not started                                                                                  |
| `suspend`  | Stops spend: marks the offer suspended, publishes `quota.budget-suspended`, stops the tenant's running instances in the pool and refuses all its new work |

* **Suspend.** While an offer is suspended, every new piece of work on the pool is `SPEND_LIMIT_REACHED`
  (`details.reason: 'offer-suspended'`), billed or not: creating or attaching an instance, creating a network or a
  reservation, and starting an instance. The `vps.suspension` reconciler stops the tenant's running instances in
  the pool (`desiredState: 'stopped'`, condition `BudgetSuspended`) within a minute, retrying a stop that failed,
  with no host wiring; vps's `sandboxIamReactions` budget reaction only makes it immediate. The offer's sandbox warm
  pools are drained and not refilled. A hand suspension (`updateOffer({ suspended: true })`) does the same.
* **Lifting** (`updateOffer({ suspended: false })`) restarts nothing: stopped instances stay stopped until the
  tenant starts them. A new alert that finds the budget still spent suspends the offer again.
* **Regions.** With `allowedRegions`, new work on a pool whose site is in another region is `ACCESS_DENIED`
  (`details.reason: 'region-not-allowed'`).

Offers inherited through `includeDescendants` are not covered, because compute does not know the tenant tree:
offer the budgeted tenant directly. With `runIamJobs: true`, the `iam` module also runs Better IAM's budget checks
every hour.

  - [Providers](/docs/guides/vps/providers#billing): Billing modes, attach billing, reservation grants and pass-through rules on provider accounts.

  - [Quotas and usage](/docs/guides/capacity/quotas-and-usage): Compute-side quotas, which cap what a tenant may hold regardless of budget.

  - [Spot and preemption](/docs/guides/capacity/spot-and-preemption): Spot reservations and how preemption cuts their billing.

  - [IAM integration](/docs/guides/iam): The `iam` module's jobs and the inbox that carries offboarding and budget events.


# Delegated confirmation (/docs/guides/iam/delegated-confirmation)

> How an AI agent acting for a person gets CONFIRMATION_REQUIRED instead of a denial, how the draft waits for approval, and how the retry completes.



An AI agent can act for a person through a Better IAM **delegation**: the person grants the agent a set of
scopes, and the agent assumes the delegation to get a session that is decided as the person, narrowed by those
scopes. The person can also hold some actions back for confirmation, such as creating a reservation. With
delegated confirmation, compute turns a denial of such an action into a
request for the person's approval instead of a dead end: the agent receives `CONFIRMATION_REQUIRED`, the person
approves in Better IAM, and the agent retries the same request, which then goes through.

Delegated confirmation is the v0.3 flow of PLAN.md §3.4 step 7. The API envelope implements it for every method
with an admission, and reservations keep the waiting request as a draft in the `awaiting-confirmation` state.

## Set up a delegation [#set-up-a-delegation]

Delegations, agents and confirmations are Better IAM features. The person grants the delegation and names the
actions to hold back in `confirm`:

```ts title="delegate.ts"
// The organization registers the agent and gives it a key.
const agent = await iam.api.agents.create(orgOwner, { tenantId, name: 'Scheduler', model: 'claude-sonnet-5' });
const key = await iam.api.credentials.create(orgOwner, { tenantId, identityId: agent.id });

// The person delegates reservation work, holding back creation for confirmation.
const delegation = await iam.api.delegations.grant(alice, {
  tenantId,
  agentId: agent.id,
  scopes: ['compute:reservations:*'],
  confirm: ['compute:reservations:create'],
});

// The agent assumes the delegation: its session acts as Alice.
const { token } = await iam.api.delegations.assume(
  { token: key.token },
  { tenantId, delegationId: delegation.id },
);
```

Compute sees the result as a caller with `sessionKind: 'delegated'`, the person's `identityId`, and the
`agentId` and `delegationId`. In policies, such a session has `principal.kind` equal to `user` and
`principal.delegated` equal to true, and `principal.mfa` is always false. See
[Policies](/docs/guides/iam/policies) for guardrails that apply to delegated sessions.

## What happens on a denial [#what-happens-on-a-denial]

<Mermaid
  chart="sequenceDiagram
  participant A as Agent
  participant C as Compute
  participant I as Better IAM
  participant P as Person
  A->>C: reservations.create, key k1
  C->>C: transaction 1 writes the draft rsv_...
  C->>I: require compute:reservations:create
  I-->>C: ACCESS_DENIED
  C->>I: requestConfirmation (reason, validSeconds 900)
  I-->>C: confirmationId, expiresAt
  C->>C: bind the reason's digest, draft to awaiting-confirmation
  C-->>A: 403 CONFIRMATION_REQUIRED
  I->>P: confirmation request
  P->>I: decideConfirmation, approve
  A->>I: getConfirmation (poll)
  A->>C: reservations.create, same input, key k1
  C->>I: require on the same draft
  I-->>C: allowed by the approval
  C->>C: same text as approved
  C-->>A: placed reservation"
/>

When the one decision of an admission is refused and the caller's session is delegated, the envelope asks Better
IAM for a confirmation with the caller's own credential:

1. It sends the decision's action and resource, the reason and `validSeconds: 900`. The reason is the method's own
   description of the request (see [what the person sees](#what-the-person-sees)), else
   `<group>.<method>: <action> on <type>/<id>`.
2. If a pending request for the same action and resource already exists, compute reuses it instead of asking the
   person twice.
3. Better IAM answers `INVALID_INPUT` when the delegation does not hold the action back. The caller then gets a
   plain `ACCESS_DENIED`.
4. Otherwise compute records a digest of the reason against the confirmation (see
   [one approval, one request](#one-approval-one-request)), the caller gets `CONFIRMATION_REQUIRED` (HTTP 403), and
   the method's denial hook parks the draft.

## What the person sees [#what-the-person-sees]

The person approves what the confirmation's reason says, so a method that can be held back describes the request
itself (`MethodAdmission.describe`, ec36c63, security review 14 H1). Without it, an agent under prompt injection
could add an SSH key, more GPUs or a longer lifetime, and the person would read only an action and an id. The
descriptions name every parameter that changes what is booked, what runs, or for how long:

| Method                | The reason names                                                                                                                                                                                                                                                                  |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reservations.create` | The pool; each device request as `<count>× <class>` with its selector, sharing and topology; the duration and start (`now` or an ISO time); a gang's size and topology key; a non-`any` topology; a whole-node binding with its profile and version; spot; the lease; the project |
| `reservations.extend` | The reservation and the absolute end: `prepare` resolves `by` into `until` first, so the handler applies what was described                                                                                                                                                       |
| `instances.create`    | The pool, flavor and image; each SSH key as its OpenSSH `SHA256:` fingerprint, never the key; user data by size and a 12-hex-digit SHA-256 prefix, or its `secret://` reference, never the text; created started or stopped; the project                                          |
| `sandboxes.create`    | The template, the lifetime (or the template's default) and the snapshot it restores                                                                                                                                                                                               |
| `sandboxes.extend`    | The sandbox and the absolute end, resolved as for reservations                                                                                                                                                                                                                    |

```text title="reason"
reservations.create: pool pool_01k5w…; 4× gpu; for 8h from now
```

The text is deterministic (no clock and no database reads), so the same request always yields the same text.
Caller-supplied values are clipped to one short line, with control, format and line-separator characters replaced,
so a name cannot forge extra lines in the prompt.

Better IAM hides the reason for a denial, so compute cannot tell "held back" from "forbidden by policy" before
asking. A request for a confirm-listed action that a policy forbids therefore also creates a confirmation request,
which the approval cannot rescue. Upstream change U8 would remove these futile requests.

## The error [#the-error]

```json title="403 response"
{
  "error": {
    "code": "CONFIRMATION_REQUIRED",
    "message": "The person this agent acts for must confirm the request; retry with the same idempotencyKey once they have",
    "details": {
      "confirmationId": "9c1f0e6a-...",
      "expiresAt": 1790001800000,
      "action": "compute:reservations:create",
      "resource": { "type": "compute-reservation", "id": "rsv_..." }
    }
  }
}
```

<TypeTable
  type="{
  confirmationId: {
    type: 'string',
    description: &#x22;Better IAM's confirmation request. Poll it with delegations.getConfirmation.&#x22;,
    required: true,
  },
  expiresAt: {
    type: 'number',
    description: 'Epoch milliseconds when the pending request lapses undecided, when Better IAM reports it.',
  },
  action: {
    type: 'string',
    description: 'The action that was held back, such as compute:reservations:create.',
    required: true,
  },
  resource: {
    type: '{ type: string; id: string }',
    description: 'The exact resource the approval will open: for a create, the draft id derived from the idempotency key.',
    required: true,
  },
}"
/>

The typed client raises it as a `ComputeClientError` with `code: 'CONFIRMATION_REQUIRED'`, `status: 403` and
these fields in `details`.

## The draft awaiting confirmation [#the-draft-awaiting-confirmation]

For `reservations.create`, the draft stays behind and waits:

| Field                            | Value                                                                                              |
| -------------------------------- | -------------------------------------------------------------------------------------------------- |
| `state`                          | `awaiting-confirmation`                                                                            |
| Condition `ConfirmationRequired` | `status: 'true'`, `reason` is the confirmation id, `message` is the deadline in epoch milliseconds |
| Deadline                         | Better IAM's `expiresAt`, but never more than 30 minutes after the denial                          |
| Event                            | `reservation.awaiting-confirmation`, with `confirmationId` and `expiresAt` in `data`               |

`reservations.get` returns the reservation in this state, so the agent and the person can see what is waiting.
It books nothing and holds no quota. At the deadline the reservation lifecycle moves it to `expired`, a terminal
state, and publishes `reservation.expired` with the reason `confirmation expired`. `reservations.cancel` accepts
only `pending` and `placed` reservations, so a waiting draft cannot be cancelled early (`INVALID_STATE`); let it
expire.

<Mermaid
  chart="stateDiagram-v2
  state &#x22;awaiting-confirmation&#x22; as awaiting
  [*] --> draft
  draft --> awaiting: delegated denial, confirmation requested
  awaiting --> draft: retry with the same key
  awaiting --> expired: deadline passed
  draft --> denied: any other denial
  draft --> placed: allowed and placed
  expired --> [*]
  denied --> [*]"
/>

Other admissions handle the waiting differently. `instances.create` deletes its draft on any denial, but the
draft id is derived from the tenant, the caller and the idempotency key, so the retry recreates the same id and
the approval still matches. A delegated `reservations.extend` clears its pending change on denial and is decided
again on the same reservation when retried.

## Approve and retry [#approve-and-retry]

The person decides in Better IAM, from their own session. Better IAM emails them when it has an address for them
and email delivery is configured, and lists pending requests with `delegations.listConfirmations`:

```ts title="approve.ts"
await iam.api.delegations.decideConfirmation(alice, {
  tenantId,
  confirmationId,
  approve: true, // false rejects
});
```

Better IAM's approval opens that action on that resource within the request's `validSeconds`: compute asks for 900
seconds (15 minutes). The person has 30 minutes to decide before the request lapses. Compute narrows the approval
further to the one request the person saw (below).

## One approval, one request [#one-approval-one-request]

Better IAM keys an approval by action and resource and keeps it open for its validity, so after one approved
`reservations.extend` of `rsv_X` an agent could extend `rsv_X` again, by more, and IAM would allow it. Compute binds
each approval to its text (06d7f04, security review 14 M1):

* When the envelope asks for a confirmation, it records the SHA-256 of the reason the person is shown, keyed by
  tenant, delegation, action and resource, with the confirmation's expiry, in `confirmation_bindings` (migration
  `confirmations/0001_bindings`, owned by an always-hosted `confirmations` module). The first text wins, so a reused
  pending confirmation keeps the text the person saw.
* After IAM allows a delegated call, every unexpired binding on its decision must match the digest of the call's own
  text. Otherwise the call is refused with `ACCESS_DENIED` and `details.reason: 'approval-mismatch'` ("The approval
  given for this resource was for a different request; ask again for this one"), and the method's `abandoned` hook
  clears what `prepare` staged. A decision with no open binding goes ahead as before.

So the retry must be the same request: an agent that changes the lifetime, the devices or an SSH key after the
approval needs a new one.

The agent polls the request and retries with the **same input and the same idempotency key**:

```ts title="agent.ts"
import { createIamClient } from '@better-iam/client';
import { ComputeClientError, createComputeClient } from 'better-compute/client';
import type { compute } from './compute.server';

const client = createComputeClient<typeof compute>({ baseURL: 'https://compute.example.com', token: agentToken });
const iamClient = createIamClient({ baseURL: 'https://iam.example.com', token: agentToken });

async function waitForDecision(confirmationId: string) {
  for (;;) {
    const request = await iamClient.delegations.getConfirmation({ tenantId, confirmationId });
    if (request.status === 'rejected') return 'rejected';
    if (request.expired) return 'expired';
    if (request.status === 'approved') return 'approved';
    await new Promise((resolve) => setTimeout(resolve, 2000));
  }
}

export async function reserve() {
  const input = {
    tenantId,
    poolId,
    devices: [{ class: 'gpu', count: 4 }],
    duration: '8h',
    idempotencyKey: 'nightly-2026-09-24',
  };
  try {
    return await client.reservations.create(input);
  } catch (error) {
    if (!(error instanceof ComputeClientError) || error.code !== 'CONFIRMATION_REQUIRED') throw error;
    const decision = await waitForDecision(String(error.details?.confirmationId));
    if (decision !== 'approved') throw error; // rejected or lapsed: stop here
    return await client.reservations.create(input); // same key: same draft, now approved
  }
}
```

On the retry, `prepare` finds the waiting draft. Before its deadline it moves the draft back to `draft`, removes
the condition and decides again; the approved triple of action, type and id allows it, the call's text matches the
one the person approved, and placement continues as usual. After the deadline the retry returns the `expired` reservation
without a decision. The same key with a different request is `CONFLICT`.

Stop when the person rejects. A retry after a rejection is decided again, denied again, and creates a new
confirmation request, because only pending requests are reused. After an expiry, start over with a new
idempotency key.

The MCP tools follow the same loop: they surface `confirmation-required` with the `confirmationId` and
`idempotencyKey`, poll for up to 25 seconds when the host gives them a way to, and retry with the same key. See
[MCP](/docs/guides/clients/mcp).

## Failing closed [#failing-closed]

Delegated confirmation never reveals more than a denial would:

* A session that is not delegated never triggers a confirmation request; it gets `ACCESS_DENIED`.

* An action the delegation does not hold back gets `ACCESS_DENIED`.

* If asking Better IAM fails, the caller gets `ACCESS_DENIED` and the server logs "Requesting a delegated
  confirmation failed; the request stays denied".

* Only methods with an admission ask. Any other method a delegated session is refused answers `ACCESS_DENIED`.

* A delegated call whose text differs from the one an open approval was given for gets `ACCESS_DENIED`
  (`approval-mismatch`).

`tests/server-admission.test.ts` covers the envelope's side (the text binding included),
`tests/approval-descriptions.test.ts` the descriptions, and `tests/iam-bridge-confirmation.test.ts` the round
trip through a real Better IAM, including the reuse of a pending request and an approval that opens only the
draft it was asked for.

  - [Reservations](/docs/guides/capacity/reservations): Drafts, admission, placement and the full reservation lifecycle.

  - [Policies](/docs/guides/iam/policies): `agent-guardrails` and the conditions for delegated sessions.

  - [MCP](/docs/guides/clients/mcp): Compute's tools for AI agents, which handle confirmation for you.


# IAM integration (/docs/guides/iam)

> How Better Compute uses Better IAM as its only identity provider, policy engine, billing ledger and flag store, and how the IAM bridge wires the two.



Better Compute has no user accounts, no roles and no policy language of its own. Every person, service account and
AI agent that calls the API is an identity in Better IAM, and every protected call is decided by Better IAM. Better
Compute keeps only what an IAM system cannot hold for it: the capacity data it decides about, and the machine PKI
that identifies its daemons.

Better IAM plays four roles for compute:

* **Identity provider.** Sessions, API keys, role sessions, delegated sessions for AI agents and session JWTs all
  come from Better IAM. Compute never issues a session.
* **Policy decision point.** Each protected method makes one decision on a `compute:*` action and a `compute-*`
  resource. Policies, roles, bindings, relationships and conditions live in Better IAM.
* **Billing ledger.** Compute measures reserved time and records hourly usage on Better IAM meters. Prices,
  budgets and invoices are Better IAM's.
* **Flag store.** Tenant feature flags such as `compute.h100` gate features inside decisions and, through a
  cache, in code.

> **Installing Better IAM.** 
  Better IAM is on npm. Better Compute 0.2.0 works with Better IAM `>=0.2.0 <0.4.0` (0.3 is current): install `better-iam` next to
  `better-compute` (`npm install better-compute better-iam`). `@better-compute/iam` needs `@better-iam/core` and
  optionally `@better-iam/server`, `@better-iam/client` and `@better-iam/adapter-sqlite`, all of which `better-iam`
  brings. See [Installation](/docs/guides/installation#optional-peers).

## What each side owns [#what-each-side-owns]

| Concern    | Better IAM                                                    | Better Compute                                                                                                            |
| ---------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Identities | People, service accounts, agents, sessions, keys, delegations | None. Daemons, site agents and gateways use compute's own mTLS certificates and never hold an IAM credential.             |
| Decisions  | Policies, roles, bindings, relationships, conditions          | Declares the action and resource of every method, and refuses callers from other tenants before asking                    |
| Resources  | Nothing registered                                            | Every `compute-*` type is app-owned: compute resolves it from its own authz projection |
| Audit      | Denials and root overrides, in the tenant's audit chain       | Allowed mutations, in `audit_events`, with the admission context                      |
| Money      | Meters, prices, budgets, invoices                             | Hourly usage windows, sent with idempotency keys                                           |
| Limits     | Spend budgets                                                 | Quotas per tenant and pool                                                                        |
| Machines   | Nothing                                                       | Node certificates, join tokens, attestors                                                                                 |

## Modes [#modes]

The bridge supports two topologies. **Embedded** is the default and the only one supported from v0.1 to v0.3:
Better IAM runs inside compute's own processes. **Remote** is the v0.4 topology for a Better IAM deployment that
compute does not run.

|                | Embedded (default)                                                 | Remote (v0.4)                                                                                                 |
| -------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| Compute's port | `bridge.embedded(iam, options)`                                    | `bridge.remote({ baseURL, serviceToken, rootTenantId, iamDatabase })`                                         |
| Authentication | `iam.authenticate`, one IAM lock acquisition                       | `sts.getCallerIdentity` over HTTP, two acquisitions                                                           |
| Decisions      | `iam.require`, `iam.authorize` and `iam.authorizeMany` in process  | The same calls over HTTP, each with the caller's own credential forwarded                                     |
| Resolver       | In process, reading compute's projection from memory               | In the IAM host (`computeIamHost`), reading a replica of compute's projection                                 |
| Events         | The plugin's `afterAudit` writes `iam_inbox` in compute's database | The plugin writes `compute_inbox` in the IAM database (`postgresInbox`) or posts signed batches (`httpInbox`) |
| Usage          | `billing.recordMany`, up to 200 events per call                    | HTTP `billing.recordMany` as the root service account, up to 100 per call                                     |

The remote pieces (`bridge.remote`, `remoteIam`, and `computeIamHost`, `replicaProjection`, `postgresInbox`,
`httpInbox` and `createInboxHandler` from `better-compute/iam/plugin`) already exist and are tested against a real
Better IAM: `pnpm test:iam-remote`, part of `pnpm check`, reruns the IAM-dependent suites with `bridge.remote`, the
authz matrix included (see [testing](/docs/operations/testing#real-better-iam-in-tests)). Remote mode belongs to the v0.4 milestone on the [roadmap](/docs/reference/roadmap). Pointing Better
IAM's own console at compute's IAM database needs an upstream Better IAM change (U0) first, in either mode.

## How the pieces fit [#how-the-pieces-fit]

<Mermaid
  chart="flowchart LR
  subgraph proc [&#x22;One compute process&#x22;]
    Envelope[&#x22;API envelope&#x22;] --> Port[&#x22;IamPort&#x22;]
    Port --> IAM[&#x22;betterIam()&#x22;]
    IAM --> Resolver[&#x22;resolveResource&#x22;]
    Resolver --> Memory[&#x22;projection in memory&#x22;]
    IAM -- &#x22;afterAudit&#x22; --> Plugin[&#x22;compute plugin&#x22;]
  end
  Memory -. &#x22;load, then tail&#x22; .-> DB[(&#x22;compute database&#x22;)]
  Plugin -- &#x22;iam_inbox&#x22; --> DB
  DB -- &#x22;iam.inbox&#x22; --> Reactions[&#x22;event reactions&#x22;]
  DB -- &#x22;iam.usage&#x22; --> Port"
/>

`createComputeIamBridge(options)` builds every piece from one merged catalog. Pass it the same `modules` and
`plugins` you pass to `betterCompute()`, and the same `typePrefix` if you change it.

| Piece                                                 | What it does                                                                                                                                                                                     |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `bridge.plugin()`                                     | The `IamPlugin` for `betterIam({ plugins })`: compute's resource types and actions, and an `afterAudit` hook that forwards the events compute reacts to into `iam_inbox`. The hook never throws. |
| `bridge.resolveResource(hostFallback?)`               | Better IAM's `resolveResource`. It answers `compute-*` types from memory and hands every other type to your own resolver.                                                                        |
| `bridge.embedded(iam, options)`                       | The IamPort compute uses, over the Better IAM instance in the same process. Call `embedded()` or `remote()` once per bridge; a second call is `INVALID_STATE`.        |
| `bridge.modules`                                      | The `iam` module: projection and inbox lifecycle, the migration `iam/0001_successors` and the reconcilers listed below.                                                                          |
| `bridge.catalog`, `bridge.projection`, `bridge.flags` | The merged catalog, the memory-resident projection and the flag cache, for tests and diagnostics.                                                                                                |

## Wire embedded mode [#wire-embedded-mode]

Every process that constructs `betterIam()` installs compute's plugin and resolver, and compute receives the port
and the `iam` module:

```ts title="compute.server.ts"
import { postgresAdapter } from '@better-iam/adapter-postgres';
import { betterIam } from '@better-iam/server';
import { createComputeIamBridge } from 'better-compute/iam';
import { metal } from 'better-compute/metal';
import { postgres } from 'better-compute/postgres';
import { betterCompute } from 'better-compute/server';

const modules = [metal()];
const bridge = createComputeIamBridge({ modules }); // the same modules compute registers

export const iam = betterIam({
  database: postgresAdapter({ connectionString: process.env.IAM_DATABASE_URL! }),
  secret: process.env.IAM_SECRET!,
  baseURL: 'https://iam.example.com',
  plugins: [bridge.plugin()], // catalog + afterAudit into iam_inbox
  resolveResource: bridge.resolveResource(), // compute-* types from memory
  // Optional: session JWTs whose audience includes compute.
  sts: { jwt: { signingKeys: [iamSigningKey], audiences: ['https://compute.example.com'] } },
});

export const compute = betterCompute({
  database: postgres({ url: process.env.BETTER_COMPUTE_DATABASE_URL! }),
  secret: process.env.BETTER_COMPUTE_SECRET!,
  baseURL: 'https://compute.example.com',
  modules: [...modules, ...bridge.modules],
  iam: bridge.embedded(iam, { rootTenantId, runIamJobs: true }),
});

await compute.initialize(); // catalog check, migrations, then the iam module loads the projection
```

* `rootTenantId` is the id of Better IAM's root tenant, which compute treats as the platform. You can also pass it
  to `betterCompute({ rootTenantId })`; when both are set they must agree.
* `initialize()` fails with `INVALID_CONFIG` when a method's action is missing from the merged catalog. See
  [Actions and resources](/docs/guides/iam/actions-and-resources).
* Until the `iam` module has loaded the projection, every compute decision is denied and inbox writes spill into
  memory. A host that constructs Better IAM outside compute calls `bridge.bind({ sql })` to do the same.
* `runIamJobs: true` makes the `iam` module run Better IAM's scheduler jobs (below). It needs a real `betterIam()`
  instance, or it throws `INVALID_CONFIG`. Pass an object instead to choose jobs (`only`), report failures
  (`onError`) or set `closeGraceHours` (default 6, from 0 to 72): the hours a month must have run before the previous
  one closes.

For local development without configured IAM, `better-compute dev --sim N` starts an in-memory Better IAM with
compute's plugin and resolver installed and prints its root administrator's token. Plain `better-compute dev`
without IAM answers only public methods. See [Local development](/docs/operations/local-development).

## Credentials compute accepts [#credentials-compute-accepts]

Callers authenticate to compute with Better IAM credentials only:

* **IAM bearer tokens** in the `Authorization` header: user sessions, API keys, role sessions and delegated
  sessions (`biam_…`).
* **Session JWTs** from Better IAM's `sts.getSessionToken({ format: 'jwt', audience: [iamIssuer, computeAudience] })`.
  Better IAM mints them only when `sts.jwt.audiences` contains compute's audience and the caller holds
  `iam:session-tokens:create` and `iam:assertions:create`; the `gpu-user` example policy grants both.
* **The IAM session cookie**, only when `betterCompute({ acceptIamCookie: true })` is set for a console on the same
  origin. Otherwise compute drops the cookie before calling IAM.

Authentication yields a `Caller`: `identityId`, `identityKind` (`user`, `service` or `agent`), `tenantId` (the
acting tenant; a role session acts in the role's tenant), `sessionId`, `sessionKind` (`user`, `role`, `api-key`,
`session-token` or `delegated`), `mfa`, and `agentId` and `delegationId` for agents. Any other identity or session
kind is refused with `UNAUTHENTICATED`.

## Fleet paths never call IAM [#fleet-paths-never-call-iam]

Every Better IAM call takes a turn on one serialized lane: its PostgreSQL adapter takes one advisory lock per
transaction. Compute therefore keeps IAM off every path that scales with the fleet. Heartbeats, inventory
reports, command traffic, joins, gateway jobs and reconcilers make no IAM call. Only API requests do:

| Request                             | IAM cost                                                                                                                |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| A mutation or admission (`require`) | 2 lock acquisitions (Better IAM authenticates again inside `require`), 3 when the 30-second authentication cache misses |
| A read whose allow is cached        | None for 10 seconds                                                                                                     |
| A list page                         | One coarse decision, then at most one `authorizeMany` per 50 items                                                      |
| A feature flag                      | One `features.isEnabled` per tenant and key on a cache miss, never one per admission                                    |

A resolver hit inside IAM's locked transaction does no I/O. A reservation or instance admission costs one more call,
`billingCheck`, when its pool's policy sets `admission.budgetCheck: 'meter'` or the tenant's offer reacts to a
spent budget with `stop-new` or `drain` (see [Billing and usage](/docs/guides/iam/billing-and-usage#budgets)).

`tests/sim/iam-negative.test.ts` enforces the rule: it wraps the port in `spyIamPort`, tags fleet code paths with
`spy.path(name, fn)`, runs ten simulated minutes of joins, heartbeats, inventory, commands and a reservation, and
asserts that every recorded IAM call came from an API admission. `spyIamPort` is exported for your own tests.

## Rules for every IAM host [#rules-for-every-iam-host]

Better IAM's action catalog and event subscribers are process-local, so every process that constructs
`betterIam()` on compute's IAM database must install compute's plugin and resolver. Two failure modes follow when
one does not:

* **No plugin.** That host knows no `compute:*` action. Its decisions come back as `UNKNOWN_ACTION`, which every
  caller sees as `ACCESS_DENIED`. `policies.create` rejects `compute:*` actions with `INVALID_ACTION`,
  `relationships.create` on `compute-pool` fails, and audit events recorded there never reach compute's inbox.
* **Plugin but no resolver.** Compute decisions fail with `RESOURCE_RESOLVER_REQUIRED`.

Until upstream change U0 lands, the supported topology is that compute's own `api` and worker processes are the
only IAM hosts on that database, with `runIamJobs: true` so that someone dispatches Better IAM's audit hooks.
Administrators author policies with a Better IAM CLI whose configuration also installs `computeIamHost`, so that
`compute:*` actions validate.

The plugin's `afterAudit` must never throw: Better IAM awaits every plugin's hook in one loop without a guard, so
one throw would stall the host's whole event pipeline. A write to the inbox that fails or takes longer than
2 seconds spills the event into a bounded in-memory buffer (10 000 events, oldest dropped first), which is flushed
after the next successful write. `timeoutMs` and `spill` on `bridge.plugin()` change both limits.

## IAM events compute reacts to [#iam-events-compute-reacts-to]

The plugin forwards only allowed audit events whose action compute reacts to, and the inbox dedupes them on the
event id, because Better IAM delivers at least once:

| Events                                                                                                                                 | Reaction                                                                                                  |
| -------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `identity:offboard`, `identity:expire`, `identity:delete`, `iam:identities:delete`, `iam:identities:update`                            | `identity`: close streams, reassign work; the successor of an offboarded identity is remembered for usage |
| `auth:session:revoke`, `auth:session:revoke-others`, `identity:revoke-sessions`, `tenant:revoke-sessions`, `iam:roles:revoke-sessions` | `sessions`: recheck streams, drop cached authentications                                                  |
| `agent:suspend`, `agent:suspend-all`, `delegation:revoke`                                                                              | `agent`: close the agent's streams, cancel its unplaced work                                              |
| `binding:activate`, `binding:deactivate`, `iam:bindings:activate`                                                                      | `binding`: cap or close break-glass sessions                                                              |
| `iam:tenants:update`, `iam:tenants:delete`                                                                                             | `tenant`: refuse a suspended tenant's mutations and a deleted tenant's calls, tear the deleted one down   |
| Policy, role, binding and relationship changes, `feature:*`                                                                            | `authz`: drop read-decision and flag caches                                                               |
| `billing:budget-alert`                                                                                                                 | `budget`: apply the offering's budget policy                                                              |

The `iam.inbox` reconciler drains the inbox per tenant in Better IAM's `sequence` order, without waiting for gaps,
and every reaction is level-triggered: it re-reads current state instead of trusting event order. Two reactions are
built in: remembering `successorId` from `identity:offboard`, and invalidating the flag cache. The rest are hooks.

`better-compute serve` installs default handlers when the configuration's host sets `host.iam: { bridge }`
(`installIamHandlers`), each running in the inbox transaction:

| Reaction   | Default                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent`    | Closes the agent's or delegation's tunnels on every live gateway within the 2-second inbox drain (`agentStreamCloser(gatewayStreamCloser(...))`), drops its cached authentications, and cancels the reservations it created that have not started (`pending`, `placed`, `awaiting-confirmation`) with capacity's `withdrawAgentReservations`                                                                                                                                                                                                                               |
| `identity` | Drops the identity's cached authentications. For an owner gone for good (offboarded, expired, deleted, or found so by the `IamReconcile` backstop), `identityEndOf` finds the successor (from the event, or the one remembered in `iam_successors`) and `reassignReservations` hands its live reservations to it (`reservation.reassigned`); without a successor its unstarted reservations are cancelled. Its break-glass SSH certificates are revoked in every tenant, then its streams and tunnels close                                                                |
| `sessions` | Clears the authentication cache. When an identity's own sessions are revoked, it revokes the identity's break-glass SSH certificates (the event does not say which session ended) and re-authorizes its tunnels at once                                                                                                                                                                                                                                                                                                                                                    |
| `authz`    | Drops read-decision and flag caches                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `binding`  | On `binding:deactivate`, revokes the holder's live SSH certificates in every tenant, including child tenants reached through an inherited binding (`node.ssh-cert.revoked`, the serial joins the node's KRL), and has every gateway re-check the holder's tunnels. The holder is the actor only when they ended their own activation; a revocation must name its holder in `metadata.identityId`, and one that does not is reported through `onError` and left to the per-minute re-authorization, so an administrator's own certificates are never taken for the holder's |
| `budget`   | `budgetAlertHandler(applyBudgetPolicy)`: for an exceeded budget, each offer to the tenant publishes `quota.budget-exceeded`, and `drain` also cancels the tenant's unstarted reservations in that pool                                                                                                                                                                                                                                                                                                                                                                     |
| `tenant`   | The server's `tenantStateHandler`: mirrors suspended and deleted tenants into `compute_tenant_states`, publishes `tenant.state-changed`, reloads the envelope's tenant gate and schedules a deleted tenant's teardown. With `host.iam.readTenants` a suspension or deletion cascades to the subtree, read by the `tenancy` reconciler outside the inbox transaction and retried while IAM is down; without it only a deletion of the named tenant applies. See [tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants)                                           |

`host.iam.handlers` replace the defaults one by one. A failed successor lookup fails the reaction, so the inbox
retries it rather than cancelling as if there were no successor.

**Remote IAM without its database.** When compute cannot read the IAM database's inbox, Better IAM can deliver the
same events as signed webhooks (v0.4 fallback intake). `host.iam.webhook: { secret, path? }` makes `serve` mount
`createIamWebhookHandler({ secret, inbox })` at `POST /iam/events` (or `path`); it verifies
`X-Better-IAM-Signature` (HMAC-SHA256 over `<timestamp>.<body>`, 300-second tolerance), writes the events compute
reacts to into `iam_inbox`, and dedupes redeliveries on the audit event id. The IAM root subscribes once with
`webhooks.create({ scope: 'subtree', events: computeIamEventPatterns, outcomes: ['allow'], url })`, and `secret` is
what that call returned. Tampered, stale and unsigned deliveries get 401.
See [tunnels](/docs/guides/access/tunnels#closing-on-iam-revocation). If you build the server yourself, wire the ones
you need:

```ts title="compute.server.ts"
import { agentStreamCloser, gatewayStreamCloser } from 'better-compute/iam';

bridge.setHandlers({
  // gatewayClients: gatewayControlClient(...) for each gateway's internal control API
  agent: agentStreamCloser(gatewayStreamCloser(() => gatewayClients)),
  async identity(_tx, event) {
    if (event.subjectId) compute.caches.dropIdentity(event.subjectId);
  },
  async sessions() {
    compute.caches.clear(); // rare; costs only re-authentication
  },
  async authz() {
    compute.caches.dropDecisions();
  },
  onError(error, event, reaction) {
    console.warn('IAM reaction failed', reaction, event.action, error);
  },
});
```

Each handler runs inside the drain transaction under its own savepoint. A handler that fails is reported to
`onError`, and the event is marked processed anyway: `iam.reconcile` repairs state, not events.

## Jobs the iam module runs [#jobs-the-iam-module-runs]

The `iam` module's reconcilers are singletons (`shardBy: 'none'`): exactly one worker running the `controllers`
role runs each one at a time, under its own lease.

| Reconciler              | Every | What it does                                                                                                                                  |
| ----------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `iam.inbox`             | 2 s   | Claims up to 500 unprocessed `iam_inbox` rows with `FOR UPDATE SKIP LOCKED`, runs the reactions, and deletes processed rows after 7 days      |
| `iam.reconcile`         | 5 min | Re-reads from IAM every identity that owns a non-terminal reservation, and hands inactive ones to the `identity` handler. Embedded mode only. |
| `iam.usage`             | 1 h   | The usage sink: sends closed usage windows to Better IAM billing                                                                              |
| `iam.pull`              | 1 s   | Remote mode with `iamDatabase`: moves rows from the IAM database's `compute_inbox` into `iam_inbox`                                           |
| `iam.jobs.audit-hooks`  | 1 s   | With `runIamJobs`: Better IAM's `events.dispatch()`, which delivers audit events to plugins, compute's inbox included                         |
| `iam.jobs.auth-outbox`  | 5 s   | With `runIamJobs`: `auth.dispatchOutbox()`                                                                                                    |
| `iam.jobs.budgets`      | 1 h   | With `runIamJobs`: `billing.checkBudgets()`                                                                                                   |
| `iam.jobs.close-period` | 1 h   | With `runIamJobs`: `billing.closePeriod()`, once the current UTC month is `closeGraceHours` old (default 6)                                   |
| `iam.jobs.sweep`        | 1 h   | With `runIamJobs`: `sweepExpired()`                                                                                                           |

`betterCompute({ iamJobs })` is an alternative for hosts that bring their own job functions: those run under the
`iam` worker role, each under a `singleton:iam:<name>` lease.

## Set up and check [#set-up-and-check]

`better-compute iam setup` prints the example policies for organization administrators to apply, and with
`--meters` creates the root billing meters. See [Policies](/docs/guides/iam/policies) and
[Billing and usage](/docs/guides/iam/billing-and-usage).

`better-compute doctor` runs `compute.selfCheck()`, which includes the `iam` module's own checks:

| Finding              | Severity                                   | Meaning                                                                                                                      |
| -------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `iam.not-configured` | warning                                    | The configuration uses `unconfiguredIam()`: no credential authenticates and only `system.version` and `system.whoami` answer |
| `iam.projection`     | error                                      | The authz projection is not loaded, so every compute decision is denied                                                      |
| `iam.inbox-spill`    | warning, or error once events were dropped | IAM events are spilled in memory because the inbox could not be written                                                      |
| `iam.inbox-backlog`  | warning                                    | More than 10 000 events wait in `iam_inbox`: no worker runs `iam.inbox`                                                      |

The CLI cannot see the IAM hosts themselves. On each process that constructs `betterIam()`, call
`checkIamHost(iam, { catalog: bridge.catalog, credential, rootTenantId, expectSessionJwts })` with a root
administrator's credential. It returns findings instead of throwing: `iam.catalog` and `iam.resource-types` (the
plugin is missing), `iam.resolver` (a synthetic decision on `compute-fleet/global` failed, typically with
`RESOURCE_RESOLVER_REQUIRED`) and `iam.session-jwt` (session JWTs are expected but `sts.jwt` is not configured).
PLAN.md lists further host checks, such as the inbox heartbeat age and a synthetic session JWT with
`doctor --token`, that are not implemented yet.

## In this section [#in-this-section]

  - [Actions and resources](/docs/guides/iam/actions-and-resources): The `compute:*` catalog, `compute-*` types, the method-to-action table and list filtering.

  - [Policies](/docs/guides/iam/policies): The `gpu-user` and `agent-guardrails` examples, pool relations and conditions.

  - [Tenancy](/docs/guides/iam/tenancy): The root tenant, organizations, offerings and projects.

  - [Billing and usage](/docs/guides/iam/billing-and-usage): Meters, hourly usage windows, the usage sink and budgets.

  - [Delegated confirmation](/docs/guides/iam/delegated-confirmation): Agents acting for people, `CONFIRMATION_REQUIRED` and the retry.


# Policies (/docs/guides/iam/policies)

> Write Better IAM policies for Better Compute, from the gpu-user and agent-guardrails examples to pool relations, resource attributes and conditions.



Who may do what in Better Compute is written in Better IAM policies. A policy names `compute:*` actions and
`compute-*` resources (see [Actions and resources](/docs/guides/iam/actions-and-resources)), and conditions on
the attributes compute attaches to each resource. Roles group policies, and bindings give roles to people,
groups, service accounts and agents, exactly as for any other Better IAM application.

Compute never writes an organization's policies. It ships two example policies that organization administrators
review and apply, and every policy that names a `compute:*` action must be created on a Better IAM host that has
compute's plugin installed. A host without it rejects the policy with `INVALID_ACTION`.

## Print the example policies [#print-the-example-policies]

`better-compute iam setup` prints the examples as JSON, `{ "policies": [...] }`, parameterized for your
deployment:

```bash
better-compute iam setup --audience https://compute.example.com \
  --gpu-model H100 --regions eu-west-1,eu-central-1 --max-gpus 8 --max-hours 72
```

| Flag          | Default                        | What it sets                                                                                     |
| ------------- | ------------------------------ | ------------------------------------------------------------------------------------------------ |
| `--audience`  | from `BETTER_COMPUTE_BASE_URL` | The compute API URL (reduced to its origin), the audience of session JWTs for compute. Required. |
| `--gpu-model` | `H100`                         | The GPU model family the bounded-reservation statement allows                                    |
| `--regions`   | `eu-west-1,eu-central-1`       | The regions it allows                                                                            |
| `--max-gpus`  | `8`                            | GPUs per reservation                                                                             |
| `--max-hours` | `72`                           | Hours per reservation, extensions included                                                       |
| `--feature`   | `compute.<model>`              | The tenant flag the statement requires in `tenant.features`                                      |

The same function is exported for code: `examplePolicies({ audience, gpuModel, regions, maxGpus, maxHours,
feature, typePrefix })` returns both documents in the `{ name, document: { version: 1, statements } }` shape that
Better IAM's `policies.create` takes:

```ts title="apply-policies.ts"
import { examplePolicies } from 'better-compute/iam';

const ids: string[] = [];
for (const { name, document } of examplePolicies({ audience: 'https://compute.example.com' }))
  ids.push((await iam.api.policies.create(orgOwner, { tenantId: orgId, name, document })).id);

const role = await iam.api.roles.create(orgOwner, { tenantId: orgId, name: 'gpu-users', policyIds: ids });
await iam.api.bindings.create(orgOwner, {
  tenantId: orgId,
  roleId: role.id,
  subjectType: 'group',
  subjectId: mlTeamGroupId,
});
```

`tests/iam-contract.test.ts` applies both policies verbatim to a real Better IAM and checks every condition below.

## The gpu-user policy [#the-gpu-user-policy]

`gpu-user` is for members who reserve GPUs from pools shared with them, within bounds:

```json title="gpu-user"
{
  "name": "gpu-user",
  "document": {
    "version": 1,
    "statements": [
      {
        "sid": "ListInTenant",
        "effect": "allow",
        "actions": ["compute:pools:read", "compute:capacity:read", "compute:reservations:read"],
        "resources": ["compute-tenant/*"]
      },
      {
        "sid": "SeeSharedPools",
        "effect": "allow",
        "actions": ["compute:pools:read", "compute:capacity:read"],
        "resources": ["compute-pool/*"],
        "conditions": { "ArrayContains": { "resource.relations": "user" } }
      },
      {
        "sid": "ReserveBoundedH100",
        "effect": "allow",
        "actions": ["compute:reservations:create"],
        "resources": ["compute-reservation/*"],
        "conditions": {
          "StringEquals": { "resource.gpuModel": "H100", "resource.region": ["eu-west-1", "eu-central-1"] },
          "NumericLessThanEquals": { "resource.gpuCount": 8, "resource.durationHours": 72 },
          "ArrayContains": { "resource.parentRelations": "user", "tenant.features": "compute.h100" }
        }
      },
      {
        "sid": "OwnReservations",
        "effect": "allow",
        "actions": ["compute:reservations:read", "compute:reservations:release"],
        "resources": ["compute-reservation/*"],
        "conditions": { "StringEquals": { "resource.ownerId": "${principal.id}" } }
      },
      {
        "sid": "BoundedExtend",
        "effect": "allow",
        "actions": ["compute:reservations:extend"],
        "resources": ["compute-reservation/*"],
        "conditions": {
          "StringEquals": { "resource.ownerId": "${principal.id}" },
          "NumericLessThanEquals": { "resource.durationHours": 72 }
        }
      },
      {
        "sid": "StopWhenOverBudget",
        "effect": "deny",
        "actions": [
          "compute:reservations:create",
          "compute:reservations:extend",
          "compute:instances:create",
          "compute:sandboxes:create",
          "compute:sandboxes:update"
        ],
        "resources": ["*"],
        "conditions": { "Bool": { "principal.spendExceeded": true } }
      },
      {
        "sid": "SessionJwtForCompute",
        "effect": "allow",
        "actions": ["iam:session-tokens:create"],
        "resources": ["iam/${principal.id}"]
      },
      {
        "sid": "AssertToCompute",
        "effect": "allow",
        "actions": ["iam:assertions:create"],
        "resources": ["iam/https://compute.example.com"]
      }
    ]
  }
}
```

| Statement                                 | What it allows or denies                                                                                                                                                                                                                                         |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ListInTenant`                            | The coarse gate of `pools.list`, `reservations.list` and `capacity.summary`, decided on the tenant scope                                                                                                                                                         |
| `SeeSharedPools`                          | Reading a pool, and its capacity, when the member holds the `user` relation on it, directly or through a group                                                                                                                                                   |
| `ReserveBoundedH100`                      | Creating a reservation when the draft asks for H100s in the listed regions, at most 8 GPUs for at most 72 hours, in a pool where the member holds `user`, and the tenant has the `compute.h100` flag                                                             |
| `OwnReservations`                         | Reading and releasing the member's own reservations. `${principal.id}` is a Better IAM policy variable.                                                                                                                                                          |
| `BoundedExtend`                           | Extending the member's own reservations while the total stays within 72 hours                                                                                                                                                                                    |
| `StopWhenOverBudget`                      | Denies new reservations, extensions and instances once an enforced Better IAM budget covering the member is spent, and new sandboxes and sandbox extensions (`compute:sandboxes:create`, and `compute:sandboxes:update`, which also covers `sandboxes.snapshot`) |
| `SessionJwtForCompute`, `AssertToCompute` | What Better IAM checks before it mints a session JWT whose audience includes compute                                                                                                                                                                             |

`BoundedExtend` cannot be bypassed by extending in small steps. Before deciding, `reservations.extend` writes the
reservation's new **total** `durationHours` into the projection row's pending overlay, and the resolver reports
the pending value. A denied extension clears the overlay.

The flag in `ReserveBoundedH100` must be on for the organization in Better IAM (its feature flags), or the
statement never applies. `--feature` names a different flag.

## The agent-guardrails policy [#the-agent-guardrails-policy]

`agent-guardrails` keeps AI agents away from the most dangerous actions and caps what they may reserve:

```json title="agent-guardrails"
{
  "name": "agent-guardrails",
  "document": {
    "version": 1,
    "statements": [
      {
        "sid": "AgentKeys",
        "effect": "deny",
        "actions": ["compute:nodes:ssh-root", "compute:nodes:provision", "compute:images:publish"],
        "resources": ["*"],
        "conditions": { "StringEquals": { "principal.kind": "agent" } }
      },
      {
        "sid": "DelegatedSessions",
        "effect": "deny",
        "actions": ["compute:nodes:ssh-root", "compute:nodes:provision", "compute:images:publish"],
        "resources": ["*"],
        "conditions": { "Bool": { "principal.delegated": true } }
      },
      {
        "sid": "DelegatedSmall",
        "effect": "deny",
        "actions": ["compute:reservations:create", "compute:reservations:extend"],
        "resources": ["compute-reservation/*"],
        "conditions": {
          "Bool": { "principal.delegated": true },
          "NumericGreaterThan": { "resource.gpuCount": 4 }
        }
      }
    ]
  }
}
```

Two statements are needed because Better IAM reports the identity's kind in `principal.kind`. An agent using its
own key is `agent`, but a delegated session acts as the person, so its kind is `user` and only
`principal.delegated` tells it apart. Both keys are always present, so these denies cannot be skipped by a
missing key. To let agents ask a person instead of being refused outright, see
[Delegated confirmation](/docs/guides/iam/delegated-confirmation).

## Resource attributes [#resource-attributes]

Conditions on `resource.*` read what compute writes into each resource's projection row. These are the attributes
the rows carry today:

| Type                       | Attributes                                                                                                                                                                                                                                                                                                                                       | Owner and parent                                                        |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| `compute-reservation`      | `gpuModel` (`other` when the pool has none), `gpuCount` (whole GPUs, times the gang size), `gpuSlices`, `gangSize`, `durationHours`, `region`, `binding` (`hold` or `node`), `spot`, `renewable`, and `projectId` when given                                                                                                                     | `ownerId` is the requesting identity; the parent is the pool            |
| `compute-instance`         | `vcpu`, `memoryGiB`, `gpus`, `imageId`, `imageVerified`, and `projectId` when given                                                                                                                                                                                                                                                              | `ownerId` is the requesting identity; the parent is the pool            |
| `compute-pool`             | `gpuModel`, `class`, `sharing`, `region` (its site's), and `owned` (true in the owner tenant's view)                                                                                                                                                                                                                                             | The parent is the site                                                  |
| `compute-node`             | `region`, `gpuCount`, `lifecycle` (the provision state), `trust`, and `gpuModel` when the node is in a pool with one                                                                                                                                                                                                                             | The parent is the pool                                                  |
| `compute-site`             | `region`, `kind`, and `iamRegion` when set                                                                                                                                                                                                                                                                                                       |                                                                         |
| `compute-project`          | none                                                                                                                                                                                                                                                                                                                                             | `ownerId` is the creator                                                |
| `compute-sandbox`          | `templateId`, `vcpu`, `memoryGiB` and `gpus` (the template's flavor), `ttlSeconds` (the lifetime asked for; during `sandboxes.extend`, the pending total from creation to the new expiry), `warm` (true when the template keeps a warm pool, `warm.min > 0`, and the create is not a restore), and `fromSnapshot` (the snapshot id) on a restore | `ownerId` is the requesting identity; the parent is the template's pool |
| `compute-sandbox-template` | none                                                                                                                                                                                                                                                                                                                                             | `ownerId` is the template's author                                      |

A reservation's `gpuCount` counts whole GPUs only; fractional requests show up in `gpuSlices`. The catalog also
declares `gpuMemoryGb` on pools and `dedicatedTo` on nodes, which no row carries yet, so a condition on them
never matches. Rows written by `metal` verbs carry `provisionState`, `poolId` and `siteId` instead of the node
controller's attributes.

A missing attribute fails every condition operator except `Exists`. For a deny statement that means the deny
does not apply, so guard denies on optional attributes such as `projectId` with an `Exists` statement.

## Pool relations [#pool-relations]

Relationships let an organization share a pool with a team without listing ids in policies. The catalog declares
four relations on `compute-pool` (`viewer`, `user`, `operator`, `admin`), `operator` on `compute-node`, and
`member` and `admin` on `compute-project`. A relation grants nothing by itself: policies test it, as
`resource.relations` on the resource itself and as `resource.parentRelations` on its children. Reservations,
instances and nodes have their pool as parent.

```ts title="share-pool.ts"
await iam.api.relationships.create(orgOwner, {
  tenantId: orgId,
  type: 'compute-pool',
  id: poolId, // the pool's compute id (pool_...)
  relation: 'user',
  subjectType: 'group',
  subjectId: mlTeamGroupId,
});
```

Relationships live in the organization's own tenant, so an organization can share a platform pool that is offered
to it. One common split of the four pool relations:

| Relation   | Typical statements                                                                                                                  |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `viewer`   | `compute:pools:read` and `compute:capacity:read` on `compute-pool/*` with `resource.relations`                                      |
| `user`     | The above, plus `compute:reservations:create` on `compute-reservation/*` with `resource.parentRelations`                            |
| `operator` | `compute:nodes:read`, `compute:nodes:cordon` and `compute:nodes:drain` on `compute-node/*` with `resource.parentRelations`          |
| `admin`    | `compute:pools:manage`, `compute:pools:share` and the `compute:join-tokens:*` actions on `compute-pool/*` with `resource.relations` |

```json title="Operate the nodes of pools you operate"
{
  "sid": "OperatePoolNodes",
  "effect": "allow",
  "actions": ["compute:nodes:read", "compute:nodes:cordon", "compute:nodes:drain"],
  "resources": ["compute-node/*"],
  "conditions": { "ArrayContains": { "resource.parentRelations": "operator" } }
}
```

> **Share pools, not reservations.** 
  Never create one relationship per reservation or instance. Better IAM loads all of a tenant's relationship
  tuples for every decision, so per-item tuples slow down every call in the tenant. Share pools and projects, and
  test ownership with `resource.ownerId`.

## Conditions compute relies on [#conditions-compute-relies-on]

These Better IAM context keys carry most compute policies:

| Key                                              | Use                                                                              |
| ------------------------------------------------ | -------------------------------------------------------------------------------- |
| `resource.<attribute>`                           | The attributes in the table above, including pending `extend` values             |
| `resource.ownerId`                               | "Own" statements, with the `${principal.id}` variable                            |
| `resource.relations`, `resource.parentRelations` | Relations on the resource and on its pool                                        |
| `principal.kind`                                 | `user`, `service` or `agent`; a delegated session is `user`                      |
| `principal.delegated`                            | True while an AI agent acts for the person                                       |
| `principal.mfa`                                  | True for MFA-verified sessions; always false for API keys and delegated sessions |
| `principal.spendExceeded`                        | True once an enforced budget covering the principal is spent                     |
| `tenant.features`                                | The flags that are on for the decision's tenant, such as `compute.h100`          |

Flags gate features inside decisions at no extra cost. PLAN.md names `compute.baremetal`, `compute.h100` and
`compute.byo-nodes` as examples; compute's code reads flags through `IamPort.featureEnabled`, backed by the flag
cache, and no built-in method requires a flag today.

## Fleet administrators [#fleet-administrators]

Fleet-wide methods decide on `compute-fleet/global` in the root tenant, so platform operators hold root-tenant
policies. This example is a starting point, not a shipped policy:

```json title="fleet-operator"
{
  "name": "fleet-operator",
  "document": {
    "version": 1,
    "statements": [
      {
        "sid": "Fleet",
        "effect": "allow",
        "actions": ["compute:fleet:admin", "compute:sites:manage", "compute:pools:create", "compute:quotas:manage"],
        "resources": ["compute-fleet/global"]
      },
      {
        "sid": "ListInRoot",
        "effect": "allow",
        "actions": ["compute:sites:read", "compute:pools:read", "compute:nodes:read"],
        "resources": ["compute-tenant/*"]
      },
      {
        "sid": "PlatformPools",
        "effect": "allow",
        "actions": [
          "compute:pools:read", "compute:pools:manage", "compute:pools:share", "compute:capacity:read",
          "compute:join-tokens:create", "compute:join-tokens:read", "compute:join-tokens:revoke"
        ],
        "resources": ["compute-pool/*"]
      },
      {
        "sid": "PlatformNodes",
        "effect": "allow",
        "actions": ["compute:nodes:read", "compute:nodes:cordon", "compute:nodes:drain", "compute:nodes:update", "compute:nodes:exec"],
        "resources": ["compute-node/*"]
      }
    ]
  }
}
```

A root-tenant operator acts on an organization's resources only through Better IAM's root override, which needs
root authority and an MFA-verified user session, and which Better IAM audits. See
[Tenancy](/docs/guides/iam/tenancy).

## Break-glass access [#break-glass-access]

PLAN.md describes break-glass root access to platform nodes for v0.3, when `nodes.issueSshCert` and
`tunnels.open` arrive (both are planned in this release). The policy allows the console and root SSH only on
platform-trust nodes from an MFA session:

```json title="break-glass"
{
  "sid": "BreakGlass",
  "effect": "allow",
  "actions": ["compute:nodes:console", "compute:nodes:ssh-root"],
  "resources": ["compute-node/*"],
  "conditions": {
    "Bool": { "principal.mfa": true },
    "StringEquals": { "resource.trust": "platform" }
  }
}
```

It is bound as an eligible binding, which a person must activate before use:
`iam.api.bindings.create(credential, { tenantId, roleId, subjectType, subjectId, eligible: true, requireMfa: true,
requireJustification: true, requireApproval: true })`. The planned certificate and tunnel lifetimes never exceed
the activation's expiry, and deactivating the binding closes the sessions. MFA on join tokens is a per-pool
policy you may add, not a default, because CI cannot carry MFA.

## Validation and testing [#validation-and-testing]

* Create compute policies on a host with compute's plugin. A wildcard such as `compute:*` validates on any host,
  but it decides only where the plugin is installed; elsewhere every compute decision denies.
* A denial never explains itself: callers see `ACCESS_DENIED`. Use Better IAM's policy simulation as an
  administrator to see why.
* `tests/iam-bridge-matrix.test.ts` grants each action and type pair of the method table in a real Better IAM
  and checks that a grant on one type never allows the action on another. Use the same approach, with
  `createEmbeddedTestIam` from `better-compute/iam/testing`, to test your own policies. See
  [Testing](/docs/operations/testing).


# Tenancy (/docs/guides/iam/tenancy)

> How Better Compute maps onto Better IAM tenants, with the root tenant as the platform, pools offered to organizations, and projects inside a tenant.



Better Compute has no tenant table of its own. A tenant is a Better IAM tenant id, every
compute record carries the id of the tenant that owns it, and every method takes a `tenantId`. Better IAM's
tenant tree gives compute its structure:

* The **root tenant** is the platform. It owns the shared fleet and decides fleet-wide operations.
* **Organization tenants** are customers. They own their reservations, instances and projects, and any hardware
  they bring.
* **Child tenants** below an organization exist only when you need billing isolation.

Inside a tenant, compute's projects group work without creating more tenants.

<Mermaid
  chart="flowchart TD
  Root[&#x22;Root tenant: the platform&#x22;] --> Acme[&#x22;Organization: Acme&#x22;]
  Root --> Globex[&#x22;Organization: Globex&#x22;]
  Acme --> Child[&#x22;Child tenant, for billing isolation&#x22;]
  Root -. &#x22;pools.offer&#x22; .-> Acme
  Acme --- Projects[&#x22;Projects in compute&#x22;]"
/>

## What lives where [#what-lives-where]

| Root tenant (platform)                                                  | Organization tenant                                                                |
| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Platform sites (`shared` and `dedicated`), their pools and nodes        | Reservations and instances                                                         |
| Offerings of platform pools to organizations | Projects                                                                           |
| Public images and profiles                                              | Private images and profiles                                                        |
| Platform provider accounts and secrets                                  | `custom` sites with their pools and `tenant-owned` nodes (bring your own hardware) |
| Billing meters and fleet roles                                          | Webhooks, tenant secrets and tenant provider accounts                              |

The method table encodes this split. `sites.create` decides on `compute-fleet` for `shared` and `dedicated` sites,
which must name the root tenant (another `tenantId` is `INVALID_INPUT`), and on `compute-tenant` for an
organization's `custom` site. `pools.create` decides on `compute-fleet` for a platform pool and on
`compute-tenant` for a pool on the organization's own `custom` site. See
[Actions and resources](/docs/guides/iam/actions-and-resources).

## The root tenant [#the-root-tenant]

Tell compute which Better IAM tenant is the root, either as `betterCompute({ rootTenantId })` or through the port
with `bridge.embedded(iam, { rootTenantId })`. When both are given they must agree, and a missing or malformed id
is `INVALID_CONFIG`.

The root tenant is special in three ways:

* `compute-fleet/global` resolves only in the root tenant, so fleet-wide methods (`system.selfCheck`,
  `quotas.set`, `fleet.*`, `rollouts.*`, platform `sites.create` and `pools.create`) are decided there, whatever
  tenant their input names.
* Root-tenant callers pass compute's cross-tenant pre-check for any tenant. Better IAM then decides: it allows a
  root-tenant caller into another tenant only through its root override, which needs root authority and an
  MFA-verified user session, and it audits every override. Platform support on a customer's objects happens only
  this way.
* The billing meters live in the root tenant, and Better IAM resolves them for every tenant below it.

## Offering pools to organizations [#offering-pools-to-organizations]

A platform pool exists in an organization's view only once the platform offers it. `pools.offer` writes a
`pool_offers` row with the organization's quota and priority, and a projection row that makes the pool resolve
in that tenant:

```ts title="offer.ts"
await compute.api.pools.offer(rootOperator, {
  tenantId: rootTenantId, // the pool's owner
  poolId,
  toTenantId: acmeTenantId,
  quota: { gpu: 16 }, // per resource; a resource left out grants 0
  priority: 10, // -1000 to 1000, default 0
});
```

* The call decides `compute:pools:share` on the pool. Offering again updates the quota and priority.
* A pool holding `tenant-owned` nodes, or a single-tenant sharing mode (`timeslice`, `mps`), can be offered only
  to its owner. A pool with `policy.singleTenant: true` (required for VPS container runtimes) can be offered to
  one tenant besides its owner, without `includeDescendants`; another offer is `INVALID_STATE`.
* `pools.unoffer` removes the pool from the organization's view and drops its quota to zero. Live reservations
  keep their charge; nothing new fits. A pool cannot be unoffered from its owner (`INVALID_INPUT`).
* `includeDescendants: true` is stored on the offering, but this release writes the projection row only for
  `toTenantId`, so child tenants do not see the pool yet. PLAN.md expands offerings to descendants from a tenant
  tree cached from `iam:tenants:*` events; until then, offer the pool to each child tenant that needs it.

Organizations then share the pools they were offered with their own teams through Better IAM relationships. See
[Pool relations](/docs/guides/iam/policies) and [Pools](/docs/guides/capacity/pools).

## Issuing compute to organizations [#issuing-compute-to-organizations]

A platform pool can be a provider pool on a root provider account. The platform then issues machines on it to an
organization: `instances.attach` hands over a VM that already runs, and `instances.create` with `ownerIdentityId`
creates one, both with the organization's `tenantId` and an identity of the organization as owner. Only a root caller
acting in another tenant names an owner other than itself, and the owner must be an active member of the tenant when
the IAM port can tell (`IamPort.isMember`, implemented by the embedded bridge). The instance is then the
organization's like any other: in its owner's `scope: 'mine'` list, under the organization's policies (an owner
condition on `resource.ownerId` matches the person it was issued to), on its quota.

The organization never sees or touches what stays the platform's: the root account (a tenant reading it gets
`NOT_FOUND`), its machines and reservations, the pool's management and flavors, quotas (`quotas.set` is decided
on the fleet), and the pool's `providerAccountId`. A pool of the organization's own runs only on the organization's own
provider accounts, whose secret references are the organization's. Modules learn the root from
`context.iam.rootTenantId`, which `betterCompute()` sets to the resolved root whether the port or
`betterCompute({ rootTenantId })` names it. See
[platform-issued compute](/docs/guides/vps/providers#platform-issued-compute) for the full split.

## Visibility is the projection [#visibility-is-the-projection]

What a tenant can see is exactly what compute's authz projection holds in its
view: one row per resource type, id and tenant. List queries filter on the same rows, and the resolver answers
only from them.

| Resource                       | The views it exists in                                                                                                                                            |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Platform pool                  | The root tenant, and each tenant it is offered to                                                                                                                 |
| Pool on a `custom` site        | Its owner tenant                                                                                                                                                  |
| Node                           | Its owner tenant                                                                                                                                                  |
| Site                           | Its owner tenant                                                                                                                                                  |
| Reservation, instance, project | Its owner tenant                                                                                                                                                  |
| Provider account, network      | Its owner tenant (a network on an offered platform pool is the organization's)                                                                                    |
| Image                          | Its owner tenant. `images.promote` writes a row under the wildcard tenant `*`, PLAN.md's public row; the resolver in this release matches a row's tenant exactly. |

Inside your own tenant, an id that is not in your view resolves to `NOT_FOUND`. An id in another tenant never gets
that far: compute refuses the caller with `ACCESS_DENIED` before reading its database.

## Projects [#projects]

A project is an app-owned grouping inside one tenant, stored in compute's `projects` table:

```sql title="projects"
CREATE TABLE projects (
  id text PRIMARY KEY,           -- prj_...
  tenant_id text NOT NULL,
  name text NOT NULL,            -- unique in the tenant
  state text NOT NULL,           -- active | archived
  version bigint NOT NULL,
  UNIQUE (tenant_id, name)
);
```

```ts title="projects.ts"
const project = await compute.api.projects.create(credential, {
  tenantId,
  name: 'ml-research', // a DNS label, at most 63 characters
  idempotencyKey: 'create-ml-research',
});
```

* `projects.create` decides `compute:projects:manage` on `compute-tenant/{t}`. It is idempotent per tenant,
  caller and key: the id derives from them, the same request again returns the project without a second decision,
  and the same key with a different name is `CONFLICT`. A second project with the same name is `CONFLICT` too.
* The creator becomes the project's `ownerId` in the projection. Membership is a Better IAM relationship on
  `compute-project/{id}`, with the relations `member` and `admin`.
* `projects.get` decides `compute:projects:read` on the project; `projects.list` follows the list recipe and
  filters by `state`.
* `projects.archive` (v0.2) decides `compute:projects:manage` on the project. It sets `state` to `archived`, bumps
  `version` and publishes `project.archived` (`projectId`, `name`); archiving an archived project returns it
  unchanged, with no event. Archiving only changes the state: compute does not check it when a reservation or
  instance names the project.

Reservations and instances take an optional `projectId`, which becomes the `projectId` attribute of their
resource and the `project` tag of their usage windows. `usage.list` with a `projectId` decides
`compute:usage:read` on the project instead of the tenant, so project members can read their project's usage:

```json title="Project members read their project and its usage"
{
  "sid": "ProjectMembers",
  "effect": "allow",
  "actions": ["compute:projects:read", "compute:usage:read"],
  "resources": ["compute-project/*"],
  "conditions": { "ArrayContains": { "resource.relations": "member" } }
}
```

Compute checks only that a reservation's `projectId` is a `prj_` id. A reservation's parent is its pool, not its
project, so a policy cannot test project membership when it decides `reservations.create`. Restrict which
projects a group may charge with `resource.projectId` conditions instead.

## Child tenants for billing isolation [#child-tenants-for-billing-isolation]

Create a Better IAM child tenant under an organization only when a unit needs its own bill. Better IAM grants
nothing downward, so a child tenant needs its own role bindings and, in this release, its own `pools.offer`.
Everything else about the organization's work fits in projects.

## Quotas, flags and residency [#quotas-flags-and-residency]

* **Quotas** live in compute, because Better IAM's tenant limits are a fixed set. They are per tenant, pool and
  resource (`gpu`, `vcpu`, and so on), set by offerings and by `quotas.set` (v0.2, decided on `compute-fleet`),
  and read with `quotas.get`. An admission that would exceed one fails with `QUOTA_EXCEEDED`. See
  [Quotas and usage](/docs/guides/capacity/quotas-and-usage).
* **Flags** such as `compute.h100` are Better IAM feature flags. Policies test them through `tenant.features`,
  and code reads them through `IamPort.featureEnabled`, backed by a cache that holds each tenant and key for up to
  5 minutes and drops entries on `feature:*` events.
* **Residency** waits for Better IAM to expose a tenant region. The offerings table already has an
  `allowed_regions` column, but `pools.offer` does not accept it and nothing enforces it yet. Until then, restrict
  regions in policy with `resource.region`, as the `gpu-user` example does.

## Suspended and deleted tenants [#suspended-and-deleted-tenants]

Compute mirrors the Better IAM tenants that are not active in its own `compute_tenant_states` table, owned by the
built-in `tenancy` module (migration `tenancy/0001_tenant_states`):

```sql title="compute_tenant_states"
CREATE TABLE compute_tenant_states (
  tenant_id text PRIMARY KEY,
  state text NOT NULL,           -- active | suspended | deleted
  reason text,                   -- the IAM action that changed it
  updated_at timestamptz NOT NULL,
  teardown_done_at timestamptz   -- set once every module released a deleted tenant
);
```

### The IAM reaction [#the-iam-reaction]

The rows are written by the `tenant` reaction of the IAM inbox, on `iam:tenants:update` and `iam:tenants:delete`.
The server ships the handler, `tenantStateHandler`, and `better-compute serve` installs it as the default `tenant`
reaction when the configuration's host sets `host.iam`. Give it `host.iam.readTenants` so suspensions and
deletions cascade (anything but a function is `INVALID_CONFIG`). `iamTenantReader` from `@better-compute/iam`
builds one over a Better IAM instance or client:

```ts title="better-compute.config.ts"
import { iamTenantReader } from '@better-compute/iam';

export const host = defineHost(async () => ({
  iam: {
    bridge,
    // The tenant and every tenant below it, with effective statuses, read as root.
    readTenants: iamTenantReader(iam, { credential: rootCredential }),
  },
}));
```

`iamTenantReader(iam, { credential, maxTenants? })` uses only `tenants.get` and `tenants.listChildren`, so `iam` can
be either of:

* **Embedded:** `betterIam()`'s result (`TenantReaderHost`); the credential is the first argument of each call.
* **Remote:** a Better IAM HTTP client, `createIamClient()` from `@better-iam/client` (`TenantReaderClient`); the
  credential goes into each call's headers. A credential IAM refuses fails the read; it never reads as `missing`.

```ts title="Remote mode"
import { createIamClient } from '@better-iam/client';

const readTenants = iamTenantReader(createIamClient({ baseURL: 'https://iam.example.com' }), {
  credential: rootCredential,
});
```

Either way:

* `credential` is a root credential (compute's service account in the IAM root tenant), or a function returning
  one: only root reads suspended and deleted tenants and every tenant's children.
* Each status is the tenant's **effective** one, the worst of its own and its ancestors' (`active` \< `pending` \<
  `suspended` \< `deleted` \< `missing`). Better IAM writes a deletion into every descendant's own status, but a
  suspension only into its decisions, so a child of a suspended tenant reads as `suspended` here.
* A tenant IAM no longer knows is `missing`, and only IAM's own "Tenant not found" answer means that. Any other
  error, a bare 404 from a proxy or a wrong base path included, throws and is retried, so it can never read as a
  deleted tenant. A subtree larger than `maxTenants` (default 10,000) is refused with `INVALID_STATE`.

`better-compute dev --sim` with the embedded Better IAM wires `iamTenantReader` in by itself, so suspending or
deleting a tenant there reaches its whole subtree.

`host.iam.handlers.tenant` replaces the default. A server you build yourself adds the handler to what it gives
`bridge.setHandlers()`, which replaces the whole set:

```ts title="compute.server.ts"
import { iamTenantReader } from '@better-compute/iam';
import { tenantStateHandler } from '@better-compute/server';

bridge.setHandlers({
  ...handlers, // agent, identity, sessions, ...
  tenant: tenantStateHandler(compute, {
    readTenants: iamTenantReader(iam, { credential: rootCredential }),
  }),
});
```

* `readTenants(tenantId)` returns `{ tenantId, status }` for the tenant acted on and its descendants, with `status`
  one of `active`, `pending`, `suspended`, `deleted` or `missing` (IAM no longer knows it). A suspension or
  deletion cascades to descendants, and the audit event names only the tenant acted on, so compute re-reads the
  subtree instead of trusting the event. `suspended` maps to `suspended`; `deleted` maps to `deleted`; anything
  else is `active`. `missing` only confirms a tenant already recorded as suspended or deleted: it never starts a
  teardown by itself (so an id IAM audits under `iam:tenants:update` without being a tenant, such as an
  invitation's, is never torn down). The root tenant is never suspended or deleted, whatever IAM answers.
* Without a reader, only `iam:tenants:delete` is applied, and only to the named tenant; an update changes nothing.
  The handler queues subtree syncs only when it was given a reader, and a sync key that reaches a process without
  one fails with `UNAVAILABLE` and is retried with back-off rather than dropped.
* Each change publishes `tenant.state-changed` (`tenantId`, `state`, `previous`, `reason`) on `tenant/{id}`. A
  replay changes nothing, and a tenant reactivated in IAM becomes `active` again.

### No IAM call in the inbox transaction [#no-iam-call-in-the-inbox-transaction]

The handler runs inside the IAM inbox's drain transaction, which holds its inbox rows `FOR UPDATE`, so it makes no
IAM call there. It only writes SQL:

1. On `iam:tenants:delete`, the named tenant is recorded as `deleted` at once, no IAM read needed, and refused from
   that commit on.
2. On every update and delete, it queues the key `sync:<tenantId>` for the `tenancy` reconciler, and it hands
   `readTenants` to `compute.tenants.setReader()`.

The reconciler then reads the subtree through the reader outside any transaction (`syncTenantStates`) and applies
it in a transaction of its own (reason `iam:tenants:sync`). When IAM cannot answer, the read throws and the key is
retried with back-off: a suspension or deletion is delayed while IAM is down, never dropped, and compute never
fails open. As a backstop, the `tenancy.resync` reconciler, separate so it never delays syncs and teardowns,
re-reads the whole tree from the root tenant every 15 minutes (reason `iam:tenants:resync`); past the reader's size
limit it re-reads only the tenants already refused.

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant B as IAM inbox
  participant H as tenantStateHandler
  participant R as tenancy reconciler
  participant I as Better IAM
  B->>H: iam:tenants:update or delete (in the drain transaction)
  H->>H: delete: record the named tenant as deleted
  H->>R: enqueue sync:tenantId
  R->>I: readTenants(tenantId), outside any transaction
  I-->>R: the subtree with effective statuses
  R->>R: apply the states in its own transaction
  Note over R,I: IAM down: the read throws, the key is retried with back-off"
/>

### What the envelope refuses [#what-the-envelope-refuses]

Every call passes a tenant gate right after the cross-tenant pre-check, before compute's database or IAM decision:

| Tenant state | Reads                    | Mutations                |
| ------------ | ------------------------ | ------------------------ |
| `active`     | allowed                  | allowed                  |
| `suspended`  | allowed                  | `TENANT_SUSPENDED` (403) |
| `deleted`    | `TENANT_SUSPENDED` (403) | `TENANT_SUSPENDED` (403) |

The gate checks the caller's tenant and the tenants the call acts in; the error's `details` carry `tenantId` and
`state`. Root-tenant callers may still act in a suspended tenant and read a deleted one, but never write into a
deleted tenant, since what they created after the teardown would never be torn down. The gate answers from memory:
each process loads the non-active states at start, or on its first call when it never ran `initialize()` (an
API-only replica started with `--no-migrate`; `compute.tenants.ready()` loads it up front), and reloads them every
5 seconds. The process that applied a change reloads before its next call (`compute.tenants.invalidate()`, a
generation counter, so a load already in flight cannot swallow it). Another process may therefore admit a suspended
tenant's mutation for up to 5 seconds. Operations built without a deployment (unit harnesses) get `noTenantGate`,
which refuses nothing. A reload that fails is reported once per run of failures,
not every 5 seconds, and the timer stops once the store is closed (a one-shot CLI command closes it without
stopping compute).

### Teardown of a deleted tenant [#teardown-of-a-deleted-tenant]

A deletion enqueues the `tenancy` reconciler for the tenant. It calls every hosted module's `tenantDeleted` hook,
plugins included; a module that answers `SYNC_AGAIN` is asked again by the 30-second `sweep`, which also recovers
a lost work item and retries the longest-waiting teardowns first (`teardown_attempted_at`, migration
`tenancy/0002_teardown_attempts`). When every hook answers `DONE` in one pass, the reconciler sets
`teardown_done_at` and publishes `tenant.torn-down`. The root tenant is never torn down, whatever a row says.

| Module      | What it releases (idempotent, in batches of 100)                                                                                                                                                                                                                                                                                                                                     |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `capacity`  | Unstarted reservations cancelled, started ones released now (bookings and quota back); offers to the tenant withdrawn (quotas zeroed, `pool.unoffered` to the owner); pools it owns archived, with every offer withdrawn (`pool.archived`), while other tenants' live bookings there run to their end; its idle quota rows deleted. A dedicated-root lease is released this way too. |
| `provision` | Active workflows aborted (their cleanup still runs); private images and profiles soft deleted with their authz rows; image signers revoked. Images it promoted to public stay, for the tenants that deploy them.                                                                                                                                                                     |
| `metal`     | Every bare-metal node it owns retired, one legal transition per sweep: `ready`, `adopted`, `enrolling` and `failed` retire at once; `deployed` and `rescued` machines are released (wiped) first; a machine mid-workflow is aborted (it fails with `TENANT_DELETED`), then retired; `releasing` and `retiring` machines finish first.                                                |
| `vps`       | Admission drafts dropped; every live instance set to `desiredState: 'deleted'` (100 per pass, `instance.deleting` with reason `tenant deleted`) and torn down through the ordinary lifecycle: runtime teardown, inline allocation released, IP freed, quota refunded. `SYNC_AGAIN` until every instance is `deleted`.                                                                |

> **Not torn down yet.** 
  Only `capacity`, `provision`, `metal` and `vps` implement `tenantDeleted` in this release. A deleted tenant's
  non-metal `tenant-owned` nodes and its sites are not released by the teardown; the gate only stops the tenant from
  reaching them. While its instances wait for teardown, their VMs cannot mint workload identity tokens (the workload
  broker refuses a suspended or deleted tenant's instances with `disabled`), and the secret broker releases none of
  the tenant's secrets or user data (`security.secret.refused`, reason `tenant-suspended` or `tenant-deleted`). Plugins release their own state through the same hook; see
  [modules](/docs/extending/modules#tenantdeleted).

  - [Actions and resources](/docs/guides/iam/actions-and-resources): Scope types, cross-tenant refusal and list filtering.

  - [Pools](/docs/guides/capacity/pools): Pool classes, sharing modes and offerings.

  - [Billing and usage](/docs/guides/iam/billing-and-usage): How usage is attributed to tenants, identities and projects.


# Attestation (/docs/guides/nodes/attestation)

> The attestors that admit machines (join-token, x509pop, provisioned, discovery, cloud-iid, tpm), what each proves, single-use evidence and two-round joins.



A join token proves that someone with access to your API wanted a machine to join. That is enough for a server
you install by hand, but not for a machine that PXE-boots on its own, a cloud VM, or a server whose certificate
expired over a long weekend. An attestor answers the question for one kind of
evidence: is this machine allowed to join, and as which node?

Each attestor is a `@better-compute/core` `AttestorVerifier` that runs inside JoinService's join transaction.
`JoinRequest.attestor` names it (empty means `join-token`), `JoinRequest.attestation` carries its evidence, and a
sealed token, when the request has one, is opened and handed over as `evidence.token`. Every refusal reaches the
agent as the same `ACCESS_DENIED`; the reason goes only to the `onRefusal` hook and the `security.join.refused`
event (see [joining](/docs/guides/nodes/joining)).

## The attestors [#the-attestors]

| Attestor      | Phase | Roles                 | Evidence                                                                                                               | Proves                                                                                       | Joins as                                     |
| ------------- | ----- | --------------------- | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `join-token`  | v0.1  | `node`, `site`        | a sealed `node` or `site` token                                                                                        | the holder of a live token for this pool or site                                             | a new node, or a reclaim with `allowReclaim` |
| `x509pop`     | v0.2  | `node`, `site`        | its last certificate chain and a proof by that key                                                                     | the machine still holds the key of a recent certificate                                      | the same node, in place                      |
| `provisioned` | v0.2  | `node`                | a sealed single-use `provisioned` token and a product UUID claim                                                       | the OS a deploy or provider bootstrap installed                                              | the target node re-keyed, or a new node      |
| `discovery`   | v0.2  | `ramdisk`             | a sealed single-use `discovery` token and a signed discovery assertion                                                 | the ramdisk the site's boot server just booted                                               | a ramdisk identity of at most 2 hours        |
| `cloud-iid`   | v0.3  | `node` (configurable) | a provider-signed instance identity                                                                                    | the cloud instance your provider account created                                             | whatever `resolve` answers                   |
| `tpm`         | v0.5  | `node` (configurable) | TPM endorsement and attestation keys, then an activated credential, a quote and the AK's certification of the node key | the TPM holding that endorsement key, and (by default) that the same TPM holds the CSR's key | the node bound to that key                   |

Only `join-token` is built into the server. You add the others, and one with id `join-token` replaces the
built-in:

```ts title="compute.ts"
import { betterCompute, pkiJoinCrypto } from 'better-compute/server';
import {
  discoveryAttestor,
  provisionedAttestor,
  x509popAttestor,
} from 'better-compute/pki';
import {
  discoveryWorkflowNodeResolver,
  provisionedTargetResolver,
} from 'better-compute/provision';

// `bundle`: the PKI bundle `better-compute pki init` wrote, loaded with importPkiBundle(text, { secret }).
export const compute = betterCompute({
  // database, secret, baseURL, iam, modules, …
  pki: {
    nodeCa: bundle.nodeCa,
    join: pkiJoinCrypto({
      trustDomain: bundle.trustDomain,
      nodeCa: bundle.nodeCa,
      signingKey: bundle.joinSigning,
      encryptionKey: bundle.joinEncryption.privateKey,
    }),
  },
  attestors: [
    x509popAttestor(),
    provisionedAttestor({ resolveTarget: provisionedTargetResolver() }),
    discoveryAttestor({
      resolveWorkflowNode: discoveryWorkflowNodeResolver(),
      approval: 'unknown-machines',
    }),
  ],
});
```

The attestors from `@better-compute/pki` also take `onRefusal(refusal)`, called with
`{ attestor, reason, tokenId?, nodeId?, tenantId? }` for your own audit or metrics.

## join-token [#join-token]

The v0.1 attestor, described in [joining](/docs/guides/nodes/joining). It locks the `join_tokens` row, checks
the secret in constant time, revocation, expiry at `compute_now()`, the remaining uses and the role (`node`
tokens admit `node`, `site` tokens admit `site`), and consumes one use. JoinService enforces the token's
`expect` binding and reclaim flag for every token-based attestor, not only this one.

## x509pop [#x509pop]

A node's certificate lasts 24 hours. A machine that was powered off or cut off longer than that comes back with
an expired certificate, and renewal refuses expired certificates. `x509pop` lets it re-join as itself: it
presents its last certificate chain and a proof by the certificate's key, bound to its new CSR.

The evidence is UTF-8 JSON:

```json title="JoinRequest.attestation"
{
  "v": 1,
  "nodeId": "node_…",
  "chain": ["<base64url DER leaf>", "<base64url DER issuer>"],
  "clientTime": 1790000000000,
  "sig": "<base64url r‖s>"
}
```

`sig` is an ES256 signature under purpose `x509pop` over the CSR's SHA-256, the node id and the client time, so
neither a Renew proof nor an earlier `x509pop` proof can be replayed as one. The attestor refuses unless:

* the leaf is byte for byte a `node_certs` row of that node, of its current identity generation, and not revoked;
* the node is not `retired`, and the certificate's role is the requested one (`node` or `site`; a lessee's
  `node-tenant` certificate re-joins as `node` and JoinService restores the restriction);
* the certificate expired at most `graceMs` ago (7 days by default);
* `clientTime` is within `windowMs` (60 seconds) of database time, and the proof was not used before
  (`replayCache`, in memory by default, 10,000 entries).

The result names the node, so JoinService reclaims it in place: it raises the identity generation, revokes every
older certificate and keeps the node's state. A revoked certificate never re-joins; neither does one from before
`nodes.revokeIdentity`. Agents build the evidence with `createX509popEvidence` (or `x509popProducer` for
`FakeDaemon`). bcd does not produce this evidence yet: it joins with `join-token` only and answers no attestor
challenge, so a bcd node whose certificate expired offline cannot re-join this way yet.

## provisioned [#provisioned]

When Better Compute deploys an operating system, the deploy renders a single-use `provisioned` token into the
machine's user data (the provisioning engine's `userdata.render` step). The installed system joins with it.
The attestor requires:

* a token of role `provisioned` with exactly one use left (`not-single-use` otherwise);
* a product UUID in the machine's claims (`no-product-uuid`);
* a binding to one machine: the token's `expect.productUuid`, or a target the host resolves (`unbound`).

`resolveTarget(tx, token)` answers with the node the token was minted for and its expected product UUID.
`provisionedTargetResolver()` from `@better-compute/provision` finds the running workflow whose `userdata.render`
step minted the token, and reads the product UUID from that node's newest inventory. With a target node, the
join re-keys that node in place and keeps its provisioning state; a UUID that differs is
`product-uuid-mismatch`. Without one, the machine joins as a new node, which is what a provider bootstrap such as
Hetzner Robot's installimage does.

## discovery [#discovery]

During commissioning, a bare-metal server PXE-boots a platform ramdisk before anything is installed on it. The
ramdisk has no identity of its own, so two parties vouch for it: the iPXE script carries a single-use
`discovery` token bound to the MAC, and the site's boot server signs a statement that it just served that MAC.

The evidence is the sealed token plus a serialized `SignedDiscoveryAssertion` (`bettercompute.site.v1`). The
assertion names the site, the vouching site agent's node id, the boot lease epoch it held, the MAC, and
optionally the IP, product UUID and workflow id, with `issued_at` and `not_after`. The attestor checks:

* the token: role `discovery`, one use left, issued for the assertion's site, and its `expect.mac` (when set)
  equal to the asserted MAC;
* the signer: a `site` node of that site, not retired, currently holding the `site:<id>:boot` lease at exactly the
  asserted epoch, so a fenced boot server can no longer vouch; the signature (purpose `discovery-assertion`) must
  verify against its unexpired certificate;
* freshness: the assertion is valid until the earlier of its `not_after` and 15 minutes after `issued_at`
  (`maxAgeMs`);
* the machine: its claimed MACs include the asserted MAC, and its product UUID matches when the assertion names
  one.

Each signed assertion admits one join (`assertion-reused` otherwise), and the token is spent. The resulting
certificate is a `ramdisk` certificate of at most 2 hours (`certTtlSeconds` may shorten it) that allows only the
commissioning command kinds, `discoveryAllowedKinds` by default: `inventory.refresh`, `logs.collect`, `diag.*`,
`disk.wipe`, `deploy.*` and `boot.kexec`, which are exactly the registered kinds whose roles include `ramdisk`
(`tests/pki-attestors.test.ts` checks the two agree). Pass `allowedKinds` to change the list.

| Option                                | Default   | Effect                                                                                                                                                   |
| ------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `resolveWorkflowNode(tx, workflowId)` | none      | A known machine re-commissions as its node (`discoveryWorkflowNodeResolver()` answers for running workflows). Without it every discovery creates a node. |
| `approval`                            | `'never'` | `'unknown-machines'` defers machines no workflow names; `'always'` defers every one.                                                                     |
| `hardwareKey(claims)`                 | none      | The trusted hardware key requests are keyed on; without one they key on site and MAC.                                                                    |
| `pendingTtlSeconds`                   | 86,400    | How long a pending request waits for a decision.                                                                                                         |
| `retryAfterSeconds`                   | 30        | When the ramdisk retries a deferred join.                                                                                                                |

### Both halves in bcd [#both-halves-in-bcd]

bcd produces this evidence on both sides. `bcd site` signs one assertion for every iPXE script its boot server
renders, while it holds the boot lease: the site id, its own node id, the lease epoch, the MAC, the IP (from iPXE's
query, else the connection), the product UUID (from iPXE, else the boot entry), the workflow, `issued_at` in server
time, and a `not_after` of at most 15 minutes and never past the boot entry's, signed by its node key under purpose
`discovery-assertion`. It fetches the machine's discovery token with a `SecretRequest` bound to the workflow and puts
both on the ramdisk's kernel command line (`bc.token`, `bc.assertion`). `bcd ramdisk` reads them back, seals the
token to the pinned JoinService key and sends `Join` with attestor `discovery`, role `ramdisk`, the serialized
`SignedDiscoveryAssertion` as the attestation, and its hardware claims (MACs, product UUID). A deferral is retried
with the same CSR. The Go tests check the assertion's fields and signature, and the TypeScript vectors pin the
signing input, but no test yet joins a real `bcd ramdisk` through this attestor. See
[the boot server](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site) and
[the ramdisk](/docs/daemon/install#the-ramdisk).

## cloud-iid [#cloud-iid]

A cloud VM that your provider account created can prove it with the identity document its cloud signs, so it
needs no token at all. `cloudIidAttestor({ providers, resolve, roles? })` verifies the document with one of its
providers, then asks `resolve(tx, identity, { role, claims })` where the instance joins. `resolve` returns an
attestation result (tenant, pool, site, and `nodeId` for a known machine) or `undefined` to refuse
(`unknown-instance`), for example for an instance in an account you do not manage.

```json title="JoinRequest.attestation"
{ "provider": "gcp", "document": "<identity token JWT>" }
```

| Provider                                                                 | Verifies                                                                                                                                                                         | Bound to the CSR                                                                                                   |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `gcpInstanceIdentity({ audience, fetch or jwks, projects?, maxAgeMs? })` | A Compute Engine identity token with `format=full`, signed by Google (RS256 or ES256, JWKS cached an hour). `projects` limits the accepted project ids.                          | Yes: the agent requests audience `cloudIidAudience(audience, sha256(csr))`, that is `<audience>#<base64url hash>`. |
| `awsInstanceIdentity({ certificates, maxAgeMs?, accounts? })`            | The EC2 identity document and its `signature`, checked with the region's AWS certificate, which you supply (`'*'` covers regions not listed). `accounts` limits the account ids. | No: accepted only within `maxAgeMs` (30 minutes) of the instance's `pendingTime`.                                  |
| `oidcInstanceIdentity({ id, issuers, jwks, audience, identity })`        | Any issuer that signs JWTs into a JWKS; `identity(claims)` maps the claims to account and instance.                                                                              | Yes, through the audience.                                                                                         |

Identity tokens older than `maxAgeMs` (10 minutes by default for OIDC and GCP) are refused. Whatever the
provider, each (provider, account, instance) admits exactly one join, ever (`instance-reused`): an AWS document
is readable by any process on the VM, and even a bound token must not mint a second node. A known instance that
lost its certificate re-joins with `x509pop`.

## tpm [#tpm]

`tpm` (v0.5) ties a node to its TPM's endorsement key (EK), an identity that survives a reinstall or a disk swap.
It follows the go-attestation enrollment flow as a two-round join.

1. **Challenge.** The agent sends its EK (public area and manufacturer certificate) and an attestation key (AK).
   The attestor checks the EK certificate chains to one of `ekRoots` and that the AK has attestation-key
   attributes, then answers with a `MakeCredential` challenge plus an opaque state sealed under `challengeKey`
   (AES-256-GCM, bound to the CSR, at most 5 minutes). It writes no row, so any `api` replica can finish the join.
2. **Answer.** Only the TPM holding that EK, with that AK loaded, can activate the credential. The agent returns
   the secret and a quote of its PCRs whose `extraData` is `sha256(secret ‖ sha256(csr))`. The attestor checks
   the secret, the quote signature, the nonce, that the PCR values match the quote's digest, and your
   `pcrPolicy`.
3. **Node key** (cf24876). The same answer carries `nodeKey { public, certifyInfo, signature }`: a `TPM2_Certify`
   of the node key by the AK, qualified by `sha256(csr)`. The attestor checks that the certification is signed by
   the AK it just proved beside the EK, that it is a `TPM_ST_ATTEST_CERTIFY` whose `extraData` is `sha256(csr)`,
   that the certified name is the TPM name of `public`, that the key was made in the TPM and can never leave it
   (`fixedTPM`, `fixedParent`, `sensitiveDataOrigin` and `sign` set, `decrypt` clear), and that its
   SubjectPublicKeyInfo is the CSR's (JoinService passes its SHA-256 as `csrSpkiSha256`). A certification that
   fails a check is refused with `node-key-malformed`, `node-key-signature`, `node-key-nonce`, `node-key-name`,
   `node-key-attributes` or `node-key-not-csr`. With the default `nodeKey: 'certified'`, an answer without one is
   refused too (`node-key-not-certified`); `nodeKey: 'any'` admits it, and the result then makes no claim about
   the key.

It then decides where the machine joins, keyed by the SHA-256 of the EK: the node that key is already bound to
(a reclaim), else your `resolve` allow-list (for example a vendor's delivery manifest), else operator
`enrollment` through [join requests](/docs/guides/nodes/joining). JoinService binds the EK to the node
(`node_bindings`): a later join proving it reclaims that node, and it can never admit another one
(`bound-elsewhere`). When the node key was certified, the result carries `nodeKey: 'tpm'`, and JoinService records
it in the `node.joined` event and in the join's audit row (`nodeKey`: `tpm`, or `unattested` for every other join;
see [joining](/docs/guides/nodes/joining#the-join-flow)). Since d94f1f3 the result also carries `nodeKeyUri`, the AK's
name as `urn:bc:tpm-ak:<nameAlg>:<digest>`, which JoinService writes into the certificate so that every renewal must
be certified by the same AK (see [renewing a TPM-held key](#renewing-a-tpm-held-key)).

| Option                            | Meaning                                                                                                                                                                                    |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `challengeKey`                    | 32 bytes shared by every `api` replica; seals the challenge state. Required.                                                                                                               |
| `ekRoots`                         | Manufacturer EK CA certificates (PEM or DER).                                                                                                                                              |
| `allowUncertifiedEk`              | Accept an EK without a certificate (swtpm, labs). Never in production.                                                                                                                     |
| `pcrSelection`                    | PCRs the quote must cover; default the SHA-256 bank, PCRs 0 to 7.                                                                                                                          |
| `pcrPolicy(pcrs)`                 | Returns a refusal reason, or `undefined` to accept.                                                                                                                                        |
| `resolve(tx, identity, { role })` | Where an unbound EK joins, or `undefined`.                                                                                                                                                 |
| `enrollment`                      | `{ tenantId, siteId, poolId?, pendingTtlSeconds?, retryAfterSeconds? }`: defer unknown EKs for approval.                                                                                   |
| `challengeTtlMs`                  | Challenge lifetime, at most 5 minutes.                                                                                                                                                     |
| `nodeKey`                         | `'certified'` (the default): admit a join only when the AK certifies the CSR's key as one this TPM made and never lets out. `'any'`: also admit a node key held elsewhere, such as a file. |

> **nodeKey defaults to certified.** 
  Since cf24876 `nodeKey: 'certified'` is the default, so a `tpm` join whose agent keeps its node key outside the TPM
  is refused (`node-key-not-certified`), including a machine that joined with a file key before and joins again. Set
  `nodeKey: 'any'` to keep admitting those; their certificates then name no AK and renew without a TPM proof.

### Renewing a TPM-held key [#renewing-a-tpm-held-key]

The join proves the first node key. Since d94f1f3 every renewal proves the next one the same way, so a certificate
issued after a certified `tpm` join names, for its whole life and every renewal after it, a key that cannot leave
that TPM.

1. **The statement.** A certified join returns `nodeKeyUri` (`AttestationResult.nodeKeyUri`), and JoinService
   issues the certificate with two URI SANs: the SPIFFE ID first, then
   `urn:bc:tpm-ak:<nameAlg>:<digest>`, the AK's TPM name in hex (`tpmAkUri`; the name covers the AK's whole public
   area, so only that key matches it). A join without a certified key (any other attestor, or `nodeKey: 'any'`
   without a certification) gets no statement and renews as before. Since 0e42018 JoinService writes a statement
   only when renewals can prove it again (`JoinCrypto.knowsKeyStatement`; `pkiJoinCrypto` knows `urn:bc:tpm-ak:`
   alone): a join whose attestor returns any other `nodeKeyUri` is refused with `node-key-statement`
   (`security.join.refused` reason `join:node-key-statement`) rather than issued a certificate that could never
   renew.
2. **Where Renew reads it.** From the presented leaf, which must already be byte for byte compute's own
   `node_certs` row, never from the CSR. A CSR that asks for no SAN at all still gets the statement back, and a CSR
   cannot remove it.
3. **The proof.** A leaf with a `urn:bc:tpm-ak:` statement renews only with `RenewRequest.key_attestation`
   (field 5, `TpmKeyAttestation`): the AK's public area (`ak_public`), the new key's public area (`key_public`),
   and the AK's `TPM2_Certify` of that key (`certify_info`, `signature`) qualified by `sha256(csr)`.
   `verifyTpmKeyAttestation` checks that the AK's TPM name is the one in the statement and that it has
   attestation-key attributes, then runs the same checks as the join's node-key proof: the AK's signature, the
   qualifying data, the certified name, `fixedTPM`, `fixedParent`, `sensitiveDataOrigin` and a signing key, and
   that the key's SPKI is the CSR's.
4. **No downgrade.** A missing or failing attestation refuses the renewal with `ACCESS_DENIED` ("Renew refused"),
   and JoinService logs `renew refused` with the node id and the reason. Since 0e42018 it also publishes
   `security.renew.refused` to the node owner, naming the node (see
   [refused renewals](/docs/operations/security#refused-renewals)). The new certificate carries the statement
   forward, so the next renewal needs the AK again.

| Reason                                                                                                                   | Meaning                                                                                                          |
| ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `key-attestation-missing`                                                                                                | The leaf names an AK and the request has no `key_attestation`                                                    |
| `key-attestation-ak`                                                                                                     | `ak_public` is not the named AK (another TPM's), or is not an attestation key                                    |
| `node-key-malformed`, `node-key-signature`, `node-key-nonce`, `node-key-name`, `node-key-attributes`, `node-key-not-csr` | The certification fails the same check as at join (a replayed certification for another CSR is `node-key-nonce`) |
| `tpm-ak-uri`                                                                                                             | The statement in the leaf is malformed                                                                           |
| `key-statement-unknown`                                                                                                  | The leaf carries a `urn:bc:` statement this control plane does not know                                          |
| `key-statement-unsupported`                                                                                              | The `JoinCrypto` has no `verifyKeyStatements` (`pkiJoinCrypto` has one)                                          |

The node certificate's URI SANs are now exactly one SPIFFE ID followed only by `urn:bc:` statements (at most four
URIs in all). The identity readers (`nodeSpiffeIdOf` in `@better-compute/pki`, used for mTLS identity, proofs of
possession, join responses and the gateway's renew relay) and bcd's `pki.NodeIdentity` take the node id from the
one SPIFFE ID and never from a statement; see [PKI](/docs/operations/pki#trust-domain-and-spiffe-ids).

**bcd's side.** At each renewal `renewOnce` reads the current leaf. When it names an AK, bcd loads that AK alone
(`tpm.Certifier`, a primary of the endorsement hierarchy, no EK), checks that its TPM name is the one the certificate
names, derives the next node key in the same TPM connection, and has the AK certify it over `sha256(csr)` into
`key_attestation`. A root `bcd node` does this on the TPM it opened; an unprivileged one under `[helper]` asks
`bcd helper` (`tpm.ak`, `tpm.nodekey`, `tpm.certify`; 8870b8d), which holds the TPM connection, and checks that the
helper certified with the AK it announced (see [the node key in the TPM](#the-node-key-in-the-tpm)). There is no fallback: where a certificate without a statement would get a file
key with a warning, this one fails the renewal instead and sends nothing.

| At renewal                                  | What bcd does                                                                                                                                                                                                                  |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| The TPM cannot be opened                    | `renewal: the certificate's node key is in the TPM, which is unavailable: ...`                                                                                                                                                 |
| The TPM derives another AK                  | `renewal: this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join` (a1c1a46) |
| The TPM cannot make or certify the next key | `renewal: the next node key in the TPM: ...` or `renewal: the TPM's attestation key certifying the next key: ...`                                                                                                              |

Each failure is logged as `renewal failed` with the attempt and `expiresIn`, and retried with backoff from
30 seconds to 10 minutes. Retries do not help a TPM that lost the AK or the key: the certificate (at most
24 hours) runs out, and the node goes offline.

#### Recovering a node whose TPM changed [#recovering-a-node-whose-tpm-changed]

A `tpm`-joined node recovers by joining again with its TPM. `bcd join` alone joins only when the old identity no
longer loads; one that still loads answers `already joined as <node>` whatever state its certificate is in. Since
a1c1a46 `bcd join --reset --yes` moves the old identity aside first, so the recovery command is the same whichever
case you are in:

```bash
# as root on the machine
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm
```

`--reset` renames `identity.json`, `node.key` and a pending `node.key.next` to `<name>.reset-<UTC time>` in the data
directory (nothing is deleted) and the join then runs as on a new machine; see
[joining again with --reset](/docs/daemon/install#joining-again-with---reset) for its guards. The attestor keys the
machine on the SHA-256 of its EK, so what happens depends on whether the EK survived:

* **The EK is the same** (for example after `TPM2_Clear`, which resets the owner hierarchy's seed but not the
  endorsement seed the EK and AK come from). The node key is a primary of the owner hierarchy, so the same
  `node.key` record now derives another key, and bcd no longer loads its identity: "the node key does not match the
  node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset
  \--yes, then bcd join" (since a1c1a46 this hint is on the key mismatch, where a clear shows; the AK survives a
  clear). The join finds no usable identity, and the attestor reclaims the node that EK is bound to: the same node
  id, a new identity generation (every older certificate revoked), a new certified key and a fresh AK statement.
  Here a plain `bcd join --attestor tpm` would work too, since the identity no longer loads; `--reset` keeps the old
  files instead of overwriting them.
* **The EK is new** (another TPM, a replaced board, a reset endorsement hierarchy). The renewal's AK check fails
  ("another TPM, or its endorsement seed changed"). The attestor does not know the EK, but the old node still
  exists: it holds the binding to the old EK and, usually, the machine's trusted hardware key, and a join whose
  hardware key another node holds is refused (`reclaim`), since a `tpm` join carries no `allowReclaim`. Remove the
  old node first, then join again:
  1. cordon the old node (`better-compute nodes cordon`, `nodes.cordon`) and wait until it holds no open
     allocation;
  2. delete it (`better-compute nodes delete <node> --yes`, 4049621; `nodes.delete`, which since 54cad9e accepts a
     cordoned `adopted` node): its certificates are revoked and its EK binding and trusted hardware key are freed
     (see [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host));
  3. on the machine, as root, run `bcd join --reset --yes --attestor tpm`. The machine joins wherever your
     `resolve` or `enrollment` sends it, as a new node (with neither, it is refused). It has a new node id; the
     `--reset` moved the old node's `journal.db` aside with the identity (b4fcc2e), so `bcd node` starts with a
     journal of its own.

`bcd join --reset` without `--yes` moves nothing: it prints what it would move and exits 2. The hints bcd prints
spell the recovery as two runs, `bcd join --reset --yes`, then `bcd join`; one run with both does the same, because
the reset happens before the join reads any credential. If the join after a reset fails, the identity stays moved
aside and a later `bcd join` joins.

A node that joined with a token has no statement, so its renewal does not depend on the TPM's AK. Its node key, if
in the TPM, still needs that TPM to load: after a clear it joins again with a token, and a token join of a known
machine needs `allowReclaim` (see [placeholder hardware and reclaims](/docs/guides/nodes/joining#placeholder-hardware-and-reclaims)).

### bcd with its TPM [#bcd-with-its-tpm]

`tpmProducer(SoftwareTpm)` is the agent side in TypeScript tests. bcd has its own, `internal/tpm` (f61b32d): a pure-Go
TPM 2.0 client over the kernel resource manager (`/dev/tpmrm0`, Linux only) that derives the default RSA-2048 EK and a
restricted P-256 AK and builds this attestor's `{ ek, ak }` evidence and its round-2 answer (the
`ActivateCredential` secret and a quote with `extraData = sha256(secret || sha256(csr))`), with a software TPM,
`tpmsim`, for tests. Since 8545688 `bcd join --attestor tpm` uses it:

```bash
bcd join --config /etc/bcd/config.toml --attestor tpm
```

* **The TPM is the credential.** No token is read or sent: `BCD_JOIN_TOKEN` is dropped from the environment, and
  `--token-file` with `--attestor tpm` is a usage error. Where the machine joins is the attestor's decision (the
  bound node, `resolve` or `enrollment` above).
* **Which TPM.** A host build opens `/dev/tpmrm0`; a fakehw build keeps a software TPM in
  `<data_dir>/fakehw-tpm.json`, the same EK on every join of that machine.
* **The rounds.** `agent.Join` sends the evidence, answers up to 3 challenge rounds for the same CSR, and waits out
  a pending approval (`enrollment`) before a fresh challenge round. A machine that already joined answers
  `already joined as <node>` without touching the TPM, unless `--reset --yes` moved its identity aside first
  ([recovering a node whose TPM changed](#recovering-a-node-whose-tpm-changed)).
* **The node key, certified** (cf24876). When node keys can use a TPM on this host (below), `agent.Join` makes the
  join's node key in the attestor's own TPM (`tpm.Attestor.HoldNodeKey`, through the `NodeKeyHolder` interface),
  and the round-2 answer carries the AK's `TPM2_Certify` of it over `sha256(csr)`. If the TPM cannot make that key,
  the join stops with an error rather than falling back to a file. When node keys cannot use the TPM (a fakehw
  build without `BCD_FAKEHW_TPM_KEYS=1`), the key is a file, the answer carries no certification, and an attestor
  left at `nodeKey: 'certified'` refuses the join. `bcd join` runs as root and opens the TPM itself, `[helper]` or
  not; since 8870b8d it no longer checks whether the helper's user can open `/dev/tpmrm0`, because that user never
  does (below).
* **The install script.** [`daemon install-script --attestor tpm`](/docs/daemon/install#generate-the-script)
  (a006d9f) writes a script that reads no token, fails early without `/dev/tpmrm0`, and runs
  `bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null` before (re)starting the units, so the same
  script installs and upgrades.

### The node key in the TPM [#the-node-key-in-the-tpm]

Independent of the attestor, bcd keeps its node key in the host's TPM when there is one (965c14d, 50e18f9,
0ba2a79). The node key signs the mTLS handshake, CSRs, renewal proofs, node statements (attested reports, command
results, discovery assertions, secret and workload-token requests) and the WebSocket proof; in bcd it is a Go
`crypto.Signer`, whatever holds it, and the wire does not change: the same P1363 signatures and the same `kid`
(SHA-256 of the key's SPKI).

* **The key.** `tpm.NodeKeyTemplate` is an unrestricted P-256 signing key (`fixedTPM`, `fixedParent`,
  `sensitiveDataOrigin`, `userWithAuth`, `noDA`, `sign`), a primary of the owner hierarchy derived from a 32-byte
  per-key unique. Nothing is stored in or evicted to the TPM: bcd keeps the unique and derives the same key again at
  every load. Each signature uses ECDSA with the digest's hash (SHA-256, -384 or -512, for TLS).
* **On disk.** `node.key` (and `node.key.next` during a renewal) then holds only the record: a PEM block of type
  `BCD TPM NODE KEY` whose bytes are the unique, mode 0600. A key in the TPM does not load without that TPM.
* **When.** Every new node key is made in the TPM when there is one: at `bcd join` and at every renewal's next
  key (under `[helper]`, in the helper's TPM connection). A host build uses `/dev/tpmrm0` when it exists; a fakehw build uses its software TPM only with
  `BCD_FAKEHW_TPM_KEYS=1`. When the TPM cannot hold the key, bcd logs a warning and makes a PKCS#8 file key instead,
  except at the renewal of a certificate that names an AK, which fails rather than leave the TPM (see
  [renewing a TPM-held key](#renewing-a-tpm-held-key)). A node that joined with a file key moves into the TPM at its
  next renewal.
* **Under `[helper]`** (8870b8d, security review 3). The unprivileged `bcd node` never opens the TPM. The kernel's
  resource manager filters no command, so whoever can open `/dev/tpmrm0` can change the hierarchies' authorization,
  clear the TPM through its (empty) lockout authorization or trip its dictionary-attack lockout, breaking root's
  TPM-sealed secrets (LUKS keys) and attestation. `bcd join` and `bcd helper` run as root and open the TPM
  themselves; an unprivileged `bcd node` reaches its node keys only through the helper's `tpm.*` operations,
  which name a key by its 32-byte unique, and checks every signature the helper returns against the key's public
  half. The helper builds every TPM command itself, so the node picks no template, handle, hierarchy or
  authorization (see [node keys through the helper](/docs/daemon#node-keys-through-the-helper)). Install scripts
  from 0ba2a79 until 8870b8d put the `bcd` user in the `tss` group instead; an upgrade run takes it out again (see
  [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)).
* **What it protects.** Such a key cannot be copied off the machine, but anyone with the owner hierarchy's
  authorization on that TPM (root on the node) can derive and use it: non-exportable, not root-proof.
* **Seeing it.** `bcd doctor` has a `node key` check: `ok` "in the TPM", `ok` for a file key on a host with no TPM,
  and `warn` for a file key on a host with a TPM ("the next renewal moves it into the TPM (through bcd helper when
  bcd node runs unprivileged)"). Under `[helper]` its `tpm` check (e6c3f53) fails while the node's user is still in
  the `tss` group. `bcd status` prints
  `node key in the TPM` or `node key a file` on its identity line, and its JSON carries `nodeKey` (`tpm` or `file`).

What the control plane knows about it depends on how the key was made:

| When                                                  | Proven to the control plane                                                                                                                                                                    | Where it shows                                                                                                                       |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| A `tpm` join (cf24876)                                | Yes: the AK certifies the key (`CertifyNodeKey`, `TPM2_Certify` over `sha256(csr)`) and `tpmAttestor` checks it, refusing the join under the default `nodeKey: 'certified'` when it is missing | `nodeKey: 'tpm'` in `node.joined` and the `node.joined` or `node.reclaimed` audit row; `urn:bc:tpm-ak:` in the certificate (d94f1f3) |
| A join with any other attestor                        | No: bcd still makes the key in the TPM when it can, but sends no certification                                                                                                                 | `nodeKey: 'unattested'`; no statement in the certificate                                                                             |
| A renewal of a certificate that names an AK (d94f1f3) | Yes: the same AK certifies the next key (`RenewRequest.key_attestation`), and JoinService refuses the renewal without it                                                                       | the statement, carried into the new certificate and its `node_certs` row                                                             |
| A renewal of any other certificate                    | No: the next key is made in the TPM when bcd can open one, but nothing proves it                                                                                                               | nothing; the certificate and `node_certs` do not record where the key lives                                                          |

So for a certified `tpm` join the proof lasts: every certificate the node holds from then on names a key that its
TPM made and certified. For every other node the control plane has only bcd's word (`bcd status` and `bcd doctor`
report it locally).

What is tested, and with what TPM:

| Test                                                                      | TPM                                                                                                                                                                      | What it proves                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `TestTSContract` (`daemon/internal/tpm`)                                  | `tpmsim` fixture, with a manufacturer EK certificate in NV (b973e05)                                                                                                     | bcd's round-1 evidence is still byte for byte the fixture's, including the EK certificate bcd reads from NV index `0x01C00002` (`tpm.EKCertIndex`) and sends as `ekCert`, and the challenge `tpmAttestor` made for it activates and quotes through bcd's client (TypeScript to Go); since cf24876 the answer also carries the AK's certification of a node key held in the sim (`answer.nodeKey`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `tests/pki-attestors-tpm.test.ts`                                         | `tpmProducer` over a `SoftwareTpm` whose `nodeKey` it certifies as bcd does                                                                                              | under `nodeKey: 'certified'`, a certified CSR key is admitted with `nodeKey: 'tpm'`; no certification is `node-key-not-certified`, and a certified key other than the CSR's is `node-key-not-csr`; `nodeKey: 'any'` admits the uncertified answer without a `nodeKey` claim                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `tests/pki-attestors-tpm-bcd.test.ts`                                     | the same fixture                                                                                                                                                         | `tpmAttestor` with `ekRoots: [ek-root.der]` and no `allowUncertifiedEk` verifies the EK certificate's chain, challenges bcd's evidence and admits bcd's answer, with its node key certified (`nodeKey: 'tpm'`), while a CSR key other than the certified one is `node-key-not-csr`; it refuses the EK under another root (`ek-certificate-chain`), and the answer for another CSR (`challenge-for-another-csr`) or after the challenge expired (`challenge-expired`) (Go to TypeScript)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `tests/e2e/bcd-tpm.test.ts` (`BCD_E2E=1`, WSL or Linux; db9bd46, 8a9af54) | `tpmsim` in a fakehw bcd, with an EK certificate from a manufacturer root the test makes (`BCD_FAKEHW_TPM_EK_CA`) and the node key in that TPM (`BCD_FAKEHW_TPM_KEYS=1`) | a real `bcd join --attestor tpm` with no token against the dev stack's JoinService, whose `tpmAttestor` trusts only that root (`ekRoots`, no `allowUncertifiedEk`): two rounds, `resolve` seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to `sha256(EK)` in `node_bindings`; `node.key` a `BCD TPM NODE KEY` record; the node key certified by the AK, so `node.joined` says `nodeKey: 'tpm'` for this join and for the rejoin below (cf24876); with its identity removed (a reinstall) the same TPM joins as the same node; `bcd node` on that identity, signing its session and statements in the TPM, is adopted; since d94f1f3 the certificate carries the AK's `urn:bc:tpm-ak:` statement, and `bcd node` renews (`BCD_RENEW_AFTER=3s`) with the AK certifying the next key, which stays a `BCD TPM NODE KEY` record while the new certificate keeps the statement |
| `tests/server-join-tpm-renew.test.ts` (d94f1f3)                           | `tpmProducer` and `softwareTpmCertify` over `SoftwareTpm`                                                                                                                | a certified join names the AK; a renewal without `key_attestation` is `key-attestation-missing`, another TPM's AK `key-attestation-ak`, a certification of another key `node-key-not-csr`, one for another CSR `node-key-nonce`; a good renewal keeps the statement although its CSR asks for no SAN, and the next renewal needs the AK again; an uncertified join (`nodeKey: 'any'`) gets no statement and renews without one                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `internal/agent` `TestRenewCertifiedByTheAK` (d94f1f3)                    | `tpmsim`                                                                                                                                                                 | a certificate naming the sim's AK renews with the next key made in that TPM and certified by it; with no TPM the renewal fails (`unavailable`), and with another TPM (another AK) it fails before any request, never falling back to a file key                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `internal/helper` `TestHelperTPMKeys` (8870b8d, Linux)                    | `tpmsim` behind a helper server on a test socket                                                                                                                         | the key and AK the node gets through `tpm.*` are the same as the TPM's own; signing and certifying; at most 8 keys loaded (least recently used unloaded) and a released key derived again on use; refusals of bad shapes, unknown verbs and a helper without a TPM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |

`scripts/daemon-tpm-fixture.mjs` regenerates the fixtures in `daemon/internal/tpm/testdata` after a deliberate change
of either side, so a format change fails a test rather than a machine's join. Since b973e05 the contract runs the
attestor as against a real TPM: `ek-root.der` is a test manufacturer root and `ek-cert.der` the EK certificate it
issued for the sim's EK (made once by the Go test, and kept with `sim.json` when the fixtures are regenerated), and
the attestor verifies that chain through `ekRoots`. Since 8a9af54 the dev-stack e2e does the same: a fakehw bcd
given `BCD_FAKEHW_TPM_EK_CA` (a PEM file with the root's certificate and its PKCS#8 key) issues its software TPM an EK
certificate in NV at each load, so `allowUncertifiedEk` is no longer used by any bcd test. Every test uses a software
TPM: no test has run against a hardware TPM, a real manufacturer's EK certificate or a real `/dev/tpmrm0`.

## Single-use evidence [#single-use-evidence]

Tokens are not the only thing that must not be used twice. `cloud-iid` and `discovery` record what they
accepted in `attestation_uses (attestor, subject, expires_at)` inside the join transaction:

| Attestor    | Subject                                   | Kept until             |
| ----------- | ----------------------------------------- | ---------------------- |
| `cloud-iid` | SHA-256 of provider, account and instance | forever                |
| `discovery` | SHA-256 of the signed assertion body      | the assertion's expiry |

A second join with the same subject is refused. `pruneAttestationUses(tx, { batch? })` from
`@better-compute/postgres` deletes expired rows, 5,000 per call by default; see
[retention](/docs/operations/retention). Single-use tokens (`provisioned`, `discovery`) must have exactly one
use left, so a token minted for several machines is never accepted as a machine credential.

## Two-round joins [#two-round-joins]

Some attestors cannot decide in one request: `tpm` must challenge the TPM, and `discovery` or `tpm` may wait for
an operator. Such an attestor returns an `AttestationDeferral` instead of a result:

```ts
interface AttestationDeferral {
  deferred: 'approval' | 'challenge';
  requestId?: string; // approval: the join request
  retryAfterSeconds?: number; // approval: default 30
  challenge?: Uint8Array; // challenge: attestor-specific bytes
  notAfter?: Date; // challenge: answers after this are refused
}
```

JoinService then commits the transaction (so the attestor's own rows, such as a pending join request, persist),
issues nothing and spends nothing, and answers with a signed `JoinResponseBody` whose `deferral` holds either
`JoinPending{ request_id, retry_after_seconds }` or `JoinChallenge{ attestor, payload, not_after }`. The node id
and chain are empty, and `csr_sha256` still binds the answer to the request. The agent re-sends `Join` with the
**same CSR**: after the wait, or with its answer to the challenge in `attestation`. Attestors therefore consume
tokens and record uses only on the final round.

On the agent side, `openJoinResponse` reads a response as joined, pending or challenge, and `verifyJoinResponse`
refuses a deferral. `runAgentJoin` in `@better-compute/testing` is the reference retry loop.

## Requiring an attestor [#requiring-an-attestor]

By default a pool takes a machine whichever attestor admitted it. Since 1516728 a pool can require hardware
attestation with `policy.joinAttestors`, the ids of the attestors that may admit a machine into it:

```ts title="tpm-pool.ts"
const pool = await client.pools.get({ tenantId: 'root', poolId: 'pool_01k…' });
await client.pools.update({
  tenantId: 'root',
  poolId: pool.id,
  // The policy is replaced as a whole, so keep the rest of it.
  policy: { ...pool.policy, joinAttestors: ['tpm'] }, // or ['tpm', 'x509pop'], or ['join-token', 'tpm'] to keep tokens
});
```

* **Where it is checked.** JoinService checks it in the transaction that resolves the pool, before any reclaim or
  insert. A join or a reclaim admitted by an attestor that is not listed is refused with `ACCESS_DENIED`, reason
  `pool-attestor`. A token join is `join-token`, so a pool that lists only `tpm` refuses tokens, and a token reclaim
  of a TPM-bound machine (`allowReclaim` with the same trusted hardware key) is refused while its TPM still reclaims
  it.
* **Who sees the refusal.** The attestor has already verified the machine, so the refusal is attributed to the pool's
  owner: `security.join.refused` with the attestor and reason `join:pool-attestor`, and the token or node id when
  known.
* **Validation.** `pools.create` and `pools.update` refuse (`INVALID_INPUT`) a value that is not an array of 1 to 16
  ids of attestors this control plane runs: `join-token` and every attestor registered in `attestors` or by a module
  or plugin.
* **Moving nodes.** No API moves a node between pools. A node gets its pool at join, where the check runs, or at
  `metal.enroll`, whose machine gets its identity only through JoinService.

Tested by `tests/server-join-pool-attestors.test.ts`.

## Choosing an attestor [#choosing-an-attestor]

| Situation                                                           | Use                                                                                                            |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| A server you install by hand or with configuration management       | `join-token`, with `expect` when the firmware reports placeholder identifiers                                  |
| A node whose certificate expired while it was offline               | `x509pop`                                                                                                      |
| The OS a Better Compute deploy or a provider bootstrap installed    | `provisioned`                                                                                                  |
| A ramdisk during bare-metal commissioning                           | `discovery`, with `approval` if unexpected machines should wait for a person                                   |
| A VM your provider account created                                  | `cloud-iid`                                                                                                    |
| Hardware identity that must survive reinstalls, on TPM 2.0 machines | `tpm` (v0.5), with `bcd join --attestor tpm`; add `policy.joinAttestors: ['tpm']` to refuse every other way in |

To write your own attestor, see [attestors](/docs/extending/attestors).


# Commands (/docs/guides/nodes/commands)

> The durable command queue from an operator's view, with commands.run, kinds and lanes, deadlines, safety flags, signing, cancellation and results.



A command is a one-shot action on one node: run a diagnostic, re-send inventory,
collect logs, reboot, wipe a disk. It is the other half of what a node receives, next to its
desired state, and the rule for choosing between them is simple:

* if losing the message would leave the node in the wrong steady state, it is **desired state** (assignments,
  instances, host configuration, access, the daemon version), which the node converges to and re-reads after
  any gap;
* if running it late could do harm, it is a **command**, with a deadline (`not_after`) and, when it presupposes
  a spec, a minimum generation.

Commands are durable rows in Postgres. They are queued in the caller's transaction, delivered in order over the
node's session, deduplicated by the node's journal, and settled by a result the node reports. See
[desired state and commands](/docs/guides/concepts/desired-state-and-commands) for the design.

## Running a command [#running-a-command]

[`commands.run`](/docs/reference/api/commands#run) (`compute:nodes:exec` on `compute-node/{nodeId}`) queues a
command from the API. It accepts an allowlist of kinds; any other kind is `UNSUPPORTED`.

| Kind                | Payload                                                                                                                                           | Does                                                                                                                        |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `diag.nvml`         | `{}`                                                                                                                                              | NVML health check of every visible GPU; attested                                                                            |
| `inventory.refresh` | `{ sections?, rescan? }`: 1 to 8 of `system`, `cpu`, `memory`, `disks`, `nics`, `gpus`, `bmc`, `os`; `rescan` re-collects from the hardware first | Re-sends inventory                                                                                                          |
| `logs.collect`      | `{ sinceSeconds?, units? }`: 60 to 604,800 seconds, up to 16 systemd units                                                                        | Collects logs and, when the server has `supportBundles`, uploads a support bundle (see [support bundles](#support-bundles)) |
| `node.reboot`       | `{}`                                                                                                                                              | Reboots the machine; completes when the node rejoins                                                                        |

bcd runs all four. Its `diag.nvml` works from the collected GPU inventory rather than the NVML library (cb987e5).
Since 662fb55, `nvml.init` passes only when the driver's management interface (`nvidia-smi`) answered, not when the
kernel driver alone names the GPUs; then it checks that the GPUs are visible (and the expected count), and per GPU
`ecc.uncorrectable`, `row-remap.pending`, `row-remap.failure`, `pcie.link` (a link narrower than its maximum width;
speed is left out, since idle GPUs lower it on purpose) and `xid.recent` (XID conditions this boot). Every failing
check says why. Its `node.reboot` goes through `systemctl reboot`. It is how a node with the condition
`RebootRequired` gets a host setting that waits for a boot (1 GiB huge pages, isolated CPUs, the IOMMU, an ECC mode,
a new driver): bcd never reboots on its own, so reboot such nodes within the pool's disruption budget (see
[RebootRequired](/docs/guides/nodes/inventory-and-health#conditions)).

bcd also runs `gpu.reset`, which `commands.run` does not accept: the provision module's remediation sends it for
`RowRemapPending` once no allocation is live. bcd refuses the whole command `gpu-busy` while any listed GPU is in use
(539c4ce): a live hold names it or one of its MIG instances or slots (`<uuid> has a live assignment`), or a VM has it
passed through (`<uuid> is passed through to <instance>`). It resets the others with `nvidia-smi --gpu-reset`,
reports per GPU whether the reset worked and whether a fresh inventory shows it healthy, and clears the XID
conditions of a GPU the reset left healthy (662fb55). `diag.dcgm` applies the same refusal from level 2, where DCGM
loads the GPUs, to the listed GPUs or, with none listed, to every GPU; level 1 only checks the deployment. See
[delivery](/docs/daemon/delivery#gpureset-and-live-holds).

```ts
const command = await client.commands.run({
  tenantId,
  nodeId,
  kind: 'inventory.refresh',
  payload: { sections: ['gpus'], rescan: true },
  idempotencyKey: 'refresh-after-gpu-swap-0412',
  notAfterSeconds: 300,
});
// { id: 'cmd_…', seq: 42, state: 'pending', notAfter: …, retrySafe: true, … }
```

```bash
better-compute api commands.run tenantId=$TENANT nodeId=node_… kind=node.reboot idempotencyKey=reboot-0412
```

* `idempotencyKey` is scoped to your tenant and identity: a retry returns the command already queued, and the
  same key for another kind is `CONFLICT`.
* `notAfterSeconds` (10 to 3,600, default 600) is how long the command may wait for the node before it expires.
* The node must be visible to your tenant (`NOT_FOUND` otherwise), and a new command publishes
  `node.command.queued`.
* `batchId` (optional, `batch_…`) puts the command into a destructive batch (46afe75); see below.

### Destructive commands under the quorum [#destructive-commands-under-the-quorum]

With the v1.0 [destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) on (approver keys in
`pki.approvers`) and no break-glass allowance (`unbatchedDestructivePerHour` 0), every daemon refuses a destructive
kind that is not part of an approved batch, so the queue refuses it up front: `node.reboot` without a `batchId` is
`INVALID_STATE` with `details: { reason: 'batch-required', kind }` (508728e). Create the batch first, have enough
approvers sign it, and run the command in it:

```ts
const batch = await client.commandBatches.create({
  tenantId,
  reason: 'Reboot node_0412 after the kernel update',
  scope: [{ subject: nodeId, kind: 'node.reboot' }], // payloadSha256 (hex) would bind one exact payload
  idempotencyKey: 'reboot-0412-batch',
});
await client.commands.run({ tenantId, nodeId, kind: 'node.reboot', idempotencyKey: 'reboot-0412', batchId: batch.batchId });
// pending until the batch is approved: better-compute batch approve --tenant … --batch batch_… --key approver.pem
```

`commandBatches.create` needs `compute:batches:create` on the tenant and is idempotent per caller and key. The
command's `(subject, kind)` must be in the batch's scope (for a node's own kinds the subject is the node id), or
`commands.run` is `INVALID_INPUT`. The command stays `pending`, held, until the batch has its approvals; then it is
sent with the approval and its inclusion proof, and the node checks both itself. `commands.run`, `get` and `list` say
so (a299e96): a batched command carries its `batchId`, and while it waits for approvals `awaitingApproval: true` and
no `notBefore`; once the batch is approved, `awaitingApproval` is gone and `notBefore` is set. Without approver keys a batch is
recorded for audit only and its commands go out at once.

The platform's own destructive work creates its batches itself: metal's workflows, `metal.bulk` and `metal.power`
(f66e236, 478025a), and the health reconciler's reboots and GPU resets, each under a batch whose leaves name the node
(e594a4f). An agent on the [MCP server](/docs/guides/clients/mcp) can propose a reboot batch with
`propose_command_batch` and pass its `batchId` to `exec_command`, but has no tool to approve one: people approve with
their approver keys (171ad2f). See [who creates batches](/docs/operations/pki#who-creates-batches).

Modules queue any registered kind through the command queue, `ctx.commands.enqueue(tx, command)`, in their own
transaction. See [modules](/docs/extending/modules).

### Support bundles [#support-bundles]

`logs.collect` uploads only when the server is given `supportBundles: { blob, maxBytes? }`, a `BlobStore` that
can `presign` (see [configuration](/docs/operations/configuration)). `commands.run` then puts a presigned PUT
URL for `support-bundles/{nodeId}/{key}.bundle` into the command (`key` is the queue's `api:…` idempotency key
derived from yours, without the `api:` prefix; `supportBundleKey(nodeId, key)` computes it), valid until the command's `not_after` plus 15 minutes, with a byte cap
(`maxBytes`, default 256 MiB; the daemon truncates oldest first) and `supportBundle: true`. No credential travels
in the command, and a retry with the same idempotency key presigns nothing new.

`commands.run`, `commands.get` and `commands.list` show the key as the command's `bundleKey`. There is no API to
download a bundle yet; operators read it from the blob store. Without `supportBundles`, `logs.collect` has no
upload destination.

## Kinds and lanes [#kinds-and-lanes]

Every command kind is registered with its wire flags, a lane, a timeout and the agent roles that may run it. The
full list, with payload types, is in [command kinds](/docs/reference/command-kinds); the built-ins are
`diag.nvml`, `inventory.refresh`, `logs.collect`, `node.reboot` and `gpu.reset`, and modules add the rest (the
metal module the BMC, boot, disk, deploy, `site.httpProxy` and `diag.dcgm` kinds, the rollout module
`daemon.upgrade`).

| Flag              | Meaning                                                                                                                                                                                                                                                                               |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lane`            | `control`, `provision` or `bulk`. Lanes keep a long disk wipe from blocking a quick diagnostic.                                                                                                                                                                                       |
| `retrySafe`       | May run again when the daemon crashed after accepting it but before it started. Kinds such as `gpu.reset`, `disk.wipe`, `deploy.image` and `bmc.power` are not.                                                                                                                       |
| `selfTerminating` | The side effect ends the daemon (`node.reboot`, `boot.kexec`, `daemon.upgrade`). The command completes when the node rejoins with a new boot, and is never replayed.                                                                                                                  |
| `attested`        | The result is signed by the node's key, for example `diag.nvml`, `diag.dcgm`, `disk.wipe` and `gpu.reset`.                                                                                                                                                                            |
| `destructive`     | Under the v1.0 quorum it runs only as part of an approved batch; for example `node.reboot`, `gpu.reset`, `disk.wipe`, `deploy.image`, `bmc.power` and, since 0d42de8, `site.httpProxy`. (Signatures no longer depend on it: with command-signing roots every command must be signed.) |
| `timeout`         | How long an in-flight command may run past `not_after` before the sweeper fails it.                                                                                                                                                                                                   |
| `roles`           | Agent roles that may run it (`node`, `site`, `ramdisk`); any other role rejects it.                                                                                                                                                                                                   |

Lanes bound the work in flight per node. The gateway keeps at most 8 `control`, 4 `provision` and 2 `bulk`
commands outstanding per session; the rest stay `pending` in Postgres until credits return. The daemon runs each
lane on its own fixed worker pool.

## States [#states]

<Mermaid
  chart="stateDiagram-v2
  [*] --> pending
  pending --> dispatched: sent under a session epoch
  pending --> expired: past not_after
  pending --> cancelled
  dispatched --> pending: BUSY, with backoff
  dispatched --> acked
  dispatched --> rejected
  dispatched --> running
  acked --> running
  running --> succeeded
  running --> failed
  dispatched --> succeeded
  dispatched --> failed
  acked --> failed
  running --> cancelled
  succeeded --> [*]
  failed --> [*]
  rejected --> [*]
  expired --> [*]
  cancelled --> [*]"
/>

Reports may skip steps: a lost acknowledgement does not stop a result from landing. `succeeded`, `failed`,
`rejected`, `expired` and `cancelled` are terminal.

Before accepting a command, the daemon checks, in this order:

1. `UNSUPPORTED`: the body names another node, or the kind is unknown or not allowed for its role;
2. `STALE_GENERATION`: `min_generation` is ahead of the spec it applied; it asks for the spec, waits until
   `not_after`, then rejects;
3. `EXPIRED`: past `not_after`, in server time (the daemon keeps the offset from `Welcome.server_time`);
4. `BAD_SIGNATURE`: once the node's trust bundle holds command-signing roots, a command of any kind without a valid
   signature (0d42de8 in bcd, 12ccfa1 in the FakeDaemon; before, only destructive kinds needed one);
5. `BATCH_UNAPPROVED` or `UNBATCHED_LIMIT`: with approver keys, a destructive command outside an approved batch
   (see [below](#destructive-commands-under-the-quorum));
6. `BUSY`: no credits; the gateway puts the command back to `pending` with a backoff from 1 to 30 seconds.

The refusals other than `BUSY` make the command `rejected`, and the daemon journals them, so a duplicate delivery
gets the same answer. See [acceptance checks](/docs/daemon/delivery#acceptance-checks).

## Timing [#timing]

| Field            | Rule                                                                                            |
| ---------------- | ----------------------------------------------------------------------------------------------- |
| `not_before`     | Not dispatched earlier. Must come before `not_after`.                                           |
| `not_after`      | Mandatory. A command without one is `EXPIRED` on the node, so the control plane always sets it. |
| `min_generation` | The node must have applied at least this desired-state generation.                              |

The CommandSweeper, a singleton that runs every 5 seconds, enforces the deadlines on the control plane side:

* `pending` past `not_after` becomes `expired` (reason `not_after`);
* `dispatched`, `acked` or `running` past `not_after` plus the kind's timeout becomes `failed` (reason
  `timeout`). For self-terminating kinds the timeout covers the rejoin window. A kind that is no longer
  registered gets 15 minutes.

Each transition wakes the node's controller and the owner named by the idempotency key (`wf:<workflowId>:…`
wakes that workflow), and publishes `command.settled`.

## Delivery, idempotency and resume [#delivery-idempotency-and-resume]

* **Identity and order.** A command gets a `cmd_…` id and a per-node `seq` from `nodes.next_cmd_seq` in the
  enqueue transaction. The idempotency key is unique per node: workflow steps use `wf:<id>:<stepId>`,
  acceptance uses `acceptance:<nodeId>:<attempt>:<kind>`, and the API derives `api:…` from your key. A second
  enqueue with the same key returns the first command.
* **Resume.** When a node reconnects, `Hello` first settles what the reconnect proves, then the gateway replays
  every non-terminal command in `seq` order. The daemon's journal answers a duplicate with the stored result, or
  re-attaches to a running one.
* **Crashes.** A command the daemon accepted but never started runs again only if it is retry-safe; otherwise
  it fails with `interrupted` and its owner decides.
* **Lost journal.** A daemon whose journal is gone (a ramdisk reboot, a reinstall, corruption) comes back with a
  new journal id. In-flight commands that are not retry-safe fail with `journal-lost` instead of being replayed.
* **Self-terminating kinds.** A new boot or role on rejoin completes them as `succeeded` with `rebooted: true`.

The details, including epoch fencing, are in [delivery](/docs/daemon/delivery).

## Restricted identities [#restricted-identities]

A node whose current certificate carries allowed kinds (a discovery ramdisk, for example) only accepts those.
The command queue refuses other kinds at enqueue with `INVALID_STATE` ("runs a restricted identity that does not
accept …"), and the gateway checks the session's certificate again at dispatch. See
[attestation](/docs/guides/nodes/attestation).

## Signatures and attested results [#signatures-and-attested-results]

Two signatures protect commands from a compromised gateway, which can delay, drop or read frames but should
never forge intent or results:

* **Attested results.** For attested kinds the node signs its result with its node key: `CommandUpdate.node_sig`,
  under purpose `command-update`, over the command id, the state and the SHA-256 of the result. The gateway
  stores it exactly as received, and consumers verify it against `node_certs` before any state transition (the
  metal module's `nodeSignedResults()` does this for wipes and diagnostics). A forged wipe result is refused.
* **Signed commands (v0.2).** `Command.signature` carries a signature by a worker key that chains to the
  command-signing root in the node's trust bundle. `@better-compute/pki` provides the pieces
  (`createCommandSigningAuthority`, 24-hour worker keys from `issueWorkerKey`, `signCommand` and
  `verifyCommand`). Once the daemon's trust bundle holds command-signing roots, it rejects every command without a
  valid signature with `BAD_SIGNATURE`, whatever its kind: exempting "harmless" kinds let a compromised gateway
  inject an unsigned `site.httpProxy` that could reach a BMC (0d42de8, 12ccfa1). When the server is
  given `pki.commandSigning` (the root from `createCommandSigningAuthority`, or the PKI bundle's `commandSigning`),
  every command is signed by a 24-hour worker key that root certifies, and JoinResponse and Welcome carry the
  trust bundle (the roots, plus `pki.commandSigningRoots` during a rotation, at a version that follows the bundle's
  content, never below `pki.trustVersion`). Without it,
  commands go unsigned as in v0.1. The same key signs each NodeSpec generation (purpose `desired-state`, v0.4); the
  gateway relays the stored signed bytes, and the daemon verifies them before applying a spec. See
  [delivery](/docs/daemon/delivery#signed-commands-and-attested-results).

With `pki.commandSigning`, GPU bindings are attested too: a reservation goes active only once the node's own signed
`AllocationBound` verifies; see [attested bindings](/docs/guides/capacity/reservations#attested-bindings).

Commands never carry secret values: they carry `secretRef`s, which the agent exchanges for a value sealed to a
one-time key. See [security](/docs/operations/security).

## Cancelling [#cancelling]

[`commands.cancel`](/docs/reference/api/commands#cancel) (`compute:nodes:exec` on `compute-node/{nodeId}`,
v0.2) takes `{ tenantId, nodeId, commandId }` and returns the command. Modules cancel through the same command
queue, `ctx.commands.cancel(tx, commandId)`:

* a `pending` command becomes `cancelled` at once and never runs;
* an in-flight command keeps its state and is marked `cancel_requested_at`; the gateway sends `Cancel` until the
  daemon reports a terminal state. The daemon stops between actions and reports the real outcome, so a side
  effect that already happened is never hidden;
* a terminal command is returned unchanged; an unknown id (or a node your tenant cannot see) is `NOT_FOUND`.

`commands.cancel` publishes `node.command.cancel-requested` (`nodeId`, `commandId`, `kind`, `state`) for a command
that was not yet terminal; `state` is `cancelled` for a pending one and the in-flight state otherwise.

```ts
const command = await client.commands.cancel({ tenantId, nodeId, commandId: 'cmd_…' });
```

Deleting a node cancels its pending commands with reason `node-deleted`.

## Results [#results]

[`commands.get`](/docs/reference/api/commands#get) and [`commands.list`](/docs/reference/api/commands#list)
(`compute:nodes:read`) show a node's commands; `list` is newest first by `seq`, filters by `state`, and returns
up to 200 per page (100 by default).

<TypeTable
  type="{
  id: { type: 'string', description: 'cmd_…' },
  nodeId: { type: 'string', description: 'The node that runs it.' },
  seq: { type: 'number', description: 'Its position in the node order.' },
  kind: { type: 'string', description: 'The command kind.' },
  state: { type: 'CommandState', description: 'pending, dispatched, acked, running, succeeded, failed, rejected, expired or cancelled.' },
  reason: { type: 'string', description: 'Why it ended as it did: not_after, timeout, journal-lost, node-deleted, …' },
  notBefore: { type: 'number', description: 'Epoch milliseconds.' },
  notAfter: { type: 'number', description: 'Epoch milliseconds.' },
  minGeneration: { type: 'number', description: 'Set when the command presupposes a spec.' },
  retrySafe: { type: 'boolean', description: 'From the kind.' },
  selfTerminating: { type: 'boolean', description: 'From the kind.' },
  attested: { type: 'boolean', description: 'From the kind.' },
  bundleKey: { type: 'string', description: 'logs.collect with supportBundles: the blob key the bundle is uploaded to.' },
}"
/>

Rather than polling, follow `command.settled` events (`commandId`, `nodeId`, `kind`, `state`, `reason`), which
the gateway publishes when a node reports a terminal state and the sweeper publishes when it expires or fails a
command. See [watching events](/docs/guides/events/watch).

```ts
const { items } = await client.commands.list({ tenantId, nodeId, state: 'failed', limit: 20 });
for (const command of items) console.log(command.seq, command.kind, command.reason);
```


# Nodes (/docs/guides/nodes)

> The node model in Better Compute, with its independent state axes, role, trust, hardware key and generations, and how a server goes from join to schedulable.



A node is a machine Better Compute manages: a server running the daemon
`bcd`, a site agent (`bcd site`, stored as a node with
`role: 'site'`), or a provider machine in a pool that runs the daemon. Everything the control plane does to
hardware hangs off a node: its devices and atoms, the
allocations booked on them, the commands it runs and the
desired state it converges to.

This section covers the node from an operator's point of view: how a machine joins, how it proves who it is,
what inventory and health say about it, how you take it out of service, and how you send it commands.

> **The daemon today.** 
  The Go daemon `bcd` in `daemon/` is built and tested but not released yet. Its node agent joins, holds a session,
  runs `inventory.refresh`, `diag.nvml`, `logs.collect`, `gpu.reset` and `node.reboot`, binds GPUs through the
  `devices` section and converges break-glass SSH; `tests/e2e/bcd.test.ts` runs it against a real control plane. Its
  site-agent and ramdisk modes are written and covered by Go unit tests only. The TypeScript `FakeDaemon` in
  `@better-compute/testing` speaks the same protocol, and it is what the simulator, most tests and
  `better-compute dev --sim N` run. Everything on these pages describes the control plane, which behaves the same
  for both. See [what bcd does today](/docs/daemon#what-bcd-does-today).

## State is several axes, not one status [#state-is-several-axes-not-one-status]

A single status field cannot say "deployed, but cordoned for a firmware update, in maintenance after a failed
workflow, unreachable since a switch reboot, with one GPU reporting uncorrectable ECC errors". So a node's state
is split into independent axes, the way OpenStack Ironic does it. Each axis has one owner, and none of them
implies another.

| Axis             | Values                                                                                                                                                      | Changed by                                                                                  |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `provisionState` | `adopting`, `adopted`, `enrolling`, `inspecting`, `commissioning`, `ready`, `deploying`, `deployed`, `releasing`, `rescue`, `failed`, `retiring`, `retired` | JoinService, the NodeLifecycle controller (acceptance), `nodes.adopt`, bare-metal workflows |
| `maintenance`    | `true` or `false`                                                                                                                                           | A bare-metal workflow that fails or is aborted, and health remediation that gives up        |
| `cordoned`       | `true` or `false`                                                                                                                                           | `nodes.cordon`, `nodes.uncordon`, `nodes.drain`                                             |
| `power`          | `on`, `off`, `unknown`                                                                                                                                      | Bare-metal power control through the BMC                                                    |
| `connectivity`   | `connected`, `disconnected`, `unreachable`, `lost`                                                                                                          | The gateway (session open and close) and the Liveness controller                            |
| `conditions`     | Typed conditions, each `true`, `false` or `unknown`                                                                                                         | Inventory, Health, Liveness, NodeLifecycle                                                  |

A seventh field ties them to scheduling: `noScheduleReasons`. It mirrors every reason the node must not receive
new work, such as an operator taint (`taint:<key>`), a NoSchedule
condition (`GpuCountMismatch`), `unreachable` or `draining`. Each writer updates it
in the same transaction as its cause, so the placer can lock the node row and read one column.

### Provision states [#provision-states]

`provisionState` follows a state table in `@better-compute/core` (`provisionStateTable`). A server you already
run joins as `adopting` and becomes `adopted` once acceptance passes. A bare-metal server enrolled through its
BMC starts at `enrolling` and moves through inspection and commissioning to `ready`. Illegal moves fail with
`INVALID_STATE`.

<Mermaid
  chart="stateDiagram-v2
  [*] --> adopting: join
  [*] --> enrolling: enroll
  adopting --> adopted: acceptance passes
  adopted --> adopting: adopt again
  enrolling --> inspecting
  inspecting --> commissioning
  commissioning --> ready
  ready --> deploying: deploy
  deploying --> deployed
  deployed --> releasing: release
  adopted --> releasing: release
  releasing --> ready: wipe verified
  deploying --> failed
  failed --> retiring: retire
  ready --> retiring: retire
  adopted --> retiring: retire
  retiring --> retired
  retired --> [*]"
/>

The diagram leaves out a few edges (every workflow state can fail, `failed` can `recover`, and `deployed` can
enter `rescue`). The bare-metal side is covered in [bare-metal workflows](/docs/guides/bare-metal/workflows).
A failure records `failedFrom` and turns `maintenance` on.

## Identity, ownership and generations [#identity-ownership-and-generations]

Besides the state axes, a node carries these fields.

| Field                               | Meaning                                                                                                                                                                                                                                                       |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                                | `node_…`, assigned by JoinService or by `metal.enroll`.                                                                                                                                                                                                       |
| `siteId`, `poolId`, `ownerTenantId` | Where the node lives and who owns it. `poolId` is absent for site agents and unassigned machines.                                                                                                                                                             |
| `role`                              | `host`, `dedicated`, `site` or `unassigned`. A node that joins a `metal-host`, `vps-host` or `provider` pool becomes `host`, one that joins a `metal-dedicated` pool becomes `dedicated`, a site agent is `site`, and a machine with no pool is `unassigned`. |
| `trust`                             | `platform`, `tenant-owned` or `tenant-controlled`. A node on a tenant's `custom` site is `tenant-owned`; a node under a dedicated-root lease is `tenant-controlled`.                                                                                          |
| `adapter`                           | How the control plane reaches the node: `bcd/v1` (the default), `provider-api/v1` for pools with `agent: 'none'`. `agentless-ssh/v1` is planned.                                                                                                              |
| `drivers`                           | Per-concern driver bindings (`power`, `boot`, `deploy`, …) inherited from the pool's hardware profile. See [drivers](/docs/extending/drivers).                                                                                                                |
| `hardwareKey`, `hardwareKeyTrusted` | SHA-256 over the machine's normalized identifiers. Trusted, and unique across nodes, when at least two independent identifiers survive. See [inventory and health](/docs/guides/nodes/inventory-and-health).                                                  |
| `generation`, `observedGeneration`  | The desired-state generation, and the one the node last reported converged.                                                                                                                                                      |
| `identityGeneration`                | Bumped whenever the node's identity is replaced: `nodes.revokeIdentity`, a reclaim or re-join in place, `nodes.delete`, and bare-metal teardown. Certificates of an earlier generation never work again.                                                      |
| `version`                           | Optimistic concurrency. Pass the `version` you read to a mutation; a node that changed since is `CONFLICT`.                                                                                                                                                   |

`nodes.get` returns the API view (`NodeDetail`): the axes except `power`, the fields above except `adapter`,
`drivers` and the key itself (only `hardwareKeyTrusted`), plus labels, conditions and the current session
(`epoch`, `gatewayId`, `connectedAt`, `disconnectedAt`, `lastSeen`). Modules and node adapters receive the full
`NodeRecord`.

Trust decides what a node's hardware may be used for. A `tenant-owned` node adds capacity only to its owner's
pools, and the offering validator refuses to offer a pool holding one to any other tenant. A
`tenant-controlled` node adds no capacity at all, and the lessee's operating system joins with a `node-tenant`
certificate: the gateway never sends it platform secrets or destructive platform commands. See
[dedicated root](/docs/guides/bare-metal/dedicated-root).

## When a node is schedulable [#when-a-node-is-schedulable]

Placement books a node's atoms only when every check below holds. The capacity module runs them in this order,
and the first that fails decides which bucket of the pool's capacity the node's GPUs are reported in
(see [capacity](/docs/guides/capacity)).

1. `role` is `host`. Otherwise the node is `excluded`.
2. `trust` is `platform`, or `tenant-owned` in its owner's pool. Otherwise `excluded`.
3. Not in `maintenance` and not `cordoned`. Otherwise `maintenance`.
4. `provisionState` is `adopted` or `deployed`. Otherwise `provisioning`.
5. `observedGeneration` is at least 1: the node converged its first spec. Otherwise `provisioning`.
6. `connectivity` is `connected`. Otherwise `unreachable`.
7. `noScheduleReasons` is empty. Otherwise `unhealthy`.

The same predicate exists in SQL (`schedulableSql`), and the placer applies it to rows it locks, so a node
cordoned in one transaction cannot receive an allocation in another.

## The lifecycle at a glance [#the-lifecycle-at-a-glance]

      ### Join [#join]

    The daemon presents a join token (or other evidence) to JoinService, which admits
    it in one transaction: a node row in `adopting`, a 24-hour node certificate
    and a `node.joined` event. See [joining](/docs/guides/nodes/joining) and [attestation](/docs/guides/nodes/attestation).
  
      ### Connect and report [#connect-and-report]

    The daemon opens a session to the gateway with `Hello`, and
    sends the inventory sections whose hashes the control plane has not seen. The
    Inventory controller turns them into devices, labels and conditions.
  
      ### Acceptance [#acceptance]

    The NodeLifecycle controller runs the pool's acceptance checks (`diag.nvml` in v0.1) as commands. On a pass the
    node becomes `adopted`, and the NodeSpec controller writes generation 1.
  
      ### Schedulable [#schedulable]

    Once the node reports `observedGeneration` 1 and nothing blocks it, reservations can land on it.
  
      ### Operate and retire [#operate-and-retire]

    You cordon, drain, taint and label it, send it commands, and eventually retire and delete it. See
    [operating nodes](/docs/guides/nodes/operating-nodes).
  
No step on this path calls better-iam: joins, heartbeats, inventory and commands are fleet paths, and IAM is
consulted only when a person or agent calls the API.

## In this section [#in-this-section]

  - [Joining](/docs/guides/nodes/joining): Join tokens, the join flow, node certificates and renewal, adopting a brownfield server, and join requests.

  - [Attestation](/docs/guides/nodes/attestation): Every attestor, what it proves, single-use evidence and two-round joins.

  - [Inventory and health](/docs/guides/nodes/inventory-and-health): NodeInventory v1, hashing, GPU naming, devices, conditions, XID handling and acceptance.

  - [Operating nodes](/docs/guides/nodes/operating-nodes): Cordon, drain, taint, labels, liveness, identity revocation and deletion.

  - [Commands](/docs/guides/nodes/commands): The command queue: kinds, lanes, timing, safety flags, results and cancellation.


# Inventory and health (/docs/guides/nodes/inventory-and-health)

> NodeInventory v1 and its section hashes, snapshots and diffs, GPU naming, devices and atoms, conditions, XID handling, remediation and acceptance.



Placement can only book what the control plane knows a node has, and should only book it on hardware that
works. Inventory is what a node reports about its hardware; health is what the
control plane concludes from it. Together they produce the node's devices and
atoms, the labels selectors match, and the conditions that keep
a broken node or GPU out of placement.

Inventory is owned by `@better-compute/inventory`, a built-in module that every server registers.

## NodeInventory v1 [#nodeinventory-v1]

The daemon reports its hardware as eight sections of the `NodeInventory v1` schema
(`bettercompute.agent.v1`). Readings that change by the second (temperatures, power draw, clocks, utilization)
are never inventory; they are telemetry, and they never go to Postgres.

| Section  | Source on the node                   | Holds                                                                                                                                                                                                       |
| -------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `system` | DMI                                  | vendor, product, serials (system, board, chassis), product UUID, BIOS, boot mode, Secure Boot, TPM                                                                                                          |
| `cpu`    | `/proc/cpuinfo`, sysfs               | vendor, model, architecture, sockets, cores, threads, flags, NUMA nodes                                                                                                                                     |
| `memory` | `/proc/meminfo`, EDAC, DMI           | total bytes, huge page pools, DIMMs, EDAC counters                                                                                                                                                          |
| `disks`  | sysfs, udev, NVMe, SMART             | name, model, serial, WWN, size, transport, SMART health                                                                                                                                                     |
| `nics`   | netlink, sysfs                       | physical NICs: permanent MAC, driver, firmware, speed, PCI address, NUMA node, SR-IOV VFs, RDMA, link, MTU                                                                                                  |
| `gpus`   | NVML                                 | driver, CUDA and NVML versions; per GPU: index, UUID, PCI bus id, raw NVML name, architecture, VRAM, serial, VBIOS, NUMA node, MIG state, NVLinks, topology row, fabric clique, ECC counters, row remapping |
| `bmc`    | in-band IPMI, Redfish host interface | IP, MAC, vendor, firmware, Redfish and IPMI support                                                                                                                                                         |
| `os`     | os-release, uname                    | distro and version, kernel, cgroup version, KVM, IOMMU, runtimes, hostname                                                                                                                                  |

The daemon collects inventory when its boot changes, on hardware events and XIDs, after provisioning steps,
on a jittered 12-hour rescan, and when you run `inventory.refresh`.

> **What bcd collects today.** 
  bcd's `inventory.Collector` reads procfs and sysfs without cgo. It fills `system`, `cpu`, `memory`, `disks`, `nics`
  (from sysfs, not netlink) and `os`, and reads NVIDIA GPUs from the kernel driver's `/proc/driver/nvidia` files
  (UUID, model, VBIOS, bus id, driver version). Where `nvidia-smi` is installed, a host build completes the GPUs from
  it (cb987e5, `nvml.SMI`, no cgo): `nvidia-smi -q -x` for identity, VRAM, ECC counters, row remapping, MIG mode and
  instances, fabric clique, power limit, persistence and compute mode, plus `-L`, `--query-gpu compute_cap`,
  `topo -m`, `nvlink -s`, `nvlink -R` and `mig -lgip`, merged into the sysfs GPUs by UUID. `nvlink -R` (5026330)
  names each NVLink's peer from the PCI bus id at its far end: the UUID of this node's GPU on that bus, else
  `nvswitch:<bus id>` for a link through an NVSwitch (`NvlinkLink.peer`); like every query besides `-q -x`, it is
  optional, and a failure leaves the peers empty. The control plane stores the peers with the section; the `nvlinkPeers`
  device attribute still counts the `NV` links of the `topo -m` row. Live samples (memory used, utilization,
  temperature, power, clocks, processes) travel with the section but stay out of its hash, so they never make a new
  inventory version. There is no `bmc` section. The `cpu` section carries the kernel's isolated CPUs
  (`/sys/devices/system/cpu/isolated`) and each NUMA node its own huge page pools (configured totals only: free pages
  change with every VM and would move the hash; badb8c2). It collects when the process starts (so after every boot),
  whenever `inventory.refresh` or `diag.nvml` asks for a rescan, after a config unit applied, every 10 minutes (GPU
  error counters and MIG state change without an event bcd sees) and about 2 seconds after an XID burst (662fb55),
  sending only the sections that changed; the 12-hour rescan is not built.
  The fakehw build reports the full fake HGX board instead. None of this has run on a real GPU host yet: the
  `nvidia-smi` parsers are tested against captured output.

  bcd does not use the NVML library or its events. It sees XIDs by following the kernel log (`/dev/kmsg`) and reports
  each one as a condition, and it samples host pressure every 30 seconds. See [XID events](#health-quarantines-and-remediation) and
  [host pressure](#host-pressure).

## How inventory travels [#how-inventory-travels]

Inventory is large and rarely changes, so it is never re-sent whole. Each section travels, and is stored, by
the hash of its bytes.

      ### Hash on the node [#hash-on-the-node]

    The daemon canonicalizes each section (repeated fields sorted by a stable key), encodes it, and takes the SHA-256
    of those exact bytes. `Hello`, every tenth heartbeat, and the first heartbeat after changed sections were sent
    (427787a) carry the section hashes.
  
      ### Pull what changed [#pull-what-changed]

    The gateway records the hashes and asks, in `Welcome.request_sections`, only for sections whose hash differs from
    the stored one. Unknown section names from a newer daemon are ignored.
  
      ### Verify over the received bytes [#verify-over-the-received-bytes]

    Each `InventorySection{ section, content, sha256 }` is checked against its SHA-256 over the bytes as received
    and decoded; it is never re-encoded to compare, so only the daemon's encoding has to be stable. A section over
    1 MiB, a hash mismatch or bytes that do not decode are `INVALID_INPUT`.
  
      ### Apply once complete [#apply-once-complete]

    The Inventory controller runs when every reported section is present. The snapshot digest is the SHA-256 over
    `name=hash` lines in section order; an unchanged digest does nothing. A changed one is applied in one transaction
    with the node row locked.
  
Applying an inventory:

* appends a snapshot (`inventory_snapshots`, id `inv_…`): snapshots are append-only;
* plans the node's devices and atoms, and its inventory labels;
* sets the hardware key;
* writes the inventory conditions and health findings, and mirrors the NoSchedule ones into
  `noScheduleReasons`, keeping every entry it does not own;
* publishes `node.inventory.changed` (`snapshotId`, `digest`, `sections`, `changes`) and one event per change:
  `node.gpu.missing`, `node.gpu.added`, `node.disk.replaced`, `node.driver.changed`, `node.mig.layout.changed`.

### Trust: tenant-controlled nodes are advisory [#trust-tenant-controlled-nodes-are-advisory]

A node under a [dedicated-root lease](/docs/guides/bare-metal/dedicated-root) has `trust: 'tenant-controlled'`:
the lessee has root, so it can report anything. Its reports are advisory (PLAN.md §10.1 Trust). For such a node:

* `applyInventory` appends the snapshot and publishes `node.inventory.changed` with `advisory: true` (and no
  per-change events), and changes nothing else. Devices, device health, labels, conditions, `noScheduleReasons`,
  the image cache and the hardware key keep the platform's last view, and the applied snapshot stays the
  platform's last one. The result status is `advisory`.
* The applied digest is stored as `advisory:<digest>`, so hash reports treat the set as applied and heartbeats do
  not queue it again.
* `recordXid` ignores its XIDs (no condition, no quarantine, no NoSchedule), and `recordHostPressure` ignores its
  pressure, so a lessee cannot make a `vps-host` pool's node NoSchedule or raise conditions.
* The pool's image cache query for the `image-cached` scorer skips tenant-controlled nodes.

Without this, a GPU the lessee reports missing or unhealthy (or a faked XID) would turn the device
`healthy: false` and leave the lease's bill. The `inventory.get`, `inventory.history` and `inventory.diff`
methods still read the appended snapshots, the lessee's included. When metal ends the lease,
`metal.exitTenantRoot` calls `endAdvisoryInventory`, which clears the advisory digest, so the next report applies
the set for real under platform trust even if it did not change.

## Reading inventory [#reading-inventory]

All three methods are decided as `compute:nodes:read` on the node, and answer `NOT_FOUND` for a node of another
tenant or one with no inventory yet.

| Method                                                              | Returns                                                                                                         |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| [`inventory.get`](/docs/reference/api/inventory#get)                | The latest snapshot: `nodeId`, `hash`, `sections` (decoded, 64-bit numbers as strings), `collectedAt`.          |
| [`inventory.history`](/docs/reference/api/inventory#history) (v0.2) | Snapshots newest first, `limit` 1 to 100 (20 by default), paged by snapshot id.                                 |
| [`inventory.diff`](/docs/reference/api/inventory#diff) (v0.2)       | The changes between two snapshots, named by id (`inv_…`) or hash; by default the latest against the one before. |

```ts
const latest = await client.inventory.get({ tenantId, nodeId });
console.log(latest.sections.gpus);

const { from, to, changes } = await client.inventory.diff({ tenantId, nodeId });
for (const change of changes) console.log(change.type, change.deviceId, change.before, change.after);
```

```bash
better-compute api inventory.diff tenantId=$TENANT nodeId=node_… --format table
```

A diff lists, in this order, GPUs missing and added (by UUID), MIG layout changes (mode plus sorted instances),
disks replaced (the same kernel name with another WWN or serial), and driver changes (NVIDIA driver or CUDA
version). A section missing on either side yields no changes for it: a node that stopped reporting a section did
not lose hardware.

## GPU naming [#gpu-naming]

NVML names are inconsistent across drivers and SKUs ("NVIDIA H100 80GB HBM3", "NVIDIA H100 PCIe"). Every GPU
gets two normalized names:

* `gpuProduct`, the exact product in `<MODEL>-<FORM>-<MEMORY>GB` form, such as `H100-SXM5-80GB`;
* `gpuModel`, its family, such as `H100`.

Pools, policies and meters use `gpuModel`: a pool holds exactly one. Selectors may use either. The mapping is the
versioned table `gpuModels` (the plan's `gpu-models.json`, version 2), and `normalizeGpuProduct(name)` applies
it: upper case, punctuation collapsed to `-`, a leading `NVIDIA` dropped.

| Family                             | Products                                                                              |
| ---------------------------------- | ------------------------------------------------------------------------------------- |
| `H100`                             | `H100-SXM5-80GB`, `H100-PCIE-80GB`, `H100-NVL-94GB`                                   |
| `H200`                             | `H200-SXM5-141GB`, `H200-NVL-141GB`                                                   |
| `B200`                             | `B200-SXM-180GB`                                                                      |
| `GB200`                            | `GB200-SUPERCHIP-186GB`                                                               |
| `A100`                             | `A100-SXM4-40GB`, `A100-SXM4-80GB`, `A100-PCIE-40GB`, `A100-PCIE-80GB`                |
| `L40S`                             | `L40S-PCIE-48GB`                                                                      |
| `L4`                               | `L4-PCIE-24GB`                                                                        |
| `RTX-PRO-6000`                     | `RTX-PRO-6000-SERVER-96GB`, `RTX-PRO-6000-WORKSTATION-96GB`, `RTX-PRO-6000-MAXQ-96GB` |
| `GH200`                            | `GH200-SUPERCHIP-96GB`, `GH200-SUPERCHIP-144GB`                                       |
| `L40`, `A40`, `A30`, `A10`         | `L40-PCIE-48GB`, `A40-PCIE-48GB`, `A30-PCIE-24GB`, `A10-PCIE-24GB`                    |
| `RTX-6000-ADA`, `RTX-A6000`        | `RTX-6000-ADA-PCIE-48GB`, `RTX-A6000-PCIE-48GB`                                       |
| `RTX-5090`, `RTX-5080`, `RTX-4090` | `RTX-5090-PCIE-32GB`, `RTX-5080-PCIE-16GB`, `RTX-4090-PCIE-24GB`                      |

Version 2 added the parts real hosts reported that version 1 missed: data-center cards outside the flagship lines,
and the workstation and GeForce cards that edge, lab and single-tenant hosts run.

A name the table does not know keeps its canonical form as `gpuProduct`, maps to `gpuModel: 'other'`, and raises
`GpuModelUnknown`, which keeps the node from scheduling until the table knows it.

## Devices and atoms [#devices-and-atoms]

The `gpus` section becomes devices with stable ids, so a reboot or a re-inventory never renames them:

| Device                   | Id                      | Class      | Bookable                   |
| ------------------------ | ----------------------- | ---------- | -------------------------- |
| A whole GPU              | its UUID, `GPU-…`       | `gpu`      | yes, unless partitioned    |
| A MIG instance           | `<gpuUuid>/gi<g>/ci<c>` | `gpu.mig`  | yes; its parent GPU is not |
| A time-slice or MPS slot | `<gpuUuid>/s<k>`        | `gpu.slot` | yes; its parent GPU is not |

Only atoms are booked. Slots exist in `timeslice` and `mps` pools: 4 per GPU for time-slicing and 10 for MPS by
default (`policy.sharing.replicas`, 1 to 48, MPS's limit; a708aa6). A GPU whose MIG mode is about to change has every atom
unbookable, with `partitionPending: true`, until the change completes. See
[fractional GPUs](/docs/guides/capacity/fractional-gpus).

Device attributes are what selectors and scorers read: `gpuProduct`, `gpuModel`, `vramGiB`, `index`,
`pciBusId`, `numaNode`, `architecture`, `computeCapability`, `driverVersion`, `migCapable`, `nvlinkPeers`,
`fabricCluster` and `fabricClique`. A device that vanished while it still has live bookings is kept with
`healthy: false` and deleted once its bookings end. A device id another node already owns is left alone and
raises `DuplicateIdentity`.

### Labels [#labels]

Inventory also writes node labels with source `inventory`, the way NVIDIA's GPU Feature Discovery does:
`gpu.present`, `gpu.count`, `gpu.product`, `gpu.model`, `gpu.vramGiB`, `cuda.driver.major`,
`nvidia.driver.version`, `mig.capable`, `fabric.domain`, `cpu.arch`, `cpu.vendor`, `cpu.model`, `os.distro`,
`os.kernel` and `kvm.available`. A label whose GPUs disagree reads `mixed`. `nodes.label` cannot overwrite them.
Since badb8c2 the host's CPU and memory have labels too, for selectors and `when:` expressions such as
`numa.nodes > 1`:

| Label                            | Value                                                                                        |
| -------------------------------- | -------------------------------------------------------------------------------------------- |
| `cpu.count`                      | Logical CPUs                                                                                 |
| `cpu.sockets`                    | Sockets                                                                                      |
| `cpu.isolated`                   | How many CPUs the kernel isolates (0 when none)                                              |
| `numa.nodes`                     | NUMA nodes                                                                                   |
| `memory.totalGiB`                | Installed memory in whole GiB                                                                |
| `hugepages.2Mi`, `hugepages.1Gi` | Configured pages in the host's 2 MiB and 1 GiB pools (0 when the size exists but holds none) |

`cpu.model` is the reported model with trademark marks dropped and every other run of characters outside
`A-Z a-z 0-9 . _` turned into `-` (`AMD EPYC 9654 96-Core Processor` reads `AMD-EPYC-9654-96-Core-Processor`);
a sandbox snapshot that includes memory restores only on hosts with an equal value (see
[snapshots](/docs/guides/vps/sandboxes#snapshots-and-restore)).

The gateway writes one more kind at every `Hello`: each driver the daemon advertises in `Hello.drivers` becomes
the label `bc.dev/driver.<id>=<version>` with source `system` (a missing version reads `unknown`; ids outside
`[a-z0-9._-]` are skipped), and driver labels the daemon no longer advertises are dropped, in one statement that
writes nothing when they did not change. Operators cannot overwrite them either. Placement can require them with
`PlacementRequest.nodeLabels`, where `'*'` accepts any version (see
[placement](/docs/guides/capacity/placement#eligibility-filters)). The vps module does this for every runtime: a hosted
instance is placed only on hosts carrying `bc.dev/driver.<runtime>`, bcd's built-in `libvirt-kvm` and
`cloud-hypervisor` included since 0fca413 (before, only plugin runtimes needed it).

Since 14ceb1e the same statement also writes one label per kind each advertised driver serves,
`bc.dev/driver-kind.<id>.<kind>=1` (a site agent hosting `bcx-ipmi` carries `bc.dev/driver-kind.ipmi.power`,
`.management` and `.inspect`), and drops the ones a later `Hello` no longer lists. Kinds outside `[a-z][a-z0-9-]`
(1 to 32 characters) are skipped, and at most 32 are read per driver. A placement or selector can require one with
`'*'`, as for driver labels. Metal reads them (a3ec8d0): before a verb starts a workflow, it keeps only the operations
of the kinds the site's agents report for the node's BMC driver, and refuses the verb with `UNSUPPORTED` (reason
`bmc-driver-unsupported`) when a step needs one they lack. The labels also make a driver metal's own table does not
know checkable. See [what metal checks before a workflow](/docs/guides/bare-metal/bmc#what-metal-checks-before-a-workflow).

### The hardware key [#the-hardware-key]

The hardware key identifies the machine rather than its operating system. It is the SHA-256 over the normalized
product UUID, serial (system, else board, else chassis), sorted permanent MACs and sorted GPU UUIDs, after
dropping firmware placeholders such as `To Be Filled By O.E.M.`. It is **trusted** when at least two independent
identifiers survive, and trusted keys are unique across nodes. JoinService computes it from the join claims and
the Inventory controller from inventory, with the same function, so both agree. A second node with the same
trusted key (a cloned disk image, a moved board) gets `DuplicateIdentity` and keeps its key untrusted until an
operator resolves it.

A copied node certificate shows up at the gateway instead: two machines with the same certificate keep taking the
node's session from each other, each second `Hello` publishing `security.node.duplicate-session`. More than 3 such
alternations within 10 minutes revoke the certificate (`security.node.quarantined`, reason `duplicate-identity`), and
the real machine must re-join with a new token. See [duplicate identity](/docs/daemon/sessions#duplicate-identity).

## Conditions [#conditions]

A condition is `{ type, status, reason?, message?, since }`, where `status` is `true`, `false` or `unknown` and
`since` moves only when the status changes. `nodes.get` lists them. A condition with a NoSchedule policy also
puts its type into `noScheduleReasons`, in the same transaction.

| Condition                                              | Raised by                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Effect                                                                                                                                                                                                                    |
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GpuCountMismatch` (v0.1)                              | Inventory: a GPU count other than the pool's `policy.expect.gpuCount`, or a family other than the pool's `gpuModel`                                                                                                                                                                                                                                                                                                                                                                                                  | NoSchedule. `unknown` with reason `NotReported` when the node sends no `gpus` section.                                                                                                                                    |
| `GpuModelUnknown` (v0.1)                               | Inventory: a GPU name missing from `gpuModels`                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | NoSchedule                                                                                                                                                                                                                |
| `DuplicateIdentity`                                    | Inventory: the trusted hardware key or a device id belongs to another node                                                                                                                                                                                                                                                                                                                                                                                                                                           | NoSchedule                                                                                                                                                                                                                |
| `AcceptanceFailed`                                     | NodeLifecycle, see below                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | NoSchedule while `true`                                                                                                                                                                                                   |
| `NodeUnreachable`                                      | Liveness, see [operating nodes](/docs/guides/nodes/operating-nodes)                                                                                                                                                                                                                                                                                                                                                                                                                                                  | the `unreachable` reason                                                                                                                                                                                                  |
| `Xid48` (v0.2)                                         | an XID 48 (double-bit ECC)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | the GPU is quarantined                                                                                                                                                                                                    |
| `Xid79`, `Xid95` (v0.2)                                | an XID 79 (fell off the bus) or 95 (uncontained ECC)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | NoSchedule; a reboot once no allocation is live                                                                                                                                                                           |
| `RowRemapPending` (v0.2)                               | XID 63, or inventory's row-remap state                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | NoSchedule; a GPU reset once no allocation is live                                                                                                                                                                        |
| `RowRemapFailed` (v0.2)                                | XID 64, or inventory's row-remap state                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | the GPU is quarantined for replacement                                                                                                                                                                                    |
| `EccUncorrectable`, `NvlinkDegraded` (v0.2)            | inventory: volatile uncorrectable ECC errors, or NVLinks down on a GPU that has NVLink peers                                                                                                                                                                                                                                                                                                                                                                                                                         | the GPU is quarantined                                                                                                                                                                                                    |
| `DriverMismatch`, `CertExpiring`, `DaemonOutdated`     | the hygiene sweep, see below                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | NoSchedule, plus `node.condition.changed`                                                                                                                                                                                 |
| `CpuPressure`, `MemoryPressure`, `DiskPressure` (v0.3) | host pressure in status reports, see below                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | NoSchedule on `vps-host` pools; an alert elsewhere                                                                                                                                                                        |
| `FabricManagerDown` (v1.1)                             | bcd's `fabric-manager` unit: an NVSwitch system without Fabric Manager running, so CUDA cannot initialize multi-GPU jobs (c7613e1)                                                                                                                                                                                                                                                                                                                                                                                   | NoSchedule while `true`                                                                                                                                                                                                   |
| `MPSDown` (v1.1)                                       | bcd's `gpu-sharing` unit: an `mps` pool's MPS control daemon is not running, so every client fails (c7613e1)                                                                                                                                                                                                                                                                                                                                                                                                         | NoSchedule while `true`                                                                                                                                                                                                   |
| `RebootRequired` (v1.1)                                | bcd's host units: a setting takes effect only after a reboot; the reason names the units (0aea5c0), see below                                                                                                                                                                                                                                                                                                                                                                                                        | Informational                                                                                                                                                                                                             |
| `HostManagedExternally` (v1.1)                         | the `host` section's state on a node with `[host] units = false` (c7613e1)                                                                                                                                                                                                                                                                                                                                                                                                                                           | Informational                                                                                                                                                                                                             |
| `ForeignGpuExposure` (v1.1)                            | bcd on a Kubernetes worker: something besides bcd hands out GPUs (another `nvidia.com/*` device plugin, another GPU DRA driver in the node's ResourceSlices, or a container toolkit that accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods, including one the container runtime runs whose configuration is missing or unreadable), or, on a DaemonSet node, no container runtime configuration is readable, so bcd offers none (9f860ad, f88bcdb; see [GPU isolation](/docs/daemon/kubernetes#gpu-isolation)) | NoSchedule while `true` (ebd9806), and the node's active reservations are `Degraded` with reason `ForeignGpuExposure` (4570dfe)                                                                                           |
| `PodResourcesUnavailable` (v1.1)                       | bcd on a Kubernetes worker: it holds devices for Kubernetes and the kubelet's pod-resources socket is missing or failing, so released devices cannot be reclaimed from pods (9f860ad)                                                                                                                                                                                                                                                                                                                                | NoSchedule while `true` (ebd9806), for the whole node                                                                                                                                                                     |
| `KubernetesReleased` (v1.1)                            | bcd: the spec dropped `k8s-join`, and bcd cordoned and drained the Node (9f860ad)                                                                                                                                                                                                                                                                                                                                                                                                                                    | Informational                                                                                                                                                                                                             |
| `KubernetesTaintsDrift` (v1.1)                         | bcd's `k8s-join` unit: the Node's taints differ from the pool's, which the kubelet may not change; the message is the `kubectl taint` command a cluster admin runs (6ef198b)                                                                                                                                                                                                                                                                                                                                         | Informational                                                                                                                                                                                                             |
| `KubernetesReleaseStuck` (v1.1)                        | bcd: a released Kubernetes allocation still held after 5 minutes (by a pod, a prepared DRA claim or a pod process, or with the kubelet unreachable), so its release is not reported yet (cd5fc82, 8bcbe92)                                                                                                                                                                                                                                                                                                           | NoSchedule while `true` (ebd9806)                                                                                                                                                                                         |
| `KubernetesReleasesUnreadable` (v1.1)                  | bcd: its pending Kubernetes releases file (`releasing.json`) could not be read, so every GPU of the node stays claimed until no pod may hold a bcd GPU of an allocation the node does not know, or an operator removes the marker `<data_dir>/releasing.unreadable` (a18f423, b01130f; see [an unreadable releases file](/docs/guides/capacity/kubernetes#an-unreadable-releases-file)); reason `ReleasesFileUnreadable`                                                                                             | NoSchedule while `true` (4884a65)                                                                                                                                                                                         |
| `SpotInterruption` (v1.1, AWS)                         | bcd on a spot EC2 instance: AWS posted a spot instance-action (polled every 5 s from IMDSv2); the reason is `<action>:<deadline>`, e.g. `terminate:2026-11-05T10:02:00Z`                                                                                                                                                                                                                                                                                                                                             | NoSchedule only while `true` (a changed action or deadline is recorded and published again); false once AWS withdraws the notice. Draining the node's work is the EC2 interruption-warning event path, not this condition |

The gateway's `daemonNodeConditions` (b91dcff, 88219ea) lists the Kubernetes conditions, so they are written as node
conditions. Since ebd9806 (security review 2) three of them, and since 4884a65 a fourth, are NoSchedule while true on
a node that is not `tenant-controlled` (on a tenant-controlled node they stay advisory, as below), because each means
the node cannot hand out GPUs safely:

* **`ForeignGpuExposure`**: another path (the GPU operator's device plugin, another GPU DRA driver, or
  `NVIDIA_VISIBLE_DEVICES` from unprivileged pods) reaches the node's GPUs, or bcd cannot rule one out, so a hold
  booked there would not be isolated. A change of it also wakes
  the node's started reservations, which then show `Degraded` / `ForeignGpuExposure` at once (4570dfe; see
  [reservation conditions](/docs/guides/capacity/reservations#states-and-conditions)).
* **`PodResourcesUnavailable`**: bcd cannot tell which pods hold which devices. Placement has no NoSchedule for one
  binding only, so the whole node takes no work until the socket answers.
* **`KubernetesReleaseStuck`**: until ebd9806 it rode bcd's `StatusReport.conditions` and the `node.status` event
  only. Now it is recorded, and while a released allocation's devices are still held by a pod the node takes no more
  work, since the GPUs may still be in use; an operator looks first.
* **`KubernetesReleasesUnreadable`**: bcd could not read its pending releases, so every GPU of the node stays claimed
  on the node itself (no hold, VM, `gpu.reset` or `diag.dcgm` gets one). Until 4884a65 the gateway dropped it, so
  placement kept booking those GPUs and each new hold failed on the node with `overlap`. Now it is recorded and the
  node takes no work until the claim clears.

`KubernetesReleased` and `KubernetesTaintsDrift` stay informational. See
[Kubernetes workers](/docs/guides/capacity/kubernetes#gpu-isolation).

`FabricManagerDown`, `MPSDown`, `RebootRequired`, `HostManagedExternally`, `SpotInterruption` and the Kubernetes
conditions above come from bcd's [host configuration](/docs/daemon/host-config#status-and-failures), its cloud
metadata watch and its Kubernetes side. The gateway's
`xidStatusHandler` makes each the node's condition exactly as the last status report says, `true` while reported and
`false` once it is not, and writes and publishes `node.condition.changed` only on a change. On a `tenant-controlled`
node they are advisory: recorded, never a NoSchedule reason.

**RebootRequired.** Some host settings wait for a reboot: 1 GiB huge pages reserved on the kernel command line, CPUs
isolated with the `cpu` unit, the `iommu` unit, an ECC mode set with `gpu-settings`, a driver swap while the GPUs are
in use, and a MIG mode reset that failed. bcd never reboots on its own, and nothing in the control plane reboots on
this condition: reboot the node yourself with [`node.reboot`](/docs/guides/nodes/commands), within the pool's
disruption budget. The node keeps running and taking work meanwhile.

The condition table in `@better-compute/inventory` (`nodeConditionTypes`) also defines `PcieReplayHigh` and
`ClockSkew`. No controller raises those on nodes yet.

### Node hygiene [#node-hygiene]

Some faults no single report shows. The inventory module's `inventory.hygiene` reconciler, a periodic singleton,
sweeps every node that is not deleted or retired every 5 minutes, 500 nodes per transaction:

| Condition        | Raised when                                                                                                                                                                              |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DriverMismatch` | The node converged (`observed_generation >= generation`) on a spec whose `nvidia-driver` host unit wants branch B, while its GPUs report a driver of another branch.                     |
| `CertExpiring`   | The newest unrevoked certificate of the node's identity generation (ramdisk certificates aside) has less than a tenth of its lifetime left, and at least an hour, so renewal is failing. |
| `DaemonOutdated` | The session's `proto_max` is below the control plane's protocol version, or `bcd` runs below a configured `minDaemonVersion`.                                                            |

Each raised condition adds a NoSchedule reason and publishes `node.condition.changed` (`condition`, `status`,
`reason`, `noSchedule`); both go when the cause does. Only changes are written. The built-in reconciler uses the
defaults; `hygieneReconciler(options)` and `runHygiene(sql, options)` take `every`, `minDaemonVersion`,
`protocolMax`, `certExpiringFraction`, `certExpiringMinSeconds` and `batch`.

### Host pressure [#host-pressure]

Since protocol v0.3, `StatusReport.pressure` carries the host's Linux pressure stall information ("some" avg10
from `/proc/pressure/*`, percent of wall time) and CPU steal (percent of the last report interval); it is absent
when the kernel exposes no PSI. `recordHostPressure(tx, nodeId, report)` turns it into conditions:

| Condition        | Raised at                 | Cleared below |
| ---------------- | ------------------------- | ------------- |
| `CpuPressure`    | CPU PSI 50%, or steal 20% | half of each  |
| `MemoryPressure` | memory PSI 20%            | 10%           |
| `DiskPressure`   | I/O PSI 40%               | 20%           |

Between the two levels a condition keeps its status, so a host hovering at a threshold does not flap. On nodes of
`vps-host` pools the conditions are NoSchedule; elsewhere they only alert through `node.condition.changed`.
Nothing is written while nothing changes, and nothing at all for a `tenant-controlled` node (see
[trust](#trust-tenant-controlled-nodes-are-advisory)). The gateway runs it only when the host installs the status handler:

```ts
import { createGateway, xidStatusHandler } from 'better-compute/gateway';
import { hostPressureStatusHandler } from 'better-compute/inventory';

createGateway({ reports: { status: [xidStatusHandler, hostPressureStatusHandler({ events })] } });
```

`hostPressureStatusHandler({ thresholds })` takes other raise thresholds (`cpu`, `steal`, `memory`, `io`). In
tests, `FakeDaemon.setPressure()` puts a `HostPressure` into every status report.

bcd reports real values (`daemon/internal/health`). Every 30 seconds it reads the `some avg10` line of
`/proc/pressure/cpu`, `memory` and `io`, and computes CPU steal from the `cpu` line of `/proc/stat` as the share of
CPU time stolen since the previous sample (0 on the first). It puts the sample into its latest `StatusReport` and
sends that report again. A kernel without PSI yields no `pressure` at all, so no pressure condition is raised.

## Health, quarantines and remediation [#health-quarantines-and-remediation]

Not every fault should take a whole node away. A GPU with a double-bit ECC error is one bad atom on an
eight-GPU machine, while a GPU that fell off the bus usually needs a reboot of the node.

* **Device quarantine.** A device-level fault writes a `device_health` row (source `xid` or `inventory`) and
  marks the device `healthy: false`. The atom leaves placement; the rest of the node keeps working. An XID
  quarantine stays until it is lifted with `clearDeviceHealth(tx, deviceId, reason?)` (a function of
  `@better-compute/inventory`) or by a successful remediation; inventory quarantines follow each new inventory.
* **XID events.** The daemon reports XIDs as `Xid<n>` conditions on a GPU in its status reports. With the
  gateway's `xidStatusHandler` installed (`createGateway({ reports: { ...gatewayReportHandlers, status:
  xidStatusHandler } })`), each one goes through `recordXid`. XIDs without a policy (13, 31, 43, 45 and other
  application faults) are only recorded as events, and XIDs from a `tenant-controlled` node change nothing. When an
  XID quarantines a GPU, the handler also wakes the node's started reservations at once (04f6bda), so a reservation
  holding that GPU turns `Degraded` with reason `AtomUnhealthy`, naming the GPU, within a pass, rather than at
  capacity's periodic recheck or provision's health reconciler. Nothing is released, and the node keeps its hold; on
  a Kubernetes worker bcd's device plugins or DRA driver stop offering the GPU as healthy (6f5a99f).
  `tests/e2e/kubernetes-bindings.test.ts` runs this with an XID 48 on a Kubernetes-bound GPU.
* **XIDs in bcd.** bcd follows `/dev/kmsg` from its end and parses driver lines of the form
  `NVRM: Xid (PCI:0000:18:00): 79, ...`. It normalizes the bus id to NVML's form (`00000000:18:00.0`), finds the GPU
  with that bus id in its inventory, and raises condition `Xid79` (status `TRUE`, `device_id` the GPU's UUID, the
  driver's text as the message), one per XID and GPU. The status report goes out at once, and the condition stays in
  every later report. An XID for a bus id the inventory does not know is only logged. bcd keeps these conditions for
  the rest of the boot: they are stored with the boot id in `data_dir/faults.json` (mode 0600), so a restart of bcd
  in the same boot reports them again even though it reads the kernel log from its end, while the first start after
  a reboot finds another boot id and drops them. The reboot that remediation runs therefore clears them, and so does
  a `gpu.reset` that leaves the GPU healthy, for that GPU (662fb55). An XID also makes bcd collect its inventory again
  about 2 seconds later, since the GPU's ECC and row-remap counters move with it.
* **Remediation.** Node-level faults queue the `health` reconciler of the provision module. It waits until
  the node holds no live allocation, then runs a `node.reboot` workflow (`Xid79`, `Xid95`) or a GPU reset
  (`RowRemapPending`; bcd's `gpu.reset` refuses a GPU that still has a live hold, see
  [delivery](/docs/daemon/delivery#gpureset-and-live-holds)). Runs share the pool's disruption budget (`policy.disruptionBudget`, 5% of the pool and
  at least one node, together with workflows and rollouts), and a circuit breaker pauses remediation for a pool
  when more than 10% of its nodes turned unhealthy within 10 minutes (`pool.remediation.paused`). A successful
  reboot clears the XID conditions and their quarantines; a GPU reset is confirmed by the next inventory. A
  failed or cancelled run, or two runs that did not clear the fault, put the node in `maintenance` for an
  operator (`node.remediation.escalated`). Progress is published as `node.remediation.started` and
  `node.remediation.succeeded`.

## Acceptance checks [#acceptance-checks]

A node that just joined has not proven its hardware works. Acceptance runs the pool's checks before the node
becomes schedulable, and again whenever you ask.

```json title="pool policy"
{
  "acceptance": [{ "kind": "diag.nvml" }],
  "expect": { "gpuCount": 8 }
}
```

The NodeLifecycle controller runs them for a node in `adopting` once it is connected:

1. It takes the pool's `policy.acceptance` list (each kind once). A site agent, a node without a pool, or an
   empty list passes at once (reason `NoChecks`).
2. It enqueues one command per check, with idempotency key `acceptance:<nodeId>:<attempt>:<kind>` and a
   10-minute `not_after`, sets `AcceptanceFailed` to `unknown` (reason `Running`), and publishes
   `node.acceptance.started` with the attempt, command ids and checks.
3. `diag.nvml` checks the GPUs inventory found against `policy.expect.gpuCount` (else against the number
   found). Other kinds run with their default payload in this release; a kind that is not registered fails the
   attempt with `UnsupportedCheck`. On bcd (662fb55), `diag.nvml` fails `nvml.init` when the driver's management
   interface (`nvidia-smi`) did not answer, and per GPU fails `ecc.uncorrectable`, `row-remap.pending`,
   `row-remap.failure`, `pcie.link` (a link narrower than its maximum width) and `xid.recent` (an XID this boot),
   each with its reason; bcd advertises the `nvml` capability in `Hello` only when that data is there.
4. When every command settled: all succeeded (and `diag.nvml` reported `passed`) makes the node `adopted`,
   sets `AcceptanceFailed` to `false` (reason `Passed`), refreshes its IAM projection, asks the NodeSpec
   controller for generation 1 and publishes `node.adopted`. Otherwise `AcceptanceFailed` turns `true` with the
   reason (`Expired`, `diag.nvml:<check>`, `<kind>:failed`, …), the node stays `adopting` with the
   `AcceptanceFailed` NoSchedule reason, and `node.acceptance.failed` carries the reason and command id.

A failure is sticky: the controller does not retry on its own. Fix the cause, then call
[`nodes.adopt`](/docs/reference/api/nodes#adopt) (`compute:nodes:provision`), which starts a new attempt. On an
`adopted` node, `nodes.adopt` moves it back to `adopting`, so it leaves placement until the checks pass again.

`diag.nvml` is the v0.1 check. The `diag.dcgm` kind comes with the metal module, whose commissioning profile
runs it at level 2 (and the dedicated-root release at level 1); as a pool acceptance check it runs with its
default payload until check arguments (`with`) are mapped per kind. From level 2, bcd refuses it `gpu-busy` for a
GPU a hold or a VM is using (539c4ce).

On the node itself, `bcd doctor` prints one `gpu` line per GPU with its load, and warns on a pending row remap or MIG
mode change and fails on failed remapping or uncorrectable ECC errors (f9d7af9; see
[bcd doctor](/docs/daemon#modes)).


# Joining (/docs/guides/nodes/joining)

> Join tokens, the join flow, node certificates and renewal, adopting a brownfield server with the install script, and the join request approval queue.



Joining turns a machine into a [node](/docs/guides/nodes): JoinService checks the machine's evidence, creates or
reclaims the node row, and signs a short-lived client certificate that the daemon uses for every later
connection. This page covers joining with a join token, the v0.1 path for servers
you already run (a brownfield adopt). Bare-metal machines join through other
attestors; see [attestation](/docs/guides/nodes/attestation). A machine with a TPM 2.0 can
join with no token at all, `bcd join --attestor tpm` (8545688; see [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm)),
and a pool can refuse token joins altogether with `policy.joinAttestors` (see
[requiring an attestor](/docs/guides/nodes/attestation#requiring-an-attestor)).

> **bcd is not released yet.** 
  Better Compute is on npm, but the Go daemon `bcd` is built and tested and not released, so there is no binary for
  the install script to download unless you build and serve one yourself. The flow below is what the control plane implements; `bcd join` performs
  it in `tests/e2e/bcd.test.ts`, and `FakeDaemon` (and `better-compute dev --sim N`) exercises it everywhere else.

## Before you start [#before-you-start]

* **A pool that runs the daemon.** The node lands in one pool of one site. The pool must have `agent: 'bcd'`;
  `joinTokens.create` answers `INVALID_STATE` for a pool whose machines run no daemon. See
  [pools](/docs/guides/capacity/pools).
* **The machine PKI.** `better-compute pki init --trust-domain compute.example.com` writes the CAs and the two
  JoinService keys. Pass them to the server as `pki.join` (`pkiJoinCrypto({ trustDomain, nodeCa, signingKey,
  encryptionKey })`); without them every join fails with `INVALID_CONFIG`. See [PKI](/docs/operations/pki).
* **A gateway the machine can reach.** The daemon only dials out, to the regional gateway on port 443. See
  [the gateway](/docs/daemon/gateway).

## Join tokens [#join-tokens]

A join token is a single-purpose credential: it lets a limited number of machines join one pool, for a limited
time. Its bearer form is `bcjt_<id>.<secret>`, where the id is `jt_…` and the secret is 256 random bits in
base64url. The token is returned once, when it is created. The control plane stores only the SHA-256 of the
secret, so a database dump cannot be replayed as tokens.

### Creating a token [#creating-a-token]

[`joinTokens.create`](/docs/reference/api/join-tokens#create) is decided as `compute:join-tokens:create` on
`compute-pool/{poolId}`. Only the pool's owner tenant can mint tokens for it; any other tenant gets
`NOT_FOUND`.

```ts title="join-token.ts"
const created = await compute.api.joinTokens.create(credential, {
  tenantId: rootTenantId,
  poolId,
  ttlSeconds: 3600,
  maxUses: 1,
  expect: { serial: 'SN-7701' },
  idempotencyKey: 'adopt-rack-b7-u12',
});

// `bcjt_jt_….…`: hand it to the machine out of band. It is never shown again.
console.log(created.token);
```

The same call from the CLI's `join-tokens create` (44915e5), which writes the token to a new 0600 file with `--out`
instead of printing it:

```bash
better-compute join-tokens create --tenant $TENANT --pool $POOL --ttl 1h \
  --expect-serial SN-7701 --key adopt-rack-b7-u12 --out ./join-token
better-compute join-tokens list --tenant $TENANT --pool $POOL
better-compute join-tokens revoke jt_01k5w… --tenant $TENANT --pool $POOL
```

`--ttl` is a duration from `1m` to `24h` in whole seconds; `--max-uses`, `--allow-reclaim` and
`--expect-product-uuid`, `--expect-serial`, `--expect-mac` and `--expect-gpu-uuid` map to the fields below. `--out`
never writes over a file (`ALREADY_EXISTS`), and it refuses a replayed `--key`, whose answer carries no token
(`INVALID_STATE`: revoke that token and create a new one). `join-tokens list` never shows a secret (`--all` adds
revoked, expired and used-up tokens), and `join-tokens revoke` takes the token id and refuses a pasted token
(anything with a `.`), so a secret is never sent as an id.

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The pool owner tenant.', required: true },
  poolId: { type: 'string', description: 'The pool machines join. Its site becomes the token site.', required: true },
  ttlSeconds: { type: 'number', description: 'Lifetime, 60 to 86,400 seconds (one day).', required: true },
  maxUses: { type: 'number', description: 'How many machines may join with it, 1 to 10,000.', default: '1' },
  expect: { type: '{ productUuid?, serial?, mac?, gpuUuid? }', description: 'Binds the token to one machine; name at least one identifier. mac is aa:bb:cc:dd:ee:ff, gpuUuid starts with GPU-.' },
  allowReclaim: { type: 'boolean', description: 'Lets a known machine with a new key reclaim its node.', default: 'false' },
  hosted: { type: &#x22;'kubernetes'&#x22;, description: 'For bcd running as a DaemonSet on a Kubernetes cluster (a27156a): its nodes join tenant-controlled unless the pool has policy.kubernetes.platformOperated true and the platform owns the pool (259d82c). See /docs/daemon/kubernetes#trust.' },
  idempotencyKey: { type: 'string', description: 'Scoped to the tenant and the calling identity.', required: true },
}"
/>

The answer is the token record (`id`, `poolId`, `siteId`, `role: 'node'`, `usesLeft`, `expect`, `allowReclaim`,
`hosted`, `expiresAt`) plus `token`. A retry with the same idempotency key returns the record without `token`; the
same key with a different pool, `expect`, `allowReclaim` or `hosted` fails with `CONFLICT`. Creating a token
publishes `security.join-token.created`, whose `pinned` field says whether it has an `expect` binding, with `hosted`
when it is set.

A **hosted** token (`--hosted kubernetes` in the CLI) is for [bcd as a DaemonSet](/docs/daemon/kubernetes) on a
cluster whose admins have root on its machines through privileged pods. A node that joins with it is
`tenant-controlled`, with a `node-tenant` certificate that never receives platform secrets, unless the token's pool
has `policy.kubernetes.platformOperated: true` and the platform owns that pool (259d82c): its owner is the owner of
its site, the site is not `custom`, and, when the deployment names a root tenant, the owner is that tenant, so a
tenant cannot vouch for its own cluster by editing its pool's policy. The token decides, never the machine's claims, and it only lowers
`platform` trust (a tenant's own site stays `tenant-owned`); a reclaim through a hosted token lowers an existing
`platform` node too.

[`joinTokens.list`](/docs/reference/api/join-tokens#list) (`compute:join-tokens:read`) lists a pool's live
tokens, and every token with `includeInactive: true`; pages hold up to 200 (100 by default).
[`joinTokens.revoke`](/docs/reference/api/join-tokens#revoke) (`compute:join-tokens:revoke`) ends a token at
once and publishes `security.join-token.revoked`. Revoking twice is not an error.

### Token roles [#token-roles]

Every token row has a role, which decides what kind of agent it can admit.

| Role          | Admits                                  | Minted by                                                                                                                 |
| ------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `node`        | a `bcd node` into one pool              | `joinTokens.create`                                                                                                       |
| `site`        | a `bcd site` agent into one site        | [`siteTokens.create`](/docs/reference/api/site-tokens#create) (v0.2, `compute:sites:manage` on the site, at most 16 uses) |
| `provisioned` | the operating system a deploy installed | the provisioning engine's `userdata.render` step, single use                                                              |
| `discovery`   | a PXE-booted ramdisk                    | the bare-metal boot step, one per MAC and bound to it, single use                                                         |

A token presented for another role is refused. Modules mint the last two through `ctx.pki.issueJoinToken`, and
they travel through the control plane only as secret references, never in a command body. They are covered in
[attestation](/docs/guides/nodes/attestation).

### Binding a token to one machine [#binding-a-token-to-one-machine]

A token with `maxUses: 1` still admits whichever machine presents it first. `expect` pins it to one machine:
every identifier you name must be among the machine's claims. The comparison normalizes formats first:

* `productUuid` is compared lower case and dashed, and a placeholder UUID never matches;
* `serial` matches the product, board or chassis serial, case-insensitively;
* `mac` must be a permanent, globally administered MAC of a physical NIC (separators and case do not matter);
* `gpuUuid` is compared case-insensitively.

A mismatch refuses the join without consuming the token.

## The join flow [#the-join-flow]

<Mermaid
  chart="sequenceDiagram
  autonumber
  participant B as bcd
  participant G as Gateway
  participant J as JoinService in api
  participant P as Postgres
  B->>B: new P-256 key and CSR, seal the token
  B->>G: Join with sealed token, CSR and claims
  G->>J: relay over internal mTLS
  J->>P: one transaction
  Note over J,P: open seal, attestor, spend token, node row, certificate
  J-->>G: JoinResponse, signed
  G-->>B: JoinResponse
  B->>B: verify signature, CSR hash, key, SAN and role
  B->>G: Session with Hello
  G-->>B: Welcome"
/>

1. **Key and CSR.** The daemon generates a P-256 key and a PKCS#10 request. bcd makes the key in the host's TPM
   when it can open one, and its 0600 `node.key` under the data directory then holds only the key's record; otherwise
   `node.key` is the key itself, a PKCS#8 file (see
   [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)). Only a `tpm` join proves
   where the key lives: there the key is made in the attestor's TPM and the TPM's attestation key certifies it
   (cf24876; see [tpm](/docs/guides/nodes/attestation#tpm)).
2. **Sealed token.** The token is encrypted as a compact JWE (`ECDH-ES+A256KW`, `A256GCM`) to the JoinService
   encryption key pinned in the daemon's config. The protected header carries `typ: bcjt+jwe`, `iat` and `csr`,
   the SHA-256 of the request. The gateway relays the call but can neither read the token nor replay the sealed
   blob with a CSR of its own: JoinService refuses a seal made for another CSR.
3. **Join.** `Join` carries the sealed token, the CSR, the hardware claims (product UUID and serials, permanent
   MACs, GPU UUIDs, hostname), the attestor name (empty means `join-token`) and the requested role. It travels
   over TLS that authenticates the gateway against the pinned serving CA.
4. **Admission, in one transaction.** JoinService opens the seal, then the `join-token` attestor locks the token
   row and checks the secret (in constant time), revocation, expiry at `compute_now()`, the remaining uses and
   the role, and consumes one use. JoinService then enforces `expect`, the placeholder rule and the reclaim
   rules below, checks that the pool runs the daemon and belongs to the token's tenant, and inserts the node in
   `adopting` with `identityGeneration` 1. It writes the node's authorization projection row, publishes
   `node.joined` and writes the `node.joined` audit row. Both carry the attestor, whether the join was a reclaim
   (the audit action is then `node.reclaimed`), `hardwareKeyTrusted`, the certificate serial and `nodeKey`: `tpm`
   when the attestor proved the node key lives in the machine's TPM, else `unattested`, which is every token join
   (cf24876). A node in a pool without acceptance checks (and a site
   agent) is queued for its controllers at once and adopted; when the pool runs acceptance checks the node is
   still disconnected, so JoinService does not queue it, and its first `Hello` wakes the controller that starts
   acceptance.
5. **Signed response.** The answer is `JoinResponse{ body, sig }`, signed with the pinned join-signing key. The
   body holds the node id, the certificate chain, the command-signing trust bundle, `csr_sha256`, the trust
   domain, the site and the role. The daemon checks, in order: the signature under the pinned key, that
   `csr_sha256` is its own request's, that the leaf certifies its key, the SPIFFE SAN, and the role.
6. **Session.** The daemon opens its session with `Hello`. See [sessions](/docs/daemon/sessions).

No step of the join calls better-iam.

### Refusals [#refusals]

The agent never learns why a join failed: every refusal is the same `ACCESS_DENIED` ("Join refused"), so a
stolen or guessed token reveals nothing. The reason is recorded server-side as a `security.join.refused` event
with `attestor`, `reason` (`<attestor>:<reason>`), `role`, and hashes only (`csrSha256`, `hardwareKey`). The
event goes to the token's tenant only once the token's secret verified; before that it goes to the root tenant,
so an unauthenticated caller cannot write into another tenant's event stream. Events are rate-limited to 60 per
attestor and reason per minute, and each carries `suppressed`, the count dropped since the last one. The
reasons you will see for token joins:

| Reason                                               | Meaning                                                                                      |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `join-token:malformed`, `unknown`, `secret-mismatch` | Not a token, or not one of ours                                                              |
| `join-token:revoked`, `expired`, `exhausted`         | A real token that no longer admits anyone                                                    |
| `join-token:role-mismatch`                           | The agent asked for a role the token does not admit                                          |
| `join:token`                                         | The seal could not be opened, or was made for another CSR                                    |
| `join:expect`                                        | The claims do not match the token's `expect`                                                 |
| `join:placeholder-hardware`                          | Placeholder identifiers without an `expect` binding                                          |
| `join:reclaim`                                       | A known machine without `allowReclaim`, or in another tenant, site, pool or role             |
| `join:pool`, `join:tenant`, `join:site`              | The pool is missing or runs no daemon, belongs to another tenant, or the site does not match |

Two failures are not refusals: a missing JoinService configuration is `INVALID_CONFIG`, and two joins of the same
machine racing each other end with one `CONFLICT` ("This machine joined concurrently; retry").

## Node certificates [#node-certificates]

The certificate is the node's identity from here on. Its properties:

* **Key and lifetime.** ECDSA P-256, at most 24 hours (JoinService's `certTtlSeconds`, default and maximum
  86,400). Ramdisk certificates last at most 2 hours, and an attestor may shorten either.
* **Name.** Exactly one SPIFFE URI SAN, the SPIFFE ID
  `spiffe://<trustDomain>/site/<siteId>/node/<nodeId>`, first; any further URI SANs are `urn:bc:` statements about
  the node key, never identities. Today the only one is `urn:bc:tpm-ak:<nameAlg>:<digest>`, written after a
  certified `tpm` join (d94f1f3). Names and statements come only from JoinService, never from the CSR, and no tenant
  appears in the certificate.
* **Role.** One subject attribute `OU=bc-role:<role>` holds `node`, `site`, `ramdisk` or `node-tenant`. The
  certificate carries no private-arc OID, so Go's `crypto/x509` (and bcd) can parse it. See
  [PKI](/docs/operations/pki).
* **Record.** Every issued certificate is a `node_certs` row with its serial, identity generation, SPKI hash and
  DER, which is what later renewals and attested results are checked against.
* **Allowed kinds.** An attestor may restrict the command kinds a certificate accepts (`allowedKinds`: exact
  kinds or `prefix.*`). The command queue refuses other kinds at enqueue with `INVALID_STATE`, and the gateway
  checks again at dispatch. See [commands](/docs/guides/nodes/commands).

### Renewal [#renewal]

`bcd` renews at about 60% of the lifetime with `Renew`, proving possession of its **current** key. The request
carries a new CSR, the current chain, `client_time` and `pop_sig`: a signature by the current key under purpose
`renew-pop` over the CSR's SHA-256, the node id and the client time. JoinService in `api` verifies it; the
gateway only relays, so a compromised gateway cannot mint certificates. A renewal is refused unless:

* the presented leaf is byte for byte a `node_certs` row, unrevoked, unexpired and of the node's current identity
  generation;
* the node is not `retired`, and the caller's mTLS identity is that node;
* `client_time` is within 60 seconds of server time, and the same proof was not used in the last 120 seconds;
* when the leaf names a TPM attestation key (`urn:bc:tpm-ak:`), the request carries `key_attestation`: that AK's
  `TPM2_Certify` of the CSR's key over `sha256(csr)`, as a key the TPM made and never lets out (d94f1f3).

The new certificate keeps the role, the allowed kinds and the `urn:bc:` statements of the old one. JoinService reads
the statements from the presented leaf, which it has just matched to its own `node_certs` row, never from the CSR,
so a node can neither drop the AK statement nor renew with a key outside its TPM: a missing or failing attestation
is refused (`key-attestation-missing`, `key-attestation-ak`, `node-key-*`), never downgraded. bcd builds the
attestation itself and fails the renewal loudly, without sending it, when the TPM is unavailable or derives another
AK. See [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key) for the checks and for how
such a node recovers by joining again. A certificate without a statement (every join other than a certified `tpm`
join) renews with a proof of possession only: bcd still makes the next key in the TPM when it can, but nothing
proves it to the control plane.

Every refusal answers the agent with the same `ACCESS_DENIED` ("Renew refused"). Since 0e42018 JoinService also
publishes `security.renew.refused` with the reason, to the node owner once the caller proved the current key and to
the root tenant otherwise; see [refused renewals](/docs/operations/security#refused-renewals).

A certificate that expired while the machine was offline cannot be renewed; when the server has the `x509pop`
attestor configured, the machine re-joins with it within 7 days instead (see
[attestation](/docs/guides/nodes/attestation)). bcd keeps such an identity: it still loads, so `bcd join` answers
`already joined`, and `bcd doctor` fails its `identity` check with "the certificate expired at ...; join again as
root: bcd join --reset --yes, then bcd join". Since a1c1a46 that is the way back: `--reset --yes` moves the old
identity aside and the join that follows presents a new token (see
[joining again with --reset](/docs/daemon/install#joining-again-with---reset)):

```bash
# as root; the token from standard input, as for any join
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt
```

## Adopting a brownfield server [#adopting-a-brownfield-server]

      ### Create the PKI and the gateway certificate [#create-the-pki-and-the-gateway-certificate]

    ```bash
    BETTER_COMPUTE_SECRET=... better-compute pki init --trust-domain compute.example.com
    better-compute pki issue-gateway --san gw.eu.example.com
    ```

    `pki init` prints the pins the install script embeds (`pki show` prints them again). See
    [PKI](/docs/operations/pki).
  
      ### Create a join token [#create-a-join-token]

    Use `joinTokens.create` (or `better-compute join-tokens create --out`) as above, one token per machine with an
    `expect` binding when you know its serial or MAC.
  
      ### Generate the install script [#generate-the-install-script]

    ```bash
    better-compute daemon install-script --pool $POOL --site $SITE \
      --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
      --bcd-version 0.1.0 --sha256 amd64=<hex>,arm64=<hex> > install-bcd.sh
    ```

    Instead of `--sha256`, pass `--release-manifest` and `--release-pins` to pin the digests of a signed release
    manifest. The script holds no secret, so you can keep it in your provisioning repository.
  
      ### Run it on the server [#run-it-on-the-server]

    ```bash
    BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh
    # or, keeping the token out of the environment:
    sudo sh install-bcd.sh < token-file
    ```

    Download the script first: piping it into `sh` leaves no standard input for the token.
  
      ### Watch it join [#watch-it-join]

    The node appears in [`nodes.list`](/docs/reference/api/nodes#list) as `adopting`, and events `node.joined`,
    `node.connected`, `node.acceptance.started` and `node.adopted` follow.
  
The install script:

* reads the token only from `BCD_JOIN_TOKEN` or standard input, never from arguments or a URL, and refuses
  anything that does not look like `bcjt_…`;
* downloads `bcd` for the machine's architecture from `{base-url}{base-path}/downloads/bcd/{version}/linux-{arch}`
  and checks its SHA-256;
* installs these files, then runs `bcd join` with the token on standard input and enables `bcd.service`.

      - `bcd`
  
      - `config.toml (0600)`

    - `join-encryption.jwk (0644, public)`
  
      - `bcd.service (hardened unit)`
  
```ini title="/etc/bcd/config.toml"
gateway_url = "https://gw.eu.example.com"
site = "site_…"
serving_ca_spki_pin = "sha256/…"
join_signing_key_pin = "sha256/…"
join_encryption_key_pin = "sha256/…"
```

The token is consumed by the join and never written to disk. Run without a token on a host that already has
`/etc/bcd/config.toml`, the script upgrades `bcd` in place and restarts it.

### The join key and its fallback [#the-join-key-and-its-fallback]

The pins alone cannot seal a token, so the script also writes the JoinService encryption key to
`/etc/bcd/join-encryption.jwk`. `bcd join` uses it only when its SPKI hash equals `join_encryption_key_pin`.
When the file is missing or the key was rotated, the daemon fetches `GET {basePath}/.well-known/better-compute`,
whose `join` section the server publishes when `pki.join` is configured:

```json title="GET /api/compute/.well-known/better-compute"
{
  "name": "better-compute",
  "version": "0.1.0",
  "basePath": "/api/compute",
  "protocol": { "min": 1, "max": 1 },
  "join": {
    "encryptionJwk": { "kty": "EC", "crv": "P-256", "x": "…", "y": "…" },
    "encryptionKeyPin": "sha256/…",
    "signingCertificate": "MIIB…",
    "signingKeyPin": "sha256/…"
  }
}
```

The daemon accepts these keys only when they match its pins, so the fetch needs no trust in the transport.

bcd implements half of it. Its configuration has no control-plane URL, so the fetch happens only when
`join-encryption.jwk` is absent and `bcd join --server https://…` names the control plane: bcd then reads
`<server>/.well-known/better-compute` and uses `join.encryptionJwk` if it matches `join_encryption_key_pin`. That is
how an OS a deploy installed joins (`--server`, `--token-file` and `--node-id`, from the deploy's user data), and how
[bcd as a DaemonSet](/docs/daemon/kubernetes) joins a cluster's nodes. `--token-file` takes only a file its owner
alone can read (0600 or 0400; a Kubernetes Secret needs `defaultMode: 0400`), a regular file of at most 4 KiB, and
deletes it after the join; a read-only Secret mount it cannot delete is a warning, not a failure (c29bb16). A key
file that is present but does not match the pin still fails `bcd join`: after rotating the key, generate the install
script again and rerun it, which rewrites the file and the pins.

When there is no `config.toml` at all, `bcd join --server` goes further and builds its configuration from the
document: the gateway from its `gateway` section, which the server publishes when its options set
`gateway: { url, serverName? }` (with `servingCaPin` computed from `pki.servingCa`). The pins do not come from the
document: a host build requires `--serving-ca-pin`, `--join-signing-pin` and `--join-encryption-pin` from the install
and refuses a document whose pins differ. It joins, then writes `config.toml` and `join-encryption.jwk`. A provider
server installed with a token, the control plane's URL and the pins (from `bootstrapPins(bundle)`) joins this way;
see [joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document).

```json title="the gateway section of the document"
{
  "gateway": { "url": "https://gw.eu.example.com", "serverName": "gw.eu.example.com", "servingCaPin": "sha256/…" }
}
```

> **The well-known document alone would be trust on first use.** 
  The document is authenticated only by the control plane's TLS certificate. Taking its pins would be trust on first
  use: whoever can serve that name with a publicly trusted certificate could point the machine at their own gateway and
  JoinService keys and take its token. Pins that travel with the install (the install script, the ramdisk's pinned
  config for `deploy.image`, user data or a Robot unit carrying `bootstrapPins(bundle)`) are stronger, so a host build
  of bcd bootstraps only with them. Only a fakehw development build joins with no pins, on trust.

## Placeholder hardware and reclaims [#placeholder-hardware-and-reclaims]

JoinService derives the machine's hardware key from its claims, exactly as the
Inventory controller does from inventory. Firmware often reports placeholder values instead of real identifiers
(`To Be Filled By O.E.M.`, `Default string`, `System Serial Number`, all-zero UUIDs, one repeated character).
They are dropped before hashing, and the key is **trusted** only when at least two independent identifiers
remain (product UUID, a serial, permanent MACs, GPU UUIDs).

* **Placeholder hardware.** When the key is missing or untrusted, or neither a product UUID nor a serial
  survived, a `join-token` join is refused (`placeholder-hardware`) unless the token has an `expect` binding. With
  one, the node joins with an untrusted key. Machines like this can also be admitted through operator approval
  with the `discovery` or `tpm` attestors (see below).
* **Reclaims.** When a machine with a trusted key that already belongs to a node joins with a new key (a
  reinstall, a replaced disk), the join is a reclaim. It needs a token with `allowReclaim`, in the same tenant,
  site and pool, for the same role; otherwise it is refused (`reclaim`). The node keeps its id, its identity
  generation goes up by one, every older certificate is revoked (`reclaimed`, with a
  `security.certificate-revoked` event), acceptance runs again, and the audit row says `node.reclaimed`. A
  machine with fewer than two identifiers never reclaims: it joins as a new node. In a pool whose
  `policy.joinAttestors` does not list `join-token`, a token join or reclaim is refused (`pool-attestor`, 1516728).

## Join requests and approvals [#join-requests-and-approvals]

Some joins need a person to decide: a PXE-booted machine nobody expected, or a TPM whose endorsement key is not
on any allow-list. The attestors that defer (`discovery` with `approval: 'unknown-machines'` or `'always'`,
and `tpm` with `enrollment`) record a **join request** and answer the daemon with a signed deferral instead of
a certificate. Nothing is issued and no token is spent.

<Mermaid
  chart="sequenceDiagram
  participant B as Agent
  participant J as JoinService
  participant O as Operator
  B->>J: Join (CSR A)
  J-->>B: pending, request id, retry after 30 s
  O->>J: joinRequests.approve
  B->>J: Join again (same CSR A)
  J-->>B: certificate, request consumed"
/>

The request records the machine (keyed by its trusted hardware key or TPM endorsement key hash, else by site and
MAC), the claims, and the SHA-256 of the CSR the retry must present. A machine has at most one pending request;
a newer attempt refreshes it with its new CSR. An approval admits exactly one join: the retry with that CSR
consumes it inside its own join transaction. A rejected request makes a retry with that CSR fail.

| State      | Meaning                                                                             |
| ---------- | ----------------------------------------------------------------------------------- |
| `pending`  | Waiting for a decision (24 hours by default, the attestor's `pendingTtlSeconds`)    |
| `approved` | Admits one retry until it expires (`ttlSeconds`, 60 to 86,400, one hour by default) |
| `rejected` | Refused; a later decision cannot change it                                          |
| `expired`  | Pending or approved past its expiry                                                 |
| `consumed` | The approved join happened                                                          |

The `joinRequests` group (v0.2) is decided on the pool, like join tokens:

| Method                                                              | Action                       | Notes                                                                                                                           |
| ------------------------------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| [`joinRequests.list`](/docs/reference/api/join-requests#list)       | `compute:join-tokens:read`   | `state` defaults to `pending`; `all` lists every state. Shows the pool's requests and its site's requests no pool claimed yet.  |
| [`joinRequests.approve`](/docs/reference/api/join-requests#approve) | `compute:join-tokens:create` | Assigns the pool. Approving twice returns the approval; any other decided state is `INVALID_STATE`, and so is an archived pool. |
| [`joinRequests.reject`](/docs/reference/api/join-requests#reject)   | `compute:join-tokens:create` | Also withdraws an approval the machine has not used yet.                                                                        |

```ts title="approve.ts"
const { items } = await client.joinRequests.list({ tenantId, poolId });
for (const request of items) {
  console.log(request.id, request.attestor, request.mac, request.vouchedBy, request.claims);
}

await client.joinRequests.approve({
  tenantId,
  poolId,
  requestId: items[0].id,
  ttlSeconds: 900,
  reason: 'rack B7 intake, ticket 4411',
});
```

From the command line (573b0b0), where `approve --ttl` is in seconds (60 to 86,400):

```bash
better-compute join-requests list --tenant $TENANT --pool $POOL            # pending by default; --state all
better-compute join-requests approve jreq_01k5w… --tenant $TENANT --pool $POOL --ttl 900 \
  --reason 'rack B7 intake, ticket 4411'
better-compute join-requests reject jreq_01k5w… --tenant $TENANT --pool $POOL --reason 'unknown serial'
```

A request shows what the attestor saw: `attestor`, `claims`, `hardwareKey` (hex), `mac`, `ip`, `workflowId`,
`vouchedBy` and `bootLeaseEpoch` (the site agent that vouched for a discovered machine), `csrSha256`, and the
decision fields `decidedAt`, `decidedBy`, `reason` and `consumedAt`. Decisions publish
`security.join-request.approved` and `security.join-request.rejected`.

## Troubleshooting [#troubleshooting]

  #### The daemon says Join refused

    Read the pool owner's `security.join.refused` events (or the root tenant's, when the token itself did not
    verify). The `reason` names the check that failed. A refused join never consumes the token: the whole join
    is one transaction, and a refusal rolls it back.
  
  #### joinTokens.create answers INVALID_STATE

    The pool has `agent: 'none'`: its machines are reached through a provider API and run no daemon.
  
  #### Every join fails with INVALID_CONFIG

    The server has no JoinService keys. Pass `pki.join`, built with `pkiJoinCrypto` from the bundle
    `pki init` wrote.
  
  #### The node joined but stays adopting

    Acceptance has not passed. Look for the `AcceptanceFailed` condition on `nodes.get` and the
    `node.acceptance.failed` event, then fix the cause and call `nodes.adopt`. See
    [inventory and health](/docs/guides/nodes/inventory-and-health).
  
  #### bcd logs renewal failed: the TPM is unavailable or names another AK

    The certificate came from a certified `tpm` join and names its AK, so bcd renews only with that TPM. Restore
    access to the TPM (`/dev/tpmrm0` for a root `bcd node`; under `[helper]`, a running `bcd helper`, which holds the
    TPM for the node) and the next retry renews. If the TPM was cleared or replaced, run
    `bcd join --reset --yes --attestor tpm` as root: an unchanged EK reclaims the same node. For a new EK, cordon
    the old node and delete it first (`better-compute nodes delete <node> --yes`, which frees its EK binding and
    hardware key), then join: the machine joins as a new node. See
    [recovering a node whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed).
  
  #### bcd join answers already joined, but the node cannot connect

    `bcd join` does nothing on a machine whose identity loads, whatever its certificate's state: expired, revoked
    (`nodes.revokeIdentity`, `nodes.delete`) or refused at renewal. `bcd doctor` shows which (`identity` fails for
    an expired certificate or an identity that does not load). Run the join again as root with `--reset --yes` and
    the join's credential; the old `identity.json`, `node.key`, `node.key.next` and `journal.db` are renamed
    `<name>.reset-<UTC time>`, not deleted. Under `[helper]` the root join hands `identity.json` and `node.key` to
    the `[helper]` user when it finishes, so `bcd node` can read them. See
    [joining again with --reset](/docs/daemon/install#joining-again-with---reset).
  

# Operating nodes (/docs/guides/nodes/operating-nodes)

> Cordon, drain, taint and label nodes, understand maintenance, liveness and connectivity, revoke a node's identity, and delete nodes, with the nodes API.



Sooner or later every node needs attention: a firmware update, a flaky GPU, a rack move, a decommission. The
`nodes` API gives you the controls, and each one moves exactly one of the node's
[independent axes](/docs/guides/nodes), so you always know what a control did and what it left alone.

Every `nodes` method except `list` is decided on `compute-node/{nodeId}`, and a tenant sees the nodes the
authorization projection holds for it; a node it cannot see is `NOT_FOUND`. Mutations accept an optional
`version`: pass the version you read, and a node that changed since fails with `CONFLICT` instead of being
overwritten. Most also take a free-text `reason` (up to 256 characters) that ends up in the event.

## Cordon and uncordon [#cordon-and-uncordon]

A cordon stops new work from landing on a node and changes nothing else: existing
allocations keep running, commands still flow, the node stays connected.

```ts
await client.nodes.cordon({ tenantId, nodeId, reason: 'BIOS update window' });
// … later
await client.nodes.uncordon({ tenantId, nodeId });
```

```bash
better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'BIOS update window'
better-compute nodes uncordon node_01k5w… --tenant $TENANT
```

The `nodes` commands (44915e5) take the node id as their argument and `--if-version N` for the `version`
check; a node id that does not look like `node_…` is a usage error before anything is sent.

[`nodes.cordon`](/docs/reference/api/nodes#cordon) (`compute:nodes:cordon`) sets `cordoned` and publishes
`node.cordoned`; cordoning a cordoned node returns it unchanged. [`nodes.uncordon`](/docs/reference/api/nodes#uncordon)
(same action) clears `cordoned`, ends a drain in progress, and publishes `node.uncordoned`.

## Drain [#drain]

A drain is a cordon that also waits for the node to empty, so you know when it is safe to
work on it. [`nodes.drain`](/docs/reference/api/nodes#drain) (`compute:nodes:drain`) cordons the node, adds the
`draining` NoSchedule reason, publishes `node.draining`, and hands the node to the Drain controller.

The Drain controller does not evict anything in this release. It waits until the node holds no live allocation
(every allocation released, or past the end of its interval), looking again at the earliest allocation end and
at least every 5 minutes. Then it removes `draining` and publishes `node.drained`. The node stays cordoned until
you call `nodes.uncordon`. Forced release within the pool's disruption budget is planned for v0.4.

```ts
await client.nodes.drain({ tenantId, nodeId, reason: 'replace PSU' });
// Wait for the node.drained event (see the events guide), or poll:
const node = await client.nodes.get({ tenantId, nodeId });
const drained = node.cordoned && !node.noScheduleReasons.includes('draining');
```

```bash
better-compute nodes drain node_01k5w… --tenant $TENANT --reason 'replace PSU'
better-compute nodes events node_01k5w… --tenant $TENANT --type node.drained
```

Draining a node that is already draining changes nothing. To shorten a drain, release the reservations on it;
see [reservations](/docs/guides/capacity/reservations).

## Taints [#taints]

A taint is a named NoSchedule reason you own. Use it when you want a node out of placement
for a reason that should stay visible, next to the conditions and liveness reasons the controllers write.

```ts
await client.nodes.taint({ tenantId, nodeId, add: ['rma-pending'], remove: ['burn-in'] });
```

```bash
better-compute nodes taint node_01k5w… --tenant $TENANT --add rma-pending --remove burn-in
```

[`nodes.taint`](/docs/reference/api/nodes#taint) (`compute:nodes:update`) adds and removes up to 32 keys each,
stored in `noScheduleReasons` as `taint:<key>`. A key is 1 to 128 letters, digits and `. _ / = : -`, starting
and ending with a letter or digit, and cannot be added and removed in one call. The method only ever moves
operator taints: conditions, `unreachable` and `draining` belong to their controllers. It publishes
`node.tainted`. Any entry in `noScheduleReasons` keeps a node out of placement, with one exception.

**Dedicating a node to a tenant.** The taint `dedicated=<tenantId>` (stored as `taint:dedicated=acme`) keeps every
other tenant off the node but not that tenant: placement for `acme` tolerates a dedication to `acme` and nothing
else, on every placement path (inline, shortlist, pool-lease index, bulk commits and BookingRepair) and for
whole-node leases (`binding: 'node'` reservations in a `tenantRoot` pool, see
[dedicated-root metal](/docs/guides/bare-metal/dedicated-root)), which ignore every other NoSchedule reason: a node
dedicated to `acme` is never leased by another tenant. A dedication together with any other reason, or
dedications to two tenants, blocks everyone. Capacity rollups still count a dedicated node as NoSchedule (bucket
`unhealthy`). A node that becomes schedulable for its one dedicated tenant still fires the wake edge, so that
tenant's pending reservations are placed at once instead of on their 30-second retry.

```ts
await client.nodes.taint({ tenantId, nodeId, add: ['dedicated=acme'] });
```

## Labels [#labels]

Labels describe a node for placement: selectors in reservations match device attributes and node labels (see
[placement](/docs/guides/capacity/placement)). Inventory writes its own labels (`gpu.model`, `cpu.arch`, …);
[`nodes.label`](/docs/reference/api/nodes#label) (`compute:nodes:update`) manages yours.

```ts
const node = await client.nodes.label({
  tenantId,
  nodeId,
  set: { rack: 'b7', 'fabric/rail': '3' },
  remove: ['burn-in'],
});
```

`set` takes up to 64 labels and `remove` up to 64 keys. A key is up to 128 characters of letters, digits and
`. _ / -`, starting and ending with a letter or digit; a value is up to 256 characters of letters, digits and
`. _ : / @ + = -`. Setting a key that inventory or the system owns is `INVALID_INPUT`, and
`remove` only removes operator labels. Since c1fd430 any `bc.dev/` key is refused in `set` and `remove` alike: those
labels are bcd's advertised drivers and capabilities, and placement, metal, provision and vps count one only with
source `system` (see [capability labels](/docs/operations/security#sandboxes-v05)). The call returns the node
with its labels and publishes `node.labeled`.

```bash
better-compute nodes label node_01k5w… --tenant $TENANT --set rack=b7,fabric/rail=3 --remove burn-in
better-compute nodes list --tenant $TENANT --cordoned
```

`--set` takes `KEY=VALUE` pairs and `--remove` keys, both comma-separated; one of them is required. The console's
node page has the same controls: it lists the node's labels (key, value, source) and taints, sets and removes
operator labels, and adds and removes taints, gated on the node's state and IAM's answer on `compute:nodes:update`
(3ff9fb3; see [the console](/docs/guides/clients/console)).

## Maintenance [#maintenance]

`maintenance` is a separate axis from `cordoned`. It marks a node that needs an operator, and a node in
maintenance is reported in the `maintenance` capacity bucket and never scheduled. In this release it is turned
on by the system, not by an API call:

* a bare-metal workflow that fails or is aborted sets `provisionState: 'failed'` and turns maintenance on (see
  [bare-metal workflows](/docs/guides/bare-metal/workflows));
* health remediation that fails, is cancelled, or does not clear the fault in two attempts turns maintenance on
  and publishes `node.remediation.escalated` (see [inventory and health](/docs/guides/nodes/inventory-and-health)).

## Liveness and connectivity [#liveness-and-connectivity]

A node's `connectivity` says whether it has a live session, not whether it is healthy or whether its workloads
run. The daemon sends a heartbeat every 10 seconds, the gateway keeps the last-seen time in memory and flushes
ages to Postgres every 15 seconds, and the node's liveness lease lasts 40 seconds.

<Mermaid
  chart="stateDiagram-v2
  [*] --> disconnected: joined
  disconnected --> connected: Hello
  connected --> disconnected: session closed or gateway lost
  disconnected --> unreachable: 40 s plus grace
  unreachable --> lost: lostAfter, provider and vps-host pools only
  unreachable --> connected: Hello
  lost --> connected: Hello"
/>

| Transition        | When                                                                                                                                                           | Effect                                                                                                                               |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| to `connected`    | the node's `Hello`                                                                                                                                             | `node.connected`; clears the `unreachable` reason and the `NodeUnreachable` condition                                                |
| to `disconnected` | the session closes, or its gateway's lease expires                                                                                                             | `node.disconnected` with reason `gateway-lost` or `no-session`                                                                       |
| to `unreachable`  | 40 seconds after the node was last seen, plus grace; a node that joined but never opened a session counts from its join (its newest certificate's `issued_at`) | adds the `unreachable` NoSchedule reason, sets `NodeUnreachable` (`HeartbeatTimeout` or `GatewayLost`), publishes `node.unreachable` |
| to `lost`         | the pool's `policy.lostAfter` after it became unreachable                                                                                                      | `node.lost`                                                                                                                          |

Grace keeps an infrastructure problem from looking like a fleet of dead machines: 120 seconds after a gateway
loss, so sessions can move to another gateway, and 5 minutes after the Liveness controller or the database was
down for more than 20 seconds, so ages that could not be flushed meanwhile do not count against nodes.

"Last seen" is the later of the session's flushed `last_seen` and `disconnected_at`, else its `connected_at`. A node
that has joined but whose first `Hello` is still queued (a whole rack powering on at once) has none of these yet, so
both the Liveness sweep and the per-node check fall back to its newest `node_certs.issued_at`: it gets the same 40
seconds from its join instead of being marked unreachable the moment it joins.

### Disconnected is not dead [#disconnected-is-not-dead]

None of these transitions releases anything. Allocations, bookings and queued commands stay exactly as they
were: an unreachable node is only kept out of new placement. A partitioned server usually keeps running its
workloads, and releasing its GPUs to someone else would double-book hardware that is still in use. `lost` is
applied automatically only to nodes of `provider` and `vps-host` pools that set `policy.lostAfter`, never to bare
metal (a `vps-host` pool opts in so that its hosted instances can be recreated elsewhere). When the
node reconnects, it resumes where it left off: the gateway replays its unfinished commands and the desired state
it missed (see [delivery](/docs/daemon/delivery)).

## Revoking a node's identity [#revoking-a-nodes-identity]

When you suspect a node's key is compromised, or you are handing the machine to someone else, revoke its
identity. [`nodes.revokeIdentity`](/docs/reference/api/nodes#revokeidentity) (`compute:nodes:revoke-identity`):

* revokes every live certificate of the node (with your `reason`, or `identity revoked`);
* raises `identityGeneration`, so no certificate of the old generation can renew or re-join with `x509pop`;
* publishes `security.node.identity-revoked` and returns `{ node, revokedCertificates }`.

A gateway that runs a revocation feed (`createRevocationFeed` from `@better-compute/pki`, which polls
`node_certs` every second by default) learns about the revocation from Postgres and closes the node's live
session at once; without a feed, a revoked certificate keeps its session until it expires. The machine can come
back only by joining again with fresh evidence, such as a new join token.

The gateway revokes a certificate on its own when it looks cloned: after more than 3 alternations between two
machines within 10 minutes it quarantines the certificate with reason `duplicate-identity` and publishes
`security.node.quarantined`. Watch `security.node.duplicate-session` to see it coming, then find the copy and
re-join the real machine. See [duplicate identity](/docs/daemon/sessions#duplicate-identity).

## Adopting again [#adopting-again]

[`nodes.adopt`](/docs/reference/api/nodes#adopt) (`compute:nodes:provision`) re-runs acceptance on a node in
`adopting` or `adopted`; any other state is `INVALID_STATE`. An adopted node goes back to `adopting` and leaves
placement until the checks pass. See [acceptance checks](/docs/guides/nodes/inventory-and-health).

## Deleting a node [#deleting-a-node]

[`nodes.delete`](/docs/reference/api/nodes#delete) (v0.2, `compute:nodes:delete`) removes a node that is done
for good or never got going. It accepts only:

* a `retired` node;
* an `adopted` node that is cordoned (54cad9e);
* a node still in `adopting` or `enrolling`, or one that failed during enrollment;
* and only when the node has no open allocation (`INVALID_STATE` with the count otherwise).

Any other state is `INVALID_STATE` with `{ provisionState, cordoned }` in its details. An adopted node that is not
cordoned gets "Cordon the adopted node first (nodes.cordon), then delete it once it holds nothing".

In one transaction it makes the node's devices unbookable, removes its scalar resources and rebuilds the pool's
capacity rollup, revokes its certificates (`node-deleted`), frees its attestor bindings and trusted hardware key,
cancels its pending commands, and removes it from every tenant's projection. The row itself stays as a tombstone
(`deletedAt`), because audit rows, certificates, inventory snapshots and past allocations still reference it.
The machine can then join again as a new node. A retry of the same delete returns `{ nodeId, deleted: false }`,
and `node.deleted` is published once.

For bare-metal nodes, retire through the metal workflow first; see
[bare-metal workflows](/docs/guides/bare-metal/workflows).

### Removing an adopted host [#removing-an-adopted-host]

An adopted host (bcd installed on a machine metal does not manage) has no retire workflow, so the retire path for it
is a cordon and then the delete. Use it when the machine is decommissioned, or when it must join again under a new
identity, for example after its TPM or disk was replaced (see
[recovering a node whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed)):

1. Cordon the node, so nothing new lands on it. [Drain](#drain) it instead when you want to wait for its
   allocations to end.
2. Wait until it holds no open allocation: release its reservations, or let them end.
3. Delete it. Its certificates are revoked, and its attestor bindings (a `tpm` join's EK binding included) and its
   trusted hardware key are freed.
4. On the machine, as root, join again with `--reset --yes` and the join's credential (a new token, or
   `--attestor tpm`). The delete revoked the old certificate but left the files on the machine, and an identity that
   still loads would only answer `already joined`; `--reset` moves it aside first (see
   [joining again with --reset](/docs/daemon/install#joining-again-with---reset)). The machine joins as a new node.

```bash
better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'TPM replaced'
# ... once the node holds no open allocation:
better-compute nodes delete node_01k5w… --tenant $TENANT --reason 'TPM replaced' --yes
# then on the machine, as root:
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm
```

[`nodes delete`](/docs/reference/cli#nodes-delete) (4049621) takes `--reason` and `--if-version` like the other node changes, and without `--yes` it
refuses with a usage error before making any call (`nodes delete removes <node> for good; re-run with --yes`). From a
client:

```ts
await client.nodes.cordon({ tenantId, nodeId, reason: 'TPM replaced' });
// ... once the node holds no open allocation:
await client.nodes.delete({ tenantId, nodeId, reason: 'TPM replaced' });
```

> **A new node id and the old journal.** 
  `bcd node` refuses a journal of another node ("the journal belongs to node `<old>`, not `<new>`"), and a plain
  `bcd join` moves an old journal aside only when the old identity still loads. After `--reset` it does not, so
  `--reset` moves `journal.db` aside with the identity (`journal.db.reset-<UTC time>`, b4fcc2e) and the new node starts
  a journal of its own. Under `[helper]` the root join also hands `identity.json` and `node.key` to the `[helper]` user
  when it finishes, so no `chown` is needed before restarting `bcd-helper.service` and `bcd.service`.

## The nodes API [#the-nodes-api]

| Method                                                                                         | Action                                            | Phase | What it does                                                                                         |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------- |
| [`list`](/docs/reference/api/nodes#list)                                                       | `compute:nodes:read` on the tenant, then per item | v0.1  | Filters: `poolId`, `siteId`, `cordoned`, `provisionState`; up to 50 per page                         |
| [`get`](/docs/reference/api/nodes#get)                                                         | `compute:nodes:read`                              | v0.1  | The node with labels, conditions and its session                                                     |
| [`events`](/docs/reference/api/nodes#events)                                                   | `compute:nodes:read`                              | v0.1  | The node's events, oldest first; `type` filters by prefix (`node.`, `security.`), up to 200 per page |
| [`cordon`](/docs/reference/api/nodes#cordon), [`uncordon`](/docs/reference/api/nodes#uncordon) | `compute:nodes:cordon`                            | v0.1  | Stop or resume new placement                                                                         |
| [`drain`](/docs/reference/api/nodes#drain)                                                     | `compute:nodes:drain`                             | v0.1  | Cordon and wait for the node to empty                                                                |
| [`label`](/docs/reference/api/nodes#label), [`taint`](/docs/reference/api/nodes#taint)         | `compute:nodes:update`                            | v0.1  | Operator labels and NoSchedule taints                                                                |
| [`adopt`](/docs/reference/api/nodes#adopt)                                                     | `compute:nodes:provision`                         | v0.1  | Re-run acceptance                                                                                    |
| [`revokeIdentity`](/docs/reference/api/nodes#revokeidentity)                                   | `compute:nodes:revoke-identity`                   | v0.1  | Revoke every certificate, new identity generation                                                    |
| [`delete`](/docs/reference/api/nodes#delete)                                                   | `compute:nodes:delete`                            | v0.2  | Tombstone a retired, cordoned adopted, or never-adopted node                                         |
| [`issueSshCert`](/docs/reference/api/nodes#issuesshcert)                                       | `compute:nodes:ssh-root`                          | v0.3  | Break-glass SSH certificate; see [SSH](/docs/guides/access/ssh)                                      |

To run diagnostics, collect logs or reboot a node, use [commands](/docs/guides/nodes/commands).


# AI platform (/docs/guides/vps/ai-platform)

> DigitalOcean's AI platform as cloud resources. Dedicated Inference endpoints booked against the GPU quota, sealed model access keys and a token-counting inference proxy, knowledge bases and guardrails, the Managed Agents previews (harness agents on containers or MicroVMs, action gateways), and their read-only signals for an attention list.



DigitalOcean's AI platform reaches tenants through the same `cloudResources` calls as its managed databases and
apps: a kind per product, created on a provider pool, converged by the `cloud.resources` reconciler. This page covers
what is particular to the AI kinds: GPUs booked against a quota without a node to place them on, keys that are only
ever sealed, and products DigitalOcean still marks as previews. Set up the account and the pool first, as in
[DigitalOcean](/docs/guides/vps/providers#digitalocean).

| Kind                     | What it is                                                              | Changes in place                                                         |
| ------------------------ | ----------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `do.dedicated-inference` | A model served on dedicated GPUs, between `replicas` and `maxReplicas`  | `replicas`, `maxReplicas`                                                |
| `do.inference-key`       | A model access key for serverless and dedicated inference (sealed)      | `rotation` (regenerates the key)                                         |
| `do.knowledge-base`      | An embedding model over web and Spaces sources, indexed into OpenSearch | `webSources`, `spacesSources` (then reindexes)                           |
| `do.guardrail`           | A jailbreak, sensitive-data or content-moderation check                 | `defaultResponse`, `description`                                         |
| `do.agent-harness`       | A Managed Agents harness agent, on a container or a MicroVM (preview)   | `model`, `instructions`, `knowledgeBases`, `guardrails`, `actionGateway` |
| `do.action-gateway`      | The tools a harness agent may call, as https upstreams (preview)        | `upstreams`, `description`                                               |

Every other spec change is refused with `replaceRequired`: create a new resource. No agents are deployed through
DigitalOcean's ADK, which is deprecated; the Managed Agents harness is the supported way to run one.

> 
  The request and response shapes of these products follow DigitalOcean's API reference, and some
  paths are assumed (guardrails, the Managed Agents and Signals endpoints). The provider's emulator
  tests pin those shapes, but the live suite has not yet recorded real bodies, so treat the kinds as
  experimental until it has.

## Dedicated Inference and the GPU quota [#dedicated-inference-and-the-gpu-quota]

A Dedicated Inference endpoint runs on GPUs that no better-compute node owns, so there is nothing to place. It still
counts against the tenant's GPU quota on the pool: the kind declares what its spec books (GPUs per replica of
`gpuSize`, times the larger of `replicas` and `maxReplicas`), and the module books it in the same transaction that
records the resource.

```ts
await compute.api.cloudResources.create(acme, {
  tenantId: 'acme',
  poolId,
  kind: 'do.dedicated-inference',
  name: 'chat',
  spec: {
    region: 'nyc3',
    model: 'llama3.3-70b-instruct',
    gpuSize: 'gpu-h100x1-80gb',
    maxReplicas: 2,
  },
  idempotencyKey: 'chat',
});
// Books 2 GPUs against acme's quota on the pool, or fails with QUOTA_EXCEEDED and records nothing.
```

* **Over the quota**, `create` is `QUOTA_EXCEEDED` and nothing is recorded. The quota is the pool offer's `gpu`
  figure (see [quotas](/docs/guides/capacity/quotas-and-usage#quotas)).
* **Scaling** books the difference: growth is charged when `update` is called (`QUOTA_EXCEEDED` refuses the update),
  a shrink is returned once DigitalOcean has scaled the endpoint down (it is `ready` again).
* **Deleting** returns the GPUs once DigitalOcean has let the endpoint go, and only once.
* **Adopting** an endpoint made outside better-compute is refused (`UNSUPPORTED`, `quotaBooked`): it would run on
  GPUs nothing was charged for.

**Reaching it.** An endpoint takes a model access key of its account. On the tenant's own pool that is a
`do.inference-key` of its own. On a shared pool the tenant may not hold one (a key reaches the whole platform
account, see below), so it calls its endpoint through the operator's proxy: `inferenceProxy` with `endpoint` set to
the resource's `outputs.endpoint`, which the proxy accepts only as `https://<id>.inference.do-ai.run`. The host
checks the endpoint is one of the caller's tenant's resources and passes an operator key; the call's tokens are
recorded with the endpoint.

The endpoint reports GPU hours on the GPU model's meter (`gpu.h100.hours` and so on). Those bill only if the host
lists them in `cloudResources({ kindMeters })` and defines them in IAM billing; `cloud.resource.hours` bills either way.

## Model access keys and the inference proxy [#model-access-keys-and-the-inference-proxy]

`do.inference-key` is a model access key: the bearer that serverless inference (chat, completions, embeddings,
batch) and dedicated endpoints take. DigitalOcean answers the secret only on create and on a regenerate, and the
module seals it in the reconciler run that received it: the resource's `secretOutputs.key` is a `SecretRef`, and the
value never reaches a row, an event or a log. Raising `rotation` regenerates the key; the old one stops working. A
create whose answer was lost is found by its name and regenerated, so a secret no one sealed is never live.

A model access key reaches the whole DigitalOcean account: every model, and every Dedicated Inference endpoint on it.
So only a pool on the tenant's own account may have one. On a shared pool (the platform's account, offered to the
tenant) the create is refused with `INVALID_INPUT` and `details.reason` `accountWide`, before anything is recorded
or sent. Otherwise a tenant that read its key could call DigitalOcean directly, past the proxy, on the operator's bill,
and could reach other tenants' endpoints.

Tokens are counted in one of two ways:

* **Through the operator's proxy.** `inferenceProxy` from `@better-compute/provider-digitalocean` forwards
  `/v1/chat/completions`, `/v1/completions` and `/v1/embeddings` to DigitalOcean and hands each answer's token counts
  to a `record` callback, which the host writes as `ai.inference.tokens` usage (the key kind declares no kind meter:
  those bill an hourly rate, and tokens are not one). A streamed answer is counted from its
  final usage chunk, even when the client stops reading first: the proxy reads the rest itself. `stream` must be
  `true` or `false` (a `"true"` an upstream would coerce is refused, `reason` `stream`), and an answer is counted
  by what DigitalOcean sent (an event stream is scanned as events), not by what the request asked. Counting fails
  closed: a whole answer with no usage is withheld (502) and recorded with `uncounted: true`, and so is a stream
  that ended with no usage chunk. Wiring it to a route is the host's job: the server does not import providers.
* **Directly.** A tenant may call DigitalOcean with its own key; the usage then comes from DigitalOcean billing.

```ts title="inference-route.ts"
import { inferenceProxy } from '@better-compute/provider-digitalocean';

const proxy = inferenceProxy({
  // Your usage writer: `meta` is what the route passed with the call.
  record: (usage) => recordTokens(usage.meta?.tenantId, usage.model, usage.totalTokens),
});
// In your route: resolve the tenant's sealed key (secretOutputs.key), then
const response = await proxy({ path: '/v1/chat/completions', body, key, meta: { tenantId } });
```

## Knowledge bases and guardrails [#knowledge-bases-and-guardrails]

A **knowledge base** indexes web pages and Spaces objects with an embedding model into an OpenSearch database
DigitalOcean makes for it (or one the tenant names). Its sources change in place: they are added and removed one by
one, then one indexing job runs, and the resource is `updating` until it finishes. The knowledge base is marked at
DigitalOcean before its sources change, so a job whose start failed is started by the next reconcile. Spaces sources, a named database and
a VPC are account-wide objects, so only a knowledge base on the tenant's own account may name them; on a shared
account they are refused with `accountWide`. Deleting a knowledge base also deletes the OpenSearch database
DigitalOcean made for it, once the knowledge base is gone. A database the spec named is left alone. So is one that a
cloud resource or another deployment manages, or that another knowledge base uses.

A **guardrail** checks for jailbreaks, sensitive data or harmful content, and answers `defaultResponse` when it
trips. Guardrails carry no tags, so ownership lives in the name: `<name>-bc<deployment hash>`. One made by hand or by
another deployment is never changed or deleted (`notManaged`).

## Managed Agents (preview) [#managed-agents-preview]

Harness agents and action gateways are DigitalOcean previews. Both kinds are listed with `preview: true` and
`available: false` until the account sets `previews: true`; before that, `create` is `UNSUPPORTED` (`preview`) and
nothing reaches DigitalOcean. Turning previews off later does not strand what was made: those resources can still be
read and deleted, by the tenant or by a tenant teardown.

* **`do.agent-harness`** runs a model with `instructions`, optionally with knowledge bases, guardrails and an action
  gateway. `runtime: 'microvm'` isolates each session in a MicroVM (the default is `container`); the runtime and the
  region cannot change in place. Knowledge bases, guardrails and gateways are named by their DigitalOcean id, so,
  like a knowledge base's Spaces sources, only a harness on the tenant's own account may name them.
* **`do.action-gateway`** lists the https endpoints (`upstreams`) an agent may call as tools.

Both report an hours meter (`do.agent-harness.hours`, `do.action-gateway.hours`) and use name-based ownership, as
guardrails do.

## Signals into attention [#signals-into-attention]

DigitalOcean raises signals about harness agents and gateways: a run that failed, an agent asking a human to approve
something. `cloudResources.signals` reads them for an attention list; it is read-only, and the provider is only asked
to list. DigitalOcean lists signals oldest first, so every page is read and the newest 500 are kept. One answer per
account is reused for 30 seconds, whichever tenant asks.

```ts
const { items } = await compute.api.cloudResources.signals(acme, { tenantId: 'acme', poolId });
// [{ id, severity: 'warning', title: 'Human input requested', detail, at, kind: 'do.agent-harness',
//    providerId, resourceId }]
```

A tenant on a shared pool sees only the signals about its own resources. The pool's owner (the root tenant for the
platform's pool) sees every signal, including ones about agents no cloud resource records. Titles and details are
one line without control characters, at most 120 and 500 characters. Signals are a preview too: on an account
without `previews: true` the call is `UNSUPPORTED` and nothing is sent.


# Flavors (/docs/guides/vps/flavors)

> Named instance shapes per VPS pool, and how each becomes scalar claims, GPU atoms and quota on a host, or a provider offer.



Tenants should not have to know how many vCPUs a host has or what a provider calls its server types. A
flavor is a named shape, such as `g1.small` or `gpu1.l40s`, that the pool's owner defines
and tenants pick by name when they create an instance. The pool decides what the name
means: in a hosted pool it becomes claims on one host, in a provider pool it names the provider's offer.

Flavors belong to one pool. The same name in two pools can mean two different shapes.

## Define a flavor [#define-a-flavor]

The pool's owner creates flavors with [flavors.create](/docs/reference/api/flavors#create), decided as
`compute:flavors:manage` on the pool:

```ts
await compute.api.flavors.create(
  { token },
  { tenantId: 'root', poolId: 'pool_01jz…', name: 'gpu1.l40s', vcpu: 8, memoryGiB: 32, diskGiB: 200, gpus: 1 },
);
// → { id: 'flv_…', poolId: 'pool_01jz…', name: 'gpu1.l40s', vcpu: 8, memoryGiB: 32, diskGiB: 200,
//     gpus: 1, archived: false, version: 1 }
```

<TypeTable
  type="{
  tenantId: { type: 'string', description: &#x22;The pool owner's tenant. Other tenants get NOT_FOUND.&#x22;, required: true },
  poolId: { type: 'string', description: 'A pool of class vps-host or provider. Any other class is INVALID_STATE.', required: true },
  name: { type: 'string', description: '1 to 63 lower-case letters, digits, dots, underscores or hyphens, starting with a letter or digit. Unique in the pool.', required: true },
  vcpu: { type: 'number', description: 'Whole vCPUs, 1 to 1024.', required: true },
  memoryGiB: { type: 'number', description: 'GiB of memory, 1 to 16384.', required: true },
  diskGiB: { type: 'number', description: 'GiB of disk, 1 to 65536.', required: true },
  gpus: { type: 'number', description: 'Whole GPUs passed through to the VM, 0 to 16.', default: '0' },
  offerId: { type: 'string', description: &#x22;Provider pools: the provider offer this flavor creates, 1 to 128 characters. Required there; hosted pools ignore it.&#x22;, },
  placement: { type: &#x22;{ numaAffine?: boolean; pinned?: boolean; hugepages?: '2Mi' | '1Gi' }&#x22;, description: 'Hosted pools only (INVALID_INPUT on a provider pool): where the guest lives on its host. Sandbox templates on the flavor inherit it. See placement below.' },
}"
/>

A second flavor with the same name in the pool is `CONFLICT`. A provider pool flavor without `offerId` is
`INVALID_INPUT`.

The CLI's `flavors` commands (d1e83ca) take the same fields as flags:

```bash
better-compute flavors create --tenant root --pool pool_01jz… --name gpu1.l40s \
  --vcpu 8 --memory-gib 32 --disk-gib 200 --gpus 1
better-compute flavors list --tenant root --pool pool_01jz…
better-compute flavors update --tenant root --pool pool_01jz… --name gpu1.l40s --vcpu 8 --memory-gib 64 --disk-gib 200 --gpus 1
better-compute flavors delete --tenant root --pool pool_01jz… --name gpu1.l40s
```

`--offer` is the provider offer of a provider pool's flavor. `flavors update` sends every flag again, as the method
expects (see [update and delete](#update-and-delete)).

You can also declare flavors in a [FleetConfig](/docs/guides/fleet-config) document, next to the pools they belong
to, and let `fleet apply` call `flavors.*` for you:

```yaml title="fleet.yaml"
flavors:
  - { pool: vps-fra, name: g1.small, vcpu: 2, memoryGiB: 4, diskGiB: 40 }
```

## List flavors [#list-flavors]

[flavors.list](/docs/reference/api/flavors#list) (`compute:flavors:read`) returns `{ items }`, sorted by name, to
the pool's owner and to every tenant the pool is offered to. That is what an instance form loads when a pool is
chosen. Archived flavors are not listed.

## Flavors on hosted pools [#flavors-on-hosted-pools]

In a `vps-host` pool, capacity places the flavor on one host of the pool as an open-ended
allocation (it lasts until the instance is deleted) with:

* **scalar claims** `vcpu`, `memoryGiB` and `diskGiB`, the flavor's numbers;
* **devices** `{ class: 'gpu', count: gpus }`: whole GPU atoms, passed through with VFIO
  and booked like any other GPU booking.

A host fits when, for every scalar, `allocated + claim <= floor(allocatable × ratio)` on its `node_resources` row.
`ratio` is the overcommit: with `allocatable` 16 vCPU and `ratio` 2, a host takes 32 vCPUs of flavors. GPUs are never
overcommitted. A host with no row for a scalar the flavor asks for is not a candidate.

The `vps.hosts` work (every 30 seconds, `vps({ hostSyncEvery })`) derives those rows from each host's applied
inventory, for nodes of `vps-host` pools that are not `tenant-controlled`:

| Scalar      | `allocatable`                                                                               |
| ----------- | ------------------------------------------------------------------------------------------- |
| `vcpu`      | logical CPUs (`cpu.threads`) minus the reserve                                              |
| `memoryGiB` | installed memory less the host's huge page pools, in whole GiB, minus the reserve (9d55b36) |
| `diskGiB`   | non-removable disks in whole GiB minus the reserve                                          |

The reserve is what a host keeps for itself, `vps({ hostReserve })`, by default 2 vCPU, 8 GiB and 100 GiB. `ratio` is
the pool's `policy.overcommit[resource]` (default 1, clamped to 0.1 to 100). Rows are written only when they change,
and a shrinking host never drops `allocatable` below what is already allocated. Until a host has reported inventory
it has no rows and is not a candidate; `doctor` warns `vps.hosts.capacity` for adopted hosts in that state.

The same shape is what the instance counts against the tenant's quota in the pool: `vcpu`, `memoryGiB`, `diskGiB`,
and `gpu` when `gpus` is above 0. The placement charges it, and deleting the instance returns it. See
[placement](/docs/guides/capacity/placement) and [quotas and usage](/docs/guides/capacity/quotas-and-usage).

Hosted instances are metered from placement until their allocation is released: `vps.vcpu.hours` at `vcpu` per hour,
`vps.memory.gib.hours` at `memoryGiB` per hour, and for GPU flavors `gpu.<model>.hours` at `gpus` per hour, where the
model is the pool's `gpuModel` in lower case. The `vps.usage` job writes the windows every 5 minutes; see
[billing and usage](/docs/guides/iam/billing-and-usage).

### Placement [#placement]

A flavor's `placement` (9d55b36, migration `vps/0018_placement`) says where its guests live on their host:

| Field                       | Effect                                                                                                                                                                                                                                                                                                                                                                             |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hugepages: '2Mi' \| '1Gi'` | Guest memory is backed by huge pages of that size, never ballooned. The instance books the host's `hugepages1Gi` (one per GiB) or `hugepages2Mi` (512 per GiB) row instead of `memoryGiB`; the capacity module keeps those rows from each host's configured pools (see [host resources](/docs/guides/capacity/reservations#host-resources)). Quota still charges it as `memoryGiB` |
| `numaAffine: true`          | vCPUs and memory on one NUMA node: the node of the instance's GPUs, or without GPUs the least-loaded node with room for the memory                                                                                                                                                                                                                                                 |
| `pinned: true`              | Each vCPU gets its own host CPU that no other pinned guest on the host has                                                                                                                                                                                                                                                                                                         |

After capacity places the instance on a host, vps chooses the host placement from the host's inventory (its NUMA
nodes, its GPUs' NUMA nodes) and the other guests' placements, stores it in `instances.host_placement`, and sends it
to the host as `InstanceSpec.placement` (`cpus`, `mems`, `hugepageSizeBytes`), which bcd maps onto `libvirt-kvm` or
`cloud-hypervisor` (see [instances](/docs/daemon/delivery#instances)). Placements on one host take turns on its `vcpu`
row, so the second sees the first's pinned CPUs. A host that cannot honour `numaAffine` or `pinned` still runs the
guest, floating, and the instance carries a `PlacementDegraded` condition naming why: `NoNumaTopology`,
`NoNumaNodeFits`, `GpuNumaUnknown`, `NumaMemoryTooSmall` or `NotEnoughFreeCpus`. Huge pages are never degraded:
they were booked, and bcd's own admission refuses a start the free pages cannot back.

* **The host's own CPUs.** Since b6b804e a guest's CPUs never include the host's reserved CPUs, the lowest
  non-isolated, `policy.hostReserve.vcpu` (default 2), whether pinned or floating on a NUMA node.
* **Isolated CPUs first.** A pinned guest takes the CPUs the kernel isolates first (no host tasks or timer ticks
  there), then the rest by id. Isolate them with the pool's [`cpu` unit](/docs/daemon/host-config#cpu).
* **Only where bcd applies it.** An instance whose flavor has a `placement` is placed only on hosts labelled
  `bc.dev/cap.instance.placement` (82f4f51), which a bcd that applies `InstanceSpec.placement` advertises: an older
  bcd would back a huge-page guest with memory placement never booked.
* **A balloon on overcommitted pools** (073324a). A pool whose `policy.overcommit.memoryGiB` is above 1 gives every
  guest a virtio balloon, so the host can reclaim the guests' free memory, except huge-page-backed and GPU (VFIO)
  guests, whose memory is pinned. An older bcd drops the field harmlessly, so no capability label is needed for it.
  Since 12e9ced bcd inflates those balloons only while the host is short of memory (under 10% available, or a
  memory stall of 10% or more), idle guests first, never below a guest's floor (`[instances] balloon_floor_percent`,
  default 50, or its use plus 10%), and deflates them once more than 20% is free again; host admission still charges
  each guest its full memory. See [memory ballooning](/docs/daemon/delivery#memory-ballooning).

## Flavors on provider pools [#flavors-on-provider-pools]

In a `provider` pool the flavor's `offerId` is what the reconciler passes to `ProviderImpl.create`. Its `vcpu`,
`memoryGiB`, `diskGiB` and `gpus` are what the tenant is charged against quota at admission, and what policies
see; nothing checks them against the offer. Copy them from the offer:

```ts
const { items } = await compute.api.providerAccounts.offers(
  { token },
  { tenantId: 'root', accountId: 'pacct_…', query: { kind: 'vps', minVcpu: 2 } },
);
// [{ id: 'cx22', kind: 'vps', region: 'fsn1', vcpu: 2, memoryGiB: 4, diskGiB: 40, pricePerHour: 0.0071, currency: 'EUR', available: true }, …]

await compute.api.flavors.create(
  { token },
  { tenantId: 'root', poolId: providerPool, name: 'cx22', vcpu: 2, memoryGiB: 4, diskGiB: 40, offerId: 'cx22' },
);
```

```bash
better-compute provider-accounts offers pacct_01jz… --tenant root --kind vps
better-compute flavors create --tenant root --pool pool_01jz… --name cx22 --vcpu 2 --memory-gib 4 --disk-gib 40 --offer cx22
```

Provider instances have no allocation and are not metered: the provider bills you. See
[providers](/docs/guides/vps/providers).

## The shape an instance keeps [#the-shape-an-instance-keeps]

An instance stores a copy of its flavor's shape when it is admitted. Changing or archiving the flavor later never
changes a live instance: its claims, quota and policy attributes stay what they were. To move an instance to another
shape, resize it.

[instances.resize](/docs/reference/api/instances#resize) works on stopped hosted instances whose GPU count stays the
same. It computes the difference per resource between the old shape and the new one, then:

1. charges the growth to quota (`QUOTA_EXCEEDED` if it does not fit);
2. calls capacity's `resizeAllocation(tx, allocationId, { vcpu, memoryGiB, diskGiB })`, which changes the
   allocation's scalar claims in place on its node, all or nothing: growth takes the same guarded
   `allocated + Δ <= floor(allocatable × ratio)` update, shrinkage returns room;
3. on a conflict, refunds the growth and fails with `CAPACITY_UNAVAILABLE`, `details.resource` naming what ran out;
4. otherwise refunds the shrinkage and stores the new shape.

The allocation never moves to another host, and GPU atoms are never added or removed: a different GPU count is
`UNSUPPORTED`, and so is a flavor with a different `placement` and resizing a provider instance. Create a new
instance instead. A resize with the same `placement` chooses the host placement again for the new shape.

## Update and delete [#update-and-delete]

[flavors.update](/docs/reference/api/flavors#update) takes the same input as `create` and replaces the shape. Every
field is required again except `gpus`, which falls back to 0 when you leave it out, and `offerId`, which keeps its
value when you leave it out. Updating an archived flavor also restores it. A missing flavor is `NOT_FOUND`.

[flavors.delete](/docs/reference/api/flavors#delete) returns `{ deleted, archived }`:

* a flavor no instance ever used is deleted: `{ deleted: true, archived: false }`;
* a flavor any instance row references, including deleted instances, is archived instead:
  `{ deleted: false, archived: true }`.

An archived flavor is hidden from `flavors.list`, and new instances and resizes that name it are `NOT_FOUND`.
Instances already using it keep running.

## Flavors in profiles [#flavors-in-profiles]

A VPS profile written with `defineProfile({ target: 'vps', runtime, flavors, network })` validates a map of flavor
names to shapes as well (see [steps and profiles](/docs/extending/steps-and-profiles)). `vps()` does not read
profiles in this release: instances always use the pool's flavor rows described here.


# VPS (/docs/guides/vps)

> Virtual machines on your own hosts or on a provider's cloud, with desired state, flavors, provider accounts and pool IPAM from the vps() module.



Some of your capacity is best sold as whole machines, and some as virtual machines: a tenant wants a 2 vCPU box with
an SSH key for a build agent, or an 8 vCPU VM with one L40S passed through, without renting a whole server. Other
times you want to burst onto a cloud you do not run, and still keep one API, one quota system and one audit trail.

The `vps()` module does both. It adds instances: VMs described by a
flavor, an image and SSH keys, created in a
pool. Where the VM runs depends on the pool:

* A **hosted pool** runs VMs on your own servers. Capacity places each instance on a host, and the host's
  bcd converges it.
* A **provider pool** creates each instance on an external cloud, through a provider
  such as Hetzner Cloud or your own Proxmox VE cluster.

Either way, the API only records what you want. A reconciler does the work, and nothing
in a request ever calls a hypervisor or a cloud API.

## Register the module [#register-the-module]

`vps()` is an injected module: the server never imports it, so you add it to `modules`. It requires the built-in
`capacity` module, which every server has.

```ts title="compute.ts"
import { betterCompute } from 'better-compute/server';
import { vps } from 'better-compute/vps';
import { hetznerCloud } from '@better-compute/provider-hetzner';
import { images } from './images';

export const compute = betterCompute({
  // database, secret, iam, …
  modules: [
    vps({
      runtimes: ['libvirt-kvm'], // hosted pools: the runtime driver bcd uses
      providers: [hetznerCloud], // provider pools: the providers accounts may use
      images, // hosted pools: resolves imageId for a tenant
    }),
  ],
});
```

The provider packages are on npm but sit outside the `better-compute` umbrella, so you install and import them by
their own names (`npm install @better-compute/provider-hetzner`).

<TypeTable
  type="{
  runtimes: {
    type: 'string[]',
    description: &#x22;Runtime driver ids that hosted pools run, such as 'libvirt-kvm' or 'cloud-hypervisor'. Every new hosted instance records the first one. Ids are lower-case letters, digits and hyphens; an empty list is INVALID_CONFIG.&#x22;,
    default: &#x22;['libvirt-kvm']&#x22;,
  },
  providers: {
    type: 'ProviderDefinition[]',
    description: 'The defineProvider() definitions that provider accounts may name. Anything else, or the same id twice, is INVALID_CONFIG.',
    default: '[]',
  },
  images: {
    type: 'ImageResolver',
    description: 'resolve(tx, tenantId, imageId) returns the image a hosted VM boots, or undefined when the tenant cannot see it. Required for hosted pools: without it, instances.create in a hosted pool is INVALID_CONFIG.',
  },
  secrets: {
    type: 'SecretResolver',
    description: &#x22;Resolves the SecretRefs in provider account configs. Defaults to the server's secret broker.&#x22;,
  },
  hostNetwork: {
    type: 'string',
    description: &#x22;The host bridge a hosted instance's NIC attaches to (InstanceSpec.nics). An interface name of 1 to 15 letters, digits, _ . -; anything else is INVALID_CONFIG.&#x22;,
    default: &#x22;'br0'&#x22;,
  },
}"
/>

vps ships no image resolver: Better Compute checks image visibility itself, and you decide where images come from.
This one serves hosted VMs the images registered through the `images` group of `provisionModule`, when the tenant
owns them or they are public:

```ts title="images.ts"
import type { ImageResolver } from '@better-compute/vps';

export const images: ImageResolver = {
  async resolve(tx, tenantId, imageId) {
    const { rows } = await tx.query<{
      source: string;
      sha256: string;
      format: 'raw' | 'qcow2' | 'iso';
      verified: boolean;
    }>(
      `SELECT source, sha256, format, verified FROM images
        WHERE id = $1 AND deleted_at IS NULL AND (tenant_id = $2 OR visibility = 'public')`,
      [imageId, tenantId],
    );
    const image = rows[0];
    // undefined: the tenant cannot see it, and instances.create answers NOT_FOUND.
    return image && { url: image.source, sha256: image.sha256, format: image.format, verified: image.verified };
  },
};
```

`verified` becomes the instance's `imageVerified` attribute, which policies can require.

Registering the module adds:

* the `instances`, `flavors` and `providerAccounts` API groups
  ([instances](/docs/reference/api/instances), [flavors](/docs/reference/api/flavors),
  [providerAccounts](/docs/reference/api/provider-accounts)), and the v0.5 `sandboxes`, `sandboxTemplates` and `sandboxSnapshots` groups
  ([sandboxes](/docs/reference/api/sandboxes), [sandboxTemplates](/docs/reference/api/sandbox-templates),
  [sandboxSnapshots](/docs/reference/api/sandbox-snapshots)) for
  [sandboxes](/docs/guides/vps/sandboxes) on hosted pools;
* the migrations `vps/0001_instances`, with the tables `flavors`, `provider_accounts`, `instances`, `ip_pools` and
  `ip_leases`, `vps/0002_sealed_user_data` (`instances.user_data_sealed`), `vps/0003_provider_bootstraps`
  (`provider_bootstraps`, for [bootstrapping provider machines](/docs/extending/providers#bootstrapping-machines))
  and, for sandboxes, `vps/0004_sandboxes` (the tables `sandbox_templates` and `sandboxes`, the columns
  `instances.sandbox_id` and `instances.warm_template_id`, and `paused` as a desired state),
  `vps/0005_sandbox_snapshots` (`sandbox_snapshots`, `instances.restore_from`, `sandboxes.from_snapshot`),
  `vps/0006_warm_claims` (`instances.claimed_at`), `vps/0007_template_env` (`sandbox_templates.env`) and
  `vps/0008_template_network` (`sandbox_templates.network` and `instance_version`, `instances.network_policy` and
  `warm_template_version`), `vps/0009_snapshot_fixes` (`sandbox_snapshots.principal_instance_id` and `purged`),
  `vps/0010_sandbox_metric_indexes` (partial indexes for the sandbox metrics), `vps/0011_host_certificates`
  (`instances.host_certified`, `sandbox_snapshots.host_certified`) and `vps/0012_host_cert_validity`
  (`host_cert_valid_before` on both);
* the `vps.instances` reconciler, woken through the work queue by every change, the
  `vps.sandboxes` reconciler (every 15 seconds: sandbox expiry, sandbox events, warm pools and settling snapshots),
  and the `vps.usage` job, which meters hosted instances every 5 minutes;
* the `instance.snapshot` command kind (lane `bulk`, timeout 1 h) for sandbox snapshots;
* a `tenantDeleted` hook that deletes every instance of a deleted IAM tenant, sandboxes' and warm ones included;
* an `instances` section in the NodeSpec of each hosted node;
* the `runtime` driver kind, merged into `DriverKinds` of `@better-compute/core` when you import the package;
* `doctor` findings for hosted pools: `vps.pools.runtime` (a runtime the deployment does not list),
  `vps.pools.single-tenant` (a container pool without `policy.singleTenant`), `vps.pools.ipam` (no IP range) and
  `vps.hosts.capacity` (adopted hosts without capacity rows); see [doctor](/docs/operations/observability#doctor).

Hosted pools also need the gateway to feed each host's runtime report to vps. Pass
`instanceStatusHandler` as a status handler. It runs next to the built-in XID health handler, each under its own
savepoint:

```ts title="better-compute.config.ts"
import { defineHost } from '@better-compute/cli';
import { instanceStatusHandler } from '@better-compute/vps';

export const host = defineHost(async ({ compute }) => ({
  gateway: {
    // region, listen, tls, …
    options: { reports: { status: instanceStatusHandler({ events: compute.context.events }) } },
  },
}));
```

## Hosted pools and provider pools [#hosted-pools-and-provider-pools]

A VPS pool is a pool of class `vps-host` or `provider`, created with
[pools.create](/docs/reference/api/pools#create) like any other. Instances in pools of other classes are refused
with `INVALID_INPUT`.

|                   | Hosted pool                                                                          | Provider pool                                                          |
| ----------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| Pool              | `class: 'vps-host'`, `agent: 'bcd'`                                                  | `class: 'provider'`, `agent: 'none'`, `providerAccountId`              |
| Where the VM runs | A node of the pool, running bcd                                                      | The provider's cloud; there is no `nodes` row                          |
| Placement         | Capacity claims vCPU, memory, disk and whole GPUs on one host, in the create request | None: the provider decides; the reconciler calls `ProviderImpl.create` |
| Quota             | Charged by the placement                                                             | Charged at admission from the flavor's shape                           |
| Who converges     | The host's runtime, from the `instances` section of its NodeSpec                     | The `vps.instances` reconciler, polling the provider                   |
| State comes from  | The host's `StatusReport.instances`                                                  | `ProviderImpl.get`                                                     |
| Addresses         | A lease from the pool's IPv4 range, plus what the runtime reports                    | What the provider reports                                              |
| Resize            | In place, while stopped, same GPU count                                              | `UNSUPPORTED`                                                          |
| Metering          | `vps.vcpu.hours`, `vps.memory.gib.hours`, `gpu.<model>.hours`                        | None: the provider bills you                                           |

No platform agent ever runs inside a tenant VM, and no join token ever goes into
tenant-readable user data. A provider machine that should become a host of your own (a Hetzner Robot server that
runs bcd) is a different thing: see [bare metal](/docs/guides/bare-metal).

## The instance model [#the-instance-model]

An instance has two states. `desiredState` is what you asked for: `running`, `stopped` or `deleted`. `state` is where
the instance actually is. `instances.create` sets `desiredState` (default `running`); `start`, `stop` and `delete`
change only `desiredState` and wake the reconciler. The reconciler moves `state` toward it, one guarded transition at
a time, from what the host or the provider reports.

<Mermaid
  chart="stateDiagram-v2
  [*] --> pending
  pending --> scheduling
  scheduling --> provisioning: placed or created
  provisioning --> starting
  provisioning --> scheduling: recreate, host cannot start it
  starting --> running
  running --> stopped: stop
  stopped --> starting: start
  stopped --> resizing: resize
  resizing --> stopped
  running --> lost: host or machine gone
  lost --> running: recover
  lost --> scheduling: recreate
  running --> error: runtime error
  error --> starting: start
  running --> deleting: delete
  deleting --> deleted
  deleted --> [*]"
/>

Every state except `deleted` and `deleting` can move to `deleting`, and most can move to `error` and `lost`. The
full table is `instanceStateTable` in `@better-compute/core`; since 564f543 it also allows `recreate` from
`provisioning` and `stopped`, for an instance whose host cannot start it (see
[hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance)). `rebooting` is in the table, but no API method enters
it in this release. `paused` (v0.5: `running` to `paused` by `pause`, back by `resume`) is used only by the warm
pools of [sandboxes](/docs/guides/vps/sandboxes#warm-pools); see [paused instances](/docs/guides/vps/instances#paused-instances).

The API returns the `Instance` view:

<TypeTable
  type="{
  id: { type: 'string', description: 'inst_ followed by 52 characters, derived from the tenant, the caller and the idempotency key.', required: true },
  state: { type: 'InstanceState', description: 'pending, scheduling, provisioning, starting, running, stopped, rebooting, resizing, deleting, deleted, error, lost or (warm sandbox instances) paused.', required: true },
  desiredState: { type: &#x22;'running' | 'stopped' | 'paused' | 'deleted'&#x22;, description: &#x22;What the reconciler converges toward. Only sandbox warm pools ask for 'paused'.&#x22;, required: true },
  flavor: { type: 'string', description: 'The flavor name the instance was admitted with.', required: true },
  imageId: { type: 'string', description: 'The image as the caller named it.', required: true },
  sshKeys: { type: 'string[]', description: 'The OpenSSH public keys from the request.', required: true },
  addresses: { type: 'string[]', description: 'IP addresses: the IPAM lease and what the runtime or provider reports.', required: true },
  conditions: { type: 'Condition[]', description: 'Why the instance is not where it should be: Unschedulable, RuntimeError or ProviderError.', required: true },
  nodeId: { type: 'string', description: 'Hosted pools: the node it is placed on.' },
  allocationId: { type: 'string', description: 'Hosted pools: the capacity allocation that holds its resources.' },
  externalId: { type: 'string', description: &#x22;Provider pools: the provider's machine id.&#x22; },
  poolId: { type: 'string', description: 'The pool.', required: true },
  projectId: { type: 'string', description: 'The project, when the request named one.' },
  ownerIdentityId: { type: 'string', description: 'Who created it; list defaults to these.', required: true },
  idempotencyKey: { type: 'string', description: 'The key of the create request.', required: true },
  version: { type: 'number', description: 'Increases with every change.', required: true },
}"
/>

[Instances](/docs/guides/vps/instances) covers each operation and how the reconciler turns reports into
transitions.

## What ships in v0.3 [#what-ships-in-v03]

The control plane is complete and tested: the three API groups, admission, the reconciler for both pool kinds,
pool IPAM, the report seam and metering run against PGlite in `tests/vps-instances.test.ts`, and
`tests/vps-churn.test.ts` runs the v0.3 acceptance shape (200 instances churned on 3 emulated hosts with no leaked
allocation, IP lease, scalar claim or quota). What runs the VMs on a host is written but not yet proven: bcd's
`libvirt-kvm` and `cloud-hypervisor` runtimes have been tested against fake tools only.

| Part                                                            | Status in this release                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `instances`, `flavors`, `providerAccounts` groups               | Served (v0.3).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `vps.instances` reconciler, IPAM, `vps.usage` metering          | Shipped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `hetzner-cloud` provider                                        | Shipped in `@better-compute/provider-hetzner`, tested against an in-memory emulator of the Hetzner Cloud API.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `proxmox` provider                                              | Shipped in `@better-compute/provider-proxmox` (PLAN.md scheduled it for v0.5), tested against an in-memory PVE emulator.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Host runtimes `libvirt-kvm` and `cloud-hypervisor`              | bcd (v0.3), built but not released. A host build with `virsh` runs `libvirt-kvm` (362317c) under the `instances` converger: qcow2 overlays on sha256-verified https base images, NoCloud seeds, VFIO GPUs, pause and resume, snapshots and restores. A host build with the `cloud-hypervisor` binary runs `cloud-hypervisor` (2598f0a): one VMM per instance as a confined systemd transient unit, over its REST API, with the same overlays and seeds, VFIO GPUs, pause and resume, memory snapshots and restores. Security review 12's two high-severity findings are fixed (933c51b): taps are bridged before the VMM starts and an `inet bcd_taps` backstop drops a tap off its bridge, and both runtimes refuse a tenant image or restored snapshot that names a backing file or an external data file. Both have been tested only against fake tools, never on a real KVM host. A host with neither and without a runtime plugin fails a non-empty `instances` section; fakehw builds run a file-backed runtime. See [the runtimes](/docs/daemon/delivery#the-runtimes). |
| Runtimes as out-of-process plugins (Kata, and others)           | v0.5: bcd serves a bcx plugin of kind `runtime` as an instances runtime (`list`, `apply`, `remove`; no snapshots), Go unit tests only. No Kata plugin is written. See [walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Single-tenant container runtimes                                | Planned for v0.3; the `RuntimeTraits` contract describes them, nothing enforces single tenancy yet.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Interactive access (exec, SSH, TCP through a tunnel)            | `tunnels.open` with an `instanceId`, decided as `compute:instances:exec`: hosted instances only, and only when the deployment configures tunnels. The API reference lists it as planned. See [tunnels](/docs/guides/access/tunnels).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Sandboxes (`sandboxes`, `sandboxTemplates`, `sandboxSnapshots`) | v0.5: served, all 15 methods. Templates with `env` and `network`, TTL expiry, warm pools of paused instances and their `vps.sandbox.warm` meter, snapshots, restores and snapshot management, `sandboxes.exec` (from the tunnels module), the instance guard, tenant teardown (`tests/vps-sandboxes.test.ts`, `tests/e2e/sandboxes.test.ts`). bcd enforces a template's `network` with host nftables rules. Like every hosted instance, a sandbox needs a host runtime. See [sandboxes](/docs/guides/vps/sandboxes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Per-instance metadata service and workload identity             | v0.5, optional. The control plane half (the workload broker, the session frames, the issuer and its published JWKS) is built and tested against the FakeDaemon's metadata service; bcd serves the same contract on `[imds] listen`, for the instances of whichever runtime it runs (end to end only with a fakehw build's file-backed runtime so far). See [workload identity](/docs/guides/vps/instances#workload-identity).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

The TypeScript `FakeDaemon` in `@better-compute/testing` speaks the real protocol for the simulator and
`better-compute dev --sim N`. As a VPS host it runs a `FakeHypervisor` (domains, overlays, NoCloud seeds, pause and
resume), which is how `tests/e2e/vps.test.ts` runs hosted instances through the real server and gateway. The unit
suites, `tests/vps-sandboxes.test.ts` among them, emulate hosts by calling `reportInstances` directly.

  - [Instances](/docs/guides/vps/instances): Create, start, stop, resize and delete; the reconciler, host reports, orphans and lost hosts.

  - [Sandboxes](/docs/guides/vps/sandboxes): Templates, a time-to-live, warm pools of paused instances, and the instance each sandbox drives.

  - [Flavors](/docs/guides/vps/flavors): Named shapes per pool, and how they become claims, quota and provider offers.

  - [Providers](/docs/guides/vps/providers): Provider accounts, defineProvider, Hetzner Cloud, Proxmox VE and the conformance cases.

  - [Networking](/docs/guides/vps/networking): Pool IPAM: one IPv4 range per hosted pool, one lease per instance.


# Instances (/docs/guides/vps/instances)

> Create, start, stop, resize and delete VPS instances, and how the vps.instances reconciler converges them from host reports or provider polls.



An instance is a VM in a VPS pool. You ask for one with a
flavor, an image and SSH keys; after that you only change what you want it to be
(`running`, `stopped` or `deleted`). The `vps.instances` reconciler works out how to
get there, from what the host's runtime reports in a hosted pool or from what the provider answers in a provider
pool.

This page assumes you registered `vps()` and have a VPS pool with at least one flavor; see [VPS](/docs/guides/vps)
and [flavors](/docs/guides/vps/flavors).

Some instances belong to [sandboxes](/docs/guides/vps/sandboxes): the instance a sandbox drives, and the warm
instances a sandbox template keeps ready. Those can be read but not moved with the methods on this page; see
[sandbox and warm instances](#sandbox-and-warm-instances).

## Create an instance [#create-an-instance]

```ts title="server.ts"
const instance = await compute.api.instances.create(
  { token },
  {
    tenantId: 'acme',
    poolId: 'pool_01jz…',
    flavor: 'g1.small',
    imageId: 'ubuntu-24.04',
    sshKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA… alice@laptop'],
    idempotencyKey: 'build-agent-7',
  },
);
// Hosted pool: { id: 'inst_…', state: 'provisioning', desiredState: 'running',
//                nodeId: 'node_…', addresses: ['10.0.0.2'], … }
// Provider pool: { id: 'inst_…', state: 'scheduling', … }
```

The instance is the caller's: `ownerIdentityId` is the caller, and it shows in their `scope: 'mine'` list. The
platform issues an instance to a person of a customer tenant by naming them:

```ts title="issue.ts"
await compute.api.instances.create(rootOperator, {
  tenantId: 'acme', // the customer, not the root tenant
  poolId, // a provider pool of the platform, offered to acme
  flavor: 'n2.standard.4',
  imageId: 'debian-12',
  sshKeys: [],
  ownerIdentityId: 'idn_…', // a person of acme
  idempotencyKey: 'issue-acme-1',
});
```

Only a root caller creating in another tenant may name someone other than itself (`ACCESS_DENIED` otherwise, before
any decision), and when the IAM port can say (`IamPort.isMember`, the embedded bridge) the owner must be an active
identity of that tenant (`INVALID_INPUT`, and no draft is left). `instances.attach` takes `ownerIdentityId` on the
same terms. See [platform-issued compute](/docs/guides/vps/providers#platform-issued-compute).

From a browser or another service, the typed client strips the credential, and every method is also a plain HTTP
route:

  **Client:**

    ```ts
    import { createComputeClient } from 'better-compute/client';

    const client = createComputeClient<typeof compute>({ baseURL, token: () => session.getToken() });
    await client.instances.create({ tenantId: 'acme', poolId, flavor: 'g1.small', imageId: 'ubuntu-24.04', idempotencyKey: 'build-agent-7' });
    ```
  
  **HTTP:**

    ```http
    POST /api/compute/instances/create
    Authorization: Bearer <token>
    Content-Type: application/json

    { "tenantId": "acme", "poolId": "pool_01jz…", "flavor": "g1.small", "imageId": "ubuntu-24.04", "idempotencyKey": "build-agent-7" }
    ```
  
  **CLI:**

    ```bash
    better-compute instances create --tenant acme --pool pool_01jz… --flavor g1.small \
      --image ubuntu-24.04 --ssh-key ~/.ssh/id_ed25519.pub --key build-agent-7
    ```
  
<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant the instance belongs to.', required: true },
  poolId: { type: 'string', description: 'A vps-host or provider pool the tenant owns or is offered, and not archived.', required: true },
  flavor: { type: 'string', description: 'A flavor name of that pool, such as g1.small.', required: true },
  imageId: { type: 'string', description: 'Up to 200 characters of letters, digits and . _ @ : / -, starting with a letter or digit. Hosted pools resolve it with vps({ images }); provider pools pass it to the provider as is.', required: true },
  sshKeys: { type: 'string[]', description: 'Up to 32 OpenSSH public key lines: ssh-ed25519, ssh-rsa, ecdsa-sha2-nistp256/384/521 or sk-ssh-ed25519@openssh.com.', default: '[]' },
  userData: { type: 'string', description: 'Your own cloud-init, at most 32 KiB, or a reference to it. When given it is used as is, and sshKeys are not added to it. A reference must be secret://<tenantId>/<id>, a secret this tenant wrote with secrets.put (a secret_refs row), since the guest reads it; a platform-minted secret, a fleet secret (secrets.put with scope fleet), the sealed user data of another instance, or any other secret://, vault:// or env:// value is INVALID_INPUT. A reference also needs compute:secrets:manage (see the decision below).' },
  desiredState: { type: &#x22;'running' | 'stopped'&#x22;, description: 'Where the instance should end up.', default: &#x22;'running'&#x22; },
  projectId: { type: 'string', description: 'A prj_ id, for usage tags and policies.' },
  idempotencyKey: { type: 'string', description: '1 to 128 of letters, digits and . _ : -. Retrying with the same key returns the same instance.', required: true },
  wait: { type: '{ timeoutMs: number }', description: '0 to 30000. Holds the answer until the instance has started or failed; see waiting for the instance below. Not part of the request hash.' },
}"
/>

Unknown fields are `INVALID_INPUT`. The API reference has the route and action:
[instances.create](/docs/reference/api/instances#create).

### How a create is admitted [#how-a-create-is-admitted]

`instances.create` follows the same request-as-resource admission as
[reservations](/docs/guides/capacity/reservations): the instance exists as a draft before
anyone decides on it, so policies can read what is being asked for.

      ### Transaction 1: the draft [#transaction-1-the-draft]

    The instance id is derived from the tenant, the caller's identity and the idempotency key. If a finished instance
    already has that id, the call returns it (a replay). If the stored request differs from this one, the call fails
    with `CONFLICT` and `details.instanceId`. Otherwise vps checks that the pool is visible and is a VPS pool, that the
    flavor exists and is not archived, and, in a hosted pool, that the image resolves for the tenant. It then writes the
    draft and its authorization row.
  
      ### The decision [#the-decision]

    Better IAM decides `compute:instances:create` on `compute-instance/<draft id>`. The row's parent is the pool, its
    owner is the caller, and it carries the attributes policies can test: `vcpu`, `memoryGiB`, `gpus`, `imageId`,
    `imageVerified` and `projectId`. A refusal deletes the draft, so a later retry is decided afresh. An agent under
    [delegated confirmation](/docs/guides/iam/delegated-confirmation) gets `CONFIRMATION_REQUIRED` and retries with the
    same key once a person approves.

    When `userData` is a `secret://` reference, the admission check first decides `compute:secrets:manage` on
    `compute-tenant/<tenantId>` with the caller's own credential, and in the root tenant also on `compute-fleet/global`,
    whose secrets share the root tenant's store: user data is handed to the guest, which reads it, while secrets are
    otherwise write-only, so `compute:instances:create` alone must not be enough to read one. A call without a caller
    credential is `ACCESS_DENIED` with `details.reason: 'credential-required'`. The decision uses the deployment's IAM
    type prefix (`ModuleContext.typePrefix`).

    Still outside any transaction, the pool's budget check runs, the same `budgetAdmissionCheck` as for
    [reservations](/docs/guides/capacity/reservations): when the pool's policy sets `admission.budgetCheck: 'meter'`, or
    the tenant's offer has `onBudgetExceeded: 'stop-new'` or `'drain'`, it calls Better IAM's `billing.check` for the
    caller, and a spent budget answers [`SPEND_LIMIT_REACHED`](/docs/reference/errors#spend_limit_reached). The draft
    stays, so the same key is decided again once the budget's window resets. See
    [budgets](/docs/guides/iam/billing-and-usage#budgets).
  
      ### Transaction 2: admission and scheduling [#transaction-2-admission-and-scheduling]

    The draft becomes an instance in `scheduling`. In a **hosted pool**, capacity places it at once and vps leases an IP
    address, so the answer is already `provisioning` with a `nodeId`. When no host fits, the answer is `scheduling` with
    an `Unschedulable` condition (reason `NoFit`), and the reconciler tries again every 30 seconds. In a **provider
    pool**, vps charges the flavor's shape to the tenant's quota and returns `scheduling`; the reconciler creates the
    machine.
  
A policy can bound what tenants create, for example only small VMs from signed images:

```json title="policy statement"
{
  "sid": "SmallVerifiedVms",
  "effect": "allow",
  "actions": ["compute:instances:create"],
  "resources": ["compute-instance/*"],
  "conditions": {
    "Bool": { "resource.imageVerified": true },
    "NumericLessThanEquals": { "resource.vcpu": 8, "resource.gpus": 0 }
  }
}
```

`imageVerified` comes from the image resolver. In a provider pool without `vps({ images })` it is `false`.

### Errors [#errors]

| Code                    | When                                                                                                                                                                                                |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INVALID_INPUT`         | A field fails validation, the pool is not of class `vps-host` or `provider`, or `userData` names a secret the tenant did not put with `secrets.put`, or a fleet secret (`scope: 'fleet'`, aac73a1). |
| `NOT_FOUND`             | The pool is not owned by or offered to the tenant, or is archived; the flavor is missing or archived; a hosted pool's image does not resolve.                                                       |
| `CONFLICT`              | The idempotency key was used for a different request.                                                                                                                                               |
| `ACCESS_DENIED`         | Better IAM refused. The draft is gone. Also a `secret://` user data without `compute:secrets:manage`, or without a caller credential.                                                               |
| `CONFIRMATION_REQUIRED` | A delegated session needs a person to approve.                                                                                                                                                      |
| `SPEND_LIMIT_REACHED`   | The pool checks budgets and a Better IAM budget covering the caller is spent. The draft stays.                                                                                                      |
| `QUOTA_EXCEEDED`        | The tenant's quota in the pool is too small. `details` names `resource`, `requested`, `max` and `used`.                                                                                             |
| `CAPACITY_UNAVAILABLE`  | Hosted pool: the pool's IP range has no free address.                                                                                                                                               |
| `INVALID_CONFIG`        | Hosted pool, but `vps()` was registered without `images`.                                                                                                                                           |
| `INVALID_STATE`         | Provider pool without a provider account.                                                                                                                                                           |

Every resource an instance uses needs a quota row in the pool: `vcpu`, `memoryGiB`, `diskGiB`, and `gpu` for GPU
flavors. [pools.offer](/docs/reference/api/pools#offer) sets them for a tenant. A pool's owner gets only a `gpu`
quota by default, so give the owner `ownerQuota` for the scalar resources before it creates instances in its own
pool. See [quotas and usage](/docs/guides/capacity/quotas-and-usage).

### Waiting for the instance [#waiting-for-the-instance]

`create` is async-with-wait (PLAN.md section 9.1). Without `wait` it answers as soon as transaction 2 commits, with
the instance in `scheduling` or `provisioning`. With `wait: { timeoutMs }` the server holds the answer while the
instance is `pending`, `scheduling`, `provisioning` or `starting`, re-reading its row in read-only transactions
(backing off from 50 ms to 1 s), and returns as soon as it reaches any other state (`running`, `stopped`, `error`,
…), `timeoutMs` passes (at most 30 seconds), or the request is aborted. A timeout is not an error: the answer is the
instance as last read.

```ts
const instance = await client.instances.create({ ...input, wait: { timeoutMs: 20_000 } });
if (instance.state !== 'running') {
  // Still starting, or failed: read it again later with instances.get, or watch instance.* events.
}
```

`wait` is kept out of the request hash, so a retry with the same key and a different `wait` replays the stored
instance instead of failing with `CONFLICT`. The MCP `create_instance` tool sends `wait` from its `waitSeconds`
(default 20).

## Read and list [#read-and-list]

[instances.get](/docs/reference/api/instances#get) returns one instance of the tenant, deleted ones included, and
sandbox and warm instances too. Drafts and other tenants' instances are `NOT_FOUND`.

[instances.list](/docs/reference/api/instances#list) returns `{ items, cursor? }`, newest first. It lists the
instances sandboxes drive, but never the warm instances of sandbox templates:

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant.', required: true },
  scope: { type: &#x22;'mine' | 'all'&#x22;, description: &#x22;mine: the caller's own instances, filtered in SQL. all: every instance of the tenant, each checked with compute:instances:read.&#x22;, default: &#x22;'mine'&#x22; },
  state: { type: 'InstanceState', description: 'Only instances in this state.' },
  poolId: { type: 'string', description: 'Only instances in this pool.' },
  limit: { type: 'number', description: '1 to 200.', default: '50' },
  cursor: { type: 'string', description: 'The cursor of the previous page (an instance id).' },
}"
/>

## Start, stop and delete [#start-stop-and-delete]

[start](/docs/reference/api/instances#start) and [stop](/docs/reference/api/instances#stop)
(`compute:instances:update`) set `desiredState` to `running` or `stopped`, and
[delete](/docs/reference/api/instances#delete) (`compute:instances:delete`) sets it to `deleted`. Each returns the
instance at once, before anything has happened, and wakes the reconciler; a hosted instance's node also gets a new
NodeSpec. Watch `state`, or the `instance.*` [events](/docs/guides/events), to see the change land.

* `start`, `stop` and `delete` (and `resize`) on a sandbox's instance or a warm instance are `INVALID_STATE`; see
  [sandbox and warm instances](#sandbox-and-warm-instances).
* `start` or `stop` on an instance that is being deleted is `INVALID_STATE`.
* `start` on an instance in `error` that has a host or a provider machine retries it: it moves to `starting` and the
  runtime or provider is asked again. An instance that failed before it was placed stays in `error`; delete it and
  create a new one. An instance whose provider create failed (`ProvisionFailed`) is refused with `INVALID_STATE`
  (`details.condition` and `details.reason`) and left unchanged.
* `delete` is idempotent. A deleted instance stays readable with `state: 'deleted'`; deleting it again returns it.
* Deletion releases everything the instance held: the capacity allocation, the IP lease and the quota, and its
  authorization row.
* When the IAM tenant is deleted, `vps().tenantDeleted` drops the tenant's drafts and sets every live instance,
  sandboxes' and warm ones included, to `desiredState: 'deleted'` (up to 100 per call), answering `SYNC_AGAIN` until
  all are `deleted`. Hosted instances hold inline allocations, which capacity's own teardown does not reach.

A hosted instance is `deleted` only once its runtime has dropped it: the next complete report from the host no
longer lists it. If the host is `lost`, or the instance was never placed, deletion finishes without waiting. A
provider instance is `deleted` once the provider's `delete` succeeds or finds nothing to delete.

### From the command line [#from-the-command-line]

The `instances` commands (44915e5) wrap the same methods as the token's identity:

```bash
better-compute instances create --tenant acme --pool pool_01jz… --flavor g1.small --image ubuntu-24.04 \
  --ssh-key ./keys/alice.pub,./keys/ci.pub --user-data ./cloud-init.yaml
better-compute instances list --tenant acme --all --state running
better-compute instances get inst_01k5w… --tenant acme --query state
better-compute instances stop inst_01k5w… --tenant acme
better-compute instances resize inst_01k5w… --tenant acme --flavor g1.large
better-compute instances start inst_01k5w… --tenant acme
better-compute instances delete inst_01k5w… --tenant acme
```

* `create` reads `--ssh-key` from files (comma-separated paths; every non-empty line that is not a comment is a key)
  and refuses a file that holds a private key. `--user-data` is a cloud-init file, or a `secret://<tenant>/<id>`
  reference passed as it is. `--stopped` creates it with `desiredState: 'stopped'`.
* `create` waits up to `--wait` seconds (default 30, the API's maximum) for the instance to settle, and sends a new
  idempotency key on every run unless you pass `--key`: repeat the command with the same `--key` to retry safely.
* `list` shows your own instances; `--all` asks for the tenant's (`scope: 'all'`).
* Instance ids that do not look like `inst_…` are usage errors (exit 2) before anything is sent.

See [the instances commands](/docs/reference/cli#instances-create) for every flag.

## Resize [#resize]

[instances.resize](/docs/reference/api/instances#resize) moves a stopped hosted instance to another flavor of its pool,
on the same host. It exists so a tenant can grow a VM without rebuilding it.

```ts
await compute.api.instances.stop({ token }, { tenantId: 'acme', instanceId });
// … wait until state is 'stopped' …
await compute.api.instances.resize({ token }, { tenantId: 'acme', instanceId, flavor: 'g1.large' });
```

The call charges the growth to quota first, then resizes the allocation in place on the node, then refunds any
shrinkage. It is all or nothing: if the host lacks room, the quota charge is undone.

| Code                   | When                                                                                                                        |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `UNSUPPORTED`          | The instance is in a provider pool, or the new flavor has a different GPU count.                                            |
| `INVALID_STATE`        | The instance is not stopped (both `state` and `desiredState`), holds no allocation, or belongs to a sandbox or a warm pool. |
| `NOT_FOUND`            | The flavor is missing or archived.                                                                                          |
| `QUOTA_EXCEEDED`       | The growth does not fit the tenant's quota.                                                                                 |
| `CAPACITY_UNAVAILABLE` | The host has no room; `details.resource` names what ran out.                                                                |

The instance passes through `resizing` and is `stopped` again when the call returns, with its policy attributes
updated. See [flavors](/docs/guides/vps/flavors) for how shapes become claims.

## Sandbox and warm instances [#sandbox-and-warm-instances]

[Sandboxes](/docs/guides/vps/sandboxes) (v0.5) are built on instances, and migration `vps/0004_sandboxes` marks the
instances they own with two columns, never both set:

| Column                       | The instance is                                        | Created by                                                             |
| ---------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------- |
| `instances.sandbox_id`       | The one instance a sandbox drives                      | `sandboxes.create` (a cold create), or a warm claim that moves it here |
| `instances.warm_template_id` | A warm instance of a sandbox template, not claimed yet | The `vps.sandboxes` reconciler, owned by the template's author         |

Such an instance is admitted, placed, charged to quota, addressed, metered and deleted exactly like one from
`instances.create`, but only its sandbox or its template moves it. `instances.start`, `stop`, `resize` and `delete`
refuse it with `INVALID_STATE` ("Instance … is managed by sandbox …" or "… is a warm instance of sandbox template
…"); use `sandboxes.delete` or `sandboxTemplates.update` instead. `instances.get` returns both kinds, and
`instances.list` returns a sandbox's instance but hides warm instances. The `Instance` view does not name the sandbox;
the sandbox's `instanceId` points the other way. The instance events vps publishes for them carry `data.sandboxId`
or `data.warmTemplateId`.

`tunnels.open` refuses both kinds too, for every op, with `INVALID_STATE` and `details.reason: 'sandbox-managed'`,
so `compute:instances:exec` on the instance cannot bypass the sandbox's own decision. The way into a sandbox is
[`sandboxes.exec`](/docs/guides/vps/sandboxes#run-code-inside), decided as `compute:sandboxes:exec` on the sandbox;
a warm instance takes no tunnels at all.

### Paused instances [#paused-instances]

`paused` is an instance state and a desired state (v0.5): vCPUs stopped, memory resident, the host and allocation
kept. No API method asks for it; only warm pools do, on runtimes whose `RuntimeTraits.pause` is true. The lifecycle
converges it like the other settled states: from `provisioning` it walks through `starting` and `running`, then
pauses (`pause`); from `paused` it resumes to `running` (`resume`) or stops. A host reports it as
`InstanceObservedState.PAUSED`, and a `lost` instance can recover to `paused`. A runtime without `pause` treats a
desired `paused` as `stopped`, and vps never asks one for it: warm pools on such runtimes stay `stopped`.

## The reconciler [#the-reconciler]

`vps.instances` runs one instance per work item, keyed by the instance id. Every API change, every host report that
changes something and every provider answer enqueues it again, and it schedules its own next look:

| Situation                                 | Next look       |
| ----------------------------------------- | --------------- |
| Nothing fits a hosted instance yet        | 30 s            |
| Transitional state (waiting on a runtime) | 10 s            |
| Steady provider instance (drift polling)  | 60 s            |
| Provider call failed and will be retried  | 30 s            |
| Settled hosted instance, error, deleted   | Only when woken |

### Hosted pools [#hosted-pools]

The API never calls a runtime. The reconciler changes the database, and the host acts on its NodeSpec:

<Mermaid
  chart="sequenceDiagram
  participant A as API
  participant R as vps.instances
  participant G as Gateway
  participant H as Host runtime
  A->>A: place, lease IP, state provisioning
  A->>G: NodeSpec gains the instance
  G->>H: desired state
  H->>G: StatusReport with instances
  G->>R: reportInstances, enqueue
  R->>R: provisioning, starting, running"
/>

Each hosted node's `instances` section lists every live instance placed on it (not `pending`, `scheduling` or
`deleted`) with its runtime, flavor, image (`url`, `sha256`, `format`), its user data as a `secret://` reference,
the GPU atoms and id of its allocation (`allocationId`), and one NIC (`nics`): the host bridge (`br0`, or
`vps({ hostNetwork })`), a stable locally administered MAC derived from the instance id, and its IPAM address in CIDR
form. NIC order is the guest's interface order and is kept. An instance being deleted is sent with
`desiredState: 'deleted'`.

User data never travels in the spec. At admission in a hosted pool, vps renders the cloud-config (the hostname from
the instance id and `sshKeys`, unless you passed `userData`), seals it through the secret broker and keeps only its
`secret://` reference (`instances.user_data_sealed`, migration `vps/0002_sealed_user_data`); a tenant's own reference
(one it wrote with `secrets.put` with the default `tenant` scope, checked in the create transaction; a fleet secret
is refused) is kept as it is, and only a sealed one is
deleted with the instance. The host fetches the value with a node-signed
`SecretRequest` naming the instance's allocation, and the broker releases it only when the secret belongs to the
instance's own tenant. The guest gets no join token, no `bcd` and no IAM key.

From the reports, the reconciler moves the instance:

* When the reported state equals `desiredState`, it walks `state` there: `provisioning` to `starting` to `running`,
  `running` to `stopped` and back, or `running` to `paused` and back for warm instances. Each step publishes
  `instance.<state>`.
* A reported `error` moves it to `error` with a `RuntimeError` condition and publishes `instance.failed`.
* A reported `stopped` whose message starts `ResourcesExhausted:` is bcd's host admission refusing the start: the
  instance gets a `HostResourcesExhausted` condition and, if nothing of its tenant's is on that host yet, is placed
  elsewhere after a minute (see [hosts that cannot start an instance](#hosts-that-cannot-start-an-instance)).
* While the report and `desiredState` disagree, it waits and looks again in 10 seconds.

> **The host side is written, not yet run on real hardware.** 
  bcd's node agent is built and tested but not released. It has the `instances` converger (the FakeHypervisor's
  rules, user data fetched with a `SecretRequest` bound to the allocation, every domain in its status reports, the
  instance network policy as host nftables rules), and a host build runs instances on the built-in `libvirt-kvm`
  runtime when `virsh` is installed (362317c) and on `cloud-hypervisor` when the `cloud-hypervisor` binary is
  (2598f0a): qcow2 overlays on sha256-verified base images, NoCloud seeds, taps, VFIO GPUs. Both runtimes have been
  tested only against fake tools (a fake `virsh`, a fake cloud-hypervisor API, a fake `qemu-img`); no VM has booted
  through either on a real KVM host yet. A host with neither and without a bcx runtime plugin fails a non-empty `instances` section with `this host
  runs no instance runtime`. bcd also refuses on its own a start whose memory (or huge pages) the host cannot back
  (0dd122a): the instance stays stopped with a status message starting `ResourcesExhausted:`, which is not an error,
  and bcd tries again on its next pass (see [host-side admission](/docs/daemon/delivery#instances)); since 564f543 vps
  records it and moves a new instance elsewhere (see
  [hosts that cannot start an instance](#hosts-that-cannot-start-an-instance)). A flavor's [placement](/docs/guides/vps/flavors#placement)
  reaches the host as `InstanceSpec.placement` (9d55b36). fakehw builds run instances on a file-backed runtime of the same name, which
  `tests/e2e/bcd-vps.test.ts` drives through the control plane; see [delivery](/docs/daemon/delivery#the-runtimes). The
  TypeScript FakeDaemon plays a VPS host on a fake hypervisor (domains, overlay disks, NoCloud seeds, taps), fetching
  user data through the broker, and `tests/e2e/vps.test.ts` runs hosted instances through the real server and gateway
  with it.

### Hosts that cannot start an instance [#hosts-that-cannot-start-an-instance]

The control plane books a host's capacity, but the host can still lack room for a start: memory or huge pages
taken by something the booking does not see, or a booking and a host that drifted apart. bcd's host admission then
leaves the instance stopped with a message starting `ResourcesExhausted:` and retries on its next pass. Since
564f543 vps acts on that report instead of leaving a new instance `provisioning` on that host for good:

1. **The condition.** `reportInstances` records `HostResourcesExhausted` on the instance: `reason` is the reporting
   node, `message` bcd's words (up to 512 characters). It is cleared as soon as that node reports the instance in
   any other way.
2. **Waiting a minute.** Once the refusal has stood for `exhaustedReplaceMs` (60 seconds), the reconciler re-places
   the instance if nothing of its tenant's is on that host yet: an instance still in `provisioning`, or a warm
   instance in `stopped` that no sandbox has claimed or whose sandbox has not become ready. It never moves one whose
   `desiredState` is `stopped`, one that already ran there, or one whose volumes live on that host.
3. **Placing it elsewhere.** It releases the allocation and the IP lease, invalidates the old host's NodeSpec (which
   now drops the instance, so that host removes the domain it could not start), and moves the instance back to
   `scheduling` (`recreate`), publishing `instance.recreating` with `reason: 'resources-exhausted'` and
   `fromNodeId`. The `ExhaustedHosts` condition lists every host that refused it (comma-separated in `message`,
   oldest first), and for `exhaustedAvoidMs` (10 minutes) after the last re-placement, scheduling passes them to
   capacity as `excludeNodes`.
4. **Giving up.** After `maxExhaustedReplacements` (3) re-placements, the next refusal that stands a minute fails
   the instance: `error`, and `instance.failed` with `reason: 'resources-exhausted'` and the `nodeId`.

An instance that is not re-placed keeps waiting with `HostResourcesExhausted`, and bcd keeps retrying the start.
Both conditions are removed once the host reports the instance in its desired state. A sandbox whose instance is
re-placed keeps its TTL: the clock does not restart. Tested in `tests/vps-instances.test.ts` (re-placement away from
the host, the growing host list and the cap, a refused restart that waits) and `tests/vps-sandboxes.test.ts`.

### Provider pools [#provider-pools]

There is no node. Each round is three steps, so no network call ever runs inside a database transaction:

1. **Plan** in a short transaction: lock the instance, start deletion if `desiredState` is `deleted`, and read the
   provider account, only if it is the pool owner's. A pool row naming another tenant's account (possible in
   0.1.x) never drives it: such an instance is not created, polled or powered (`INVALID_STATE`, retried), and
   deleting it finishes without a provider call, leaving any machine it made for the account's owner to remove.
2. **Call** the account's `ProviderImpl` with the instance id as the idempotency key.
3. **Apply** the answer in a second transaction, only if the instance's `version` has not moved; otherwise try again
   at once.

| Instance                                      | Provider call                                                                                       |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `scheduling`, no machine yet                  | `create` with the flavor's `offerId`, the `imageId`, rendered user data and the instance id as name |
| `running`, wanted `stopped`, reported running | `power(id, 'graceful-off')`, then `get`                                                             |
| `stopped`, wanted `running`, reported stopped | `power(id, 'on')`, then `get`                                                                       |
| attached, not claimed yet                     | `claim(id, instanceId)`, which writes the ownership label                                           |
| `running`, a reboot asked for                 | `power(id, 'graceful-restart')`, or `'reset'` when hard, then `get`                                 |
| `deleting` with a machine                     | `delete` (`NOT_FOUND` counts as done)                                                               |
| `deleting`, attached                          | `release` (`onDelete: 'detach'`), or `claim` with `managed` then `delete`                           |
| Anything else with a machine                  | `get`                                                                                               |

The machine's `id` becomes `externalId`, its addresses replace `addresses`, and its state maps to `running`,
`stopped`, `error` or, for `deleted`, absent. The first answer moves `scheduling` to `provisioning` and publishes
`instance.provisioning`.

A failed call records a `ProviderError` condition with the error code as reason and the message (up to 500
characters). `UNSUPPORTED`, `INVALID_STATE` and `INVALID_CONFIG` move the instance to `error`; anything else,
`RATE_LIMITED` and `PROVIDER_ERROR` included, is retried in 30 seconds. The exception is a create the provider
refuses for capacity or quota: `CAPACITY_UNAVAILABLE` or `QUOTA_EXCEEDED` from `create`, or from `get` while the
instance is `scheduling` or `provisioning` (GCE reports a stockout after accepting the insert), or refuses as invalid
(`INVALID_INPUT` from `create`, such as an image from a project the account does not allow), moves it to `error`
with a `ProvisionFailed` condition naming the code and publishes `instance.failed` with that `reason`. Nothing more is
asked of the provider, `start` is refused with `INVALID_STATE` (delete it and create again), and its quota stays charged until it is deleted, which refunds it.
An attached instance whose machine lost its ownership label fails with `OwnershipLost` and is never acted on again. A machine that `get` no longer finds (or
reports `deleted`) makes the instance `lost` and publishes `instance.lost`; the reconciler stops polling it, and deleting it
refunds its quota. A Spot instance is the exception below.

#### Spot preemption [#spot-preemption]

An instance created on Spot capacity, an offer id ending in `@spot` on a provider that declares the `spot-offers`
capability (DigitalOcean with `allowSpot`), was reclaimed by the provider when its machine goes. It still moves to
`lost`, and also gets a `Preempted` condition (`status: 'true'`, reason `Reclaimed`) and the event
`instance.preempted` (`data.externalId`, the reclaimed machine). An attached instance is never treated as preempted,
whatever its offer (better-compute did not create that capacity), and neither is an `@spot` offer on a provider without
the capability, where the suffix may name something else.

With the pool policy `replacePreempted: true` the instance is recreated in place: it goes back to `scheduling` with no
machine (`instance.recreating` with `reason: 'preempted'`), keeps its id, and the next step creates a new machine.
`Preempted` turns to `status: 'false'` (reason `Replaced`), and a later reclaim sets it again. The condition
`PreemptedReplacements` counts the replacements as `<generation>/<consecutive>`:

* **Provider key.** A replacement's provider calls carry the key `<id>:r<generation>` instead of the instance id, so a
  provider that dedupes creates by key (a GCE request id, a DigitalOcean key tag that still lists the reclaimed Droplet)
  never hands back the reclaimed machine. The generation only grows.
* **Backoff.** The first replacement in a row is immediate; the second waits 30 seconds, and each one after it twice as
  long.
* **Cap.** `consecutive` counts replacements that never reached `running` and is reset once one does. After three in a
  row a fourth reclaim leaves the instance `lost` with `Preempted` set; delete it to refund its quota.

A replacement whose create the provider refuses for capacity or quota (`CAPACITY_UNAVAILABLE`, `QUOTA_EXCEEDED`) ends
in `error` with `ProvisionFailed`, as any create does.

## Reports from hosts [#reports-from-hosts]

The gateway hands each node's coalesced `StatusReport` to `instanceStatusHandler`, inside its ingest transaction. The
report's `instances` lists every VM the host's runtimes hold, managed or not:

* States map from `InstanceObservedState`: `RUNNING`, `STOPPED`, `PAUSED` and `ERROR`. `UNSPECIFIED` entries are
  skipped.
* Only instances placed on the reporting node are touched, so one node can never move another node's instances.
* Reported addresses are added to the instance's `addresses`.
* With `instancesComplete: true` the list is the whole inventory: a placed instance missing from it is marked
  absent. That is how deletion completes. An empty, incomplete report changes nothing.

You can call the same seam yourself, which is what the tests and the churn acceptance do to emulate hosts:

```ts
import { reportInstances } from '@better-compute/vps';

// `work` is the runtime's work queue (ModuleContext['work']): changed instances are enqueued on it.
await db.transaction((tx) =>
  reportInstances(tx, work, nodeId, [{ id: instanceId, state: 'running', addresses: ['fe80::1'] }], {
    complete: true,
  }),
);
// → { updated: 1, orphans: [] }
```

### Orphans and leak alerts [#orphans-and-leak-alerts]

An instance the runtime holds that no live row places on that node is an orphan: a leaked VM, overlay or tap, or one
reported by the wrong host. `reportInstances` returns orphan ids sorted, and never changes anything for them.

`instanceStatusHandler({ events })` publishes `instance.orphaned` with subject `node/<id>` and
`data: { nodeId, instanceIds }` (up to 100 ids). It announces a node's orphan set only when the set changes; hosts
report about once a second and leaks persist. The last announced set is kept in memory, so a restarted process
announces current leaks once more. Without `events`, orphans go to `log` (default `console.warn`).
`vpsInstanceStatusHandler` is the handler with no event publisher.

The v0.3 acceptance checks the other direction too: after 200 instances churn on 3 emulated hosts, every runtime
domain is a live placed instance and every allocation, IP lease, scalar claim and quota counter matches the live
instances (`tests/vps-churn.test.ts`).

## Lost hosts [#lost-hosts]

A host becomes `lost` only when its `vps-host` pool sets `policy.lostAfter`: the liveness controller then marks it
`lost` that long after it became unreachable (see [operating nodes](/docs/guides/nodes/operating-nodes)). Without
`lostAfter` a host stays `unreachable` and its instances wait for it. When a hosted instance's node becomes `lost`,
the instance moves to `lost` and `instance.lost` is published with the `nodeId`. The reconciler then looks every 60
seconds:

* If the node reconnects and reports the instance `running`, `stopped` or `paused`, it recovers to that state
  (`instance.recovered`).
* If the pool's policy has `recreateOnHostLoss: true`, the instance releases its allocation and IP lease, goes back to
  `scheduling` (`instance.recreating`) and is placed again, on a host that is still schedulable, where it boots
  from its image.

The same `recreate` verb also moves an instance in `provisioning`, or a warm one in `stopped`, off a host whose bcd
cannot start it (564f543; `instanceStateTable` allows `recreate` from those two states); see
[hosts that cannot start an instance](#hosts-that-cannot-start-an-instance).

`recreateOnHostLoss` is read from the pool's stored `policy` document; it is not part of the `PoolPolicy` type yet,
so a typed caller setting it through `pools.create` or `pools.update` needs a cast.

## Runtimes [#runtimes]

A runtime is the driver that runs VMs on a host, under the `runtime` driver kind that importing `@better-compute/vps`
adds to `DriverKinds`. It runs on the node (`runsOn: 'node'`), and every operation is idempotent by instance id:

```ts title="@better-compute/vps"
interface RuntimeDriver {
  readonly traits: RuntimeTraits; // isolation 'vm' | 'microvm' | 'container', multiTenant, gpuPassthrough, imageFormats
  list(): Promise<RuntimeInstanceStatus[]>; // everything it holds, so leaks are found by diffing
  apply(spec: InstanceSpec): Promise<RuntimeInstanceStatus>; // cache the image, build the overlay and NoCloud seed, start/stop/resize
  reboot(instanceId: string): Promise<void>;
  remove(instanceId: string): Promise<void>; // an absent instance counts as removed
  console?(instanceId: string): Promise<DuplexStream>; // the serial console, for tunnels
  // v0.5, for sandboxes: warm pools and snapshots
  pause?(instanceId: string): Promise<void>; // vCPUs stopped, memory resident
  resume?(instanceId: string): Promise<void>;
  snapshot?(instanceId: string, target: RuntimeSnapshotTarget): Promise<RuntimeSnapshotResult>;
}
```

`pause` and `resume` serve [sandbox warm pools](/docs/guides/vps/sandboxes#warm-pools): a runtime that implements
them sets `pause: true` in its `RuntimeTraits`, and `builtinRuntimeTraits` does so for `libvirt-kvm`,
`cloud-hypervisor` and `firecracker` (not `containerd`). vps then keeps warm instances `paused` on that runtime; see
[paused instances](#paused-instances). `snapshot` and `InstanceSpec.restoreFrom` serve
[sandbox snapshots](/docs/guides/vps/sandboxes#snapshots-and-restore): `sandboxes.snapshot` sends the host an
`instance.snapshot` command, and `sandboxes.create({ fromSnapshot })` boots a new instance from one. A runtime that
implements `snapshot` sets `snapshot: true` in its `RuntimeTraits` (the three built-in VM runtimes do); on any other,
`sandboxes.snapshot` is `UNSUPPORTED` up front. No `instances.*` method snapshots a plain instance.

`libvirt-kvm` and `cloud-hypervisor` are the runtimes built into bcd (v0.3), which its `instances` converger drives
through its `Runtime` interface. Both are written, and a host build runs each one whose tools are on its `PATH`
(there is no configuration key):

* `libvirt-kvm` (362317c), with `virsh`: `virsh -c qemu:///system` and `qemu-img`, a host-passthrough CPU, domains
  marked as bcd's in their metadata (others are reported as unmanaged and never touched), and `iso` images booted
  from a cdrom.
* `cloud-hypervisor` (2598f0a), with the `cloud-hypervisor` binary: one VMM per instance, started as a confined
  systemd transient unit `bcd-ch-<id>.service` in `bcd-instances.slice` so the VM outlives bcd, and driven over its
  REST API; bcd's own `vm.json` per instance says what it owns, since cloud-hypervisor keeps no registry. It boots
  disk images only (`iso` is refused), and needs `systemd-run` and the firmware at
  `/usr/share/cloud-hypervisor/CLOUDHV.fd`, which `bcd doctor` checks.

Both runtimes treat a tenant's image as hostile (security review 12, fixed in 933c51b and hardened in b12c217). bcd
reads a `qcow2` base image's header in Go when it is downloaded and again each time an instance uses it, with no tool
probing the image: one that names a backing file or an external data file, sets an unknown incompatible feature, is
encrypted or is not a qcow2 is refused, which fails that instance. `raw` and `iso` images are opened with their
format stated and never probed. An instance's disk must name exactly its recorded base before every start, snapshot
and resize, and a restored snapshot's disk must name none. On `cloud-hypervisor` bcd creates each instance's taps on
their bridge before its VMM starts and hands them to the VMM as file descriptors, so a guest reboot never puts a tap
outside the instance firewall; the VMM runs as the instance's own uid with no capabilities and no network, its unit
sees only its own instance directory and base image with `/run` and `/var` hidden, opens only KVM and (with GPUs)
VFIO devices, and is bounded by the flavor's memory, vCPUs and a task limit; and a memory snapshot restores only
into an instance with the same vCPUs and memory. See
[the runtimes](/docs/daemon/delivery#the-runtimes).

Both give a 60-second graceful shutdown, VFIO GPUs, a serial console for instance console tunnels, and snapshots and
restores that refuse another runtime or CPU model; see [the runtimes](/docs/daemon/delivery#the-runtimes). Both have
been tested only against fakes, not on real hardware. fakehw builds run a file-backed runtime named `libvirt-kvm`
instead.
Out-of-process runtimes are served too: on `bcd node`, a bcx plugin whose driver lists the kind `runtime` becomes a
runtime (`list`, `apply(spec)`, `remove(id)` over `Invoke`; no snapshots), selected by pools like any other. No such
plugin (a Kata runtime is PLAN.md's walkthrough 4) is written yet; see
[walkthrough 4](/docs/extending/walkthroughs#walkthrough-4-a-kata-runtime-over-bcx). `vps({ runtimes })` only names the ids: the first one is recorded on every new
hosted instance and sent in its spec.

A hosted instance is placed only on hosts carrying the node label `bc.dev/driver.<runtime>`, in any version
(`runtimeNodeLabels`), for plugin runtimes and, since 0fca413, for bcd's built-ins (`libvirt-kvm`,
`cloud-hypervisor`) too. The gateway writes that label from the drivers a daemon advertises in `Hello`; every bcd
since v0.3 (fab94d4) advertises each runtime it runs, and runs one only where its tools are (`cloud-hypervisor` under
`bcd helper` only since 7fe74a1). A pool with no host advertising the runtime leaves the instance `Unschedulable`,
with a message naming the missing label; see [placement](/docs/guides/capacity/placement#eligibility-filters).

## Events [#events]

Instance events have subject `instance/<id>` and carry `state`, `desiredState` and `poolId` (plus `sandboxId` or
`warmTemplateId` for [sandbox and warm instances](#sandbox-and-warm-instances), and `reason: 'tenant deleted'` on
`instance.deleting` during a tenant teardown):

| Event                                                         | When                                                                                                                                                                       |
| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `instance.created`                                            | The create was admitted.                                                                                                                                                   |
| `instance.scheduled`                                          | A hosted instance was placed (`data.nodeId`).                                                                                                                              |
| `instance.provisioning`                                       | A provider machine exists (`data.externalId`).                                                                                                                             |
| `instance.starting`, `instance.stopping`, `instance.deleting` | `start`, `stop` or `delete` was called.                                                                                                                                    |
| `instance.running`, `instance.stopped`, …                     | The reconciler moved `state` (named after the new state).                                                                                                                  |
| `instance.resized`                                            | A resize finished (`data.flavor`).                                                                                                                                         |
| `instance.failed`                                             | The instance entered `error` (`data.reason`; `resources-exhausted` with `data.nodeId` after three re-placements).                                                          |
| `instance.lost`, `instance.recovered`, `instance.recreating`  | Host or machine loss, and what followed.                                                                                                                                   |
| `instance.preempted`                                          | A Spot machine was reclaimed (`data.externalId`); with `replacePreempted`, `instance.recreating` (`reason: 'preempted'`) follows. See [Spot preemption](#spot-preemption). |
| `instance.recreating` with `reason: 'resources-exhausted'`    | The host's bcd could not start it for a minute; placed elsewhere (`data.fromNodeId`). See [hosts that cannot start an instance](#hosts-that-cannot-start-an-instance).     |
| `instance.deleted`                                            | Everything it held was released.                                                                                                                                           |
| `instance.orphaned`                                           | A host holds instances no row places there (subject `node/<id>`).                                                                                                          |

## Access to an instance [#access-to-an-instance]

Exec, the serial console, SSH and TCP port forwards into a hosted instance go through a
tunnel: `tunnels.open({ tenantId, instanceId, op })` with `op` `exec`, `console`, `ssh` or
`tcp` (plus a `port`) is decided as `compute:instances:exec` on the instance and returns a single-use
exec ticket for the instance's node, which carries the instance (`inst`). The instance
must be `running` and its node connected; provider instances are `UNSUPPORTED`, since no platform agent runs next to
them. `console`, `ssh` and `tcp` also need the node's daemon to advertise `tunnel.instance-dial`, or they are
`UNSUPPORTED` with `details.reason: 'daemon-capability'`. The method is registered only when the deployment
configures tunnels. See [tunnels](/docs/guides/access/tunnels) and [SSH](/docs/guides/access/ssh).

`console` (798e710) attaches to the instance's serial console, which bcd opens from its runtime (`libvirt-kvm` hands
it the pty of the domain's serial port, `cloud-hypervisor` its serial socket), never a host shell. bcd refuses `exec` into an instance until guests have an
agent, so the serial console is the terminal that works on a real host: `better-compute console --instance ID` from
a terminal, or **Serial console** (`/instances/<id>/console`) on the instance's page in [the console](/docs/guides/clients/console).
Sandbox and warm sandbox instances stay refused for every op (above).

## Workload identity [#workload-identity]

A hosted instance can prove who it is to a third party (Better IAM's `sts/assumeRoleWithWebIdentity`, a cloud STS)
with a short-lived OIDC token, `sub = "t:<tenantId>:instance:<instanceId>"`, that it fetches from its host's
metadata service (IMDS). This is v0.5 and optional: the control plane half is built and tested end to end against
the FakeDaemon (`tests/e2e/imds.test.ts`). `bcd node` has both host halves: the session half (`RequestWorkloadToken`:
a node-signed `WorkloadTokenRequest` with a one-time key, the sealed answer opened and checked against the instance
and request id) and the metadata service, served on `[imds] listen` in its `config.toml` (for example
`169.254.169.254:80` on the instances' bridge). It resolves the caller as the running instance whose address is the
request's source. It serves the instances of whichever runtime bcd runs; since neither built-in runtime has run on
real hardware yet, no real VM has reached it, and its Go tests and fakehw builds cover it.

<Mermaid
  chart="sequenceDiagram
  participant I as Instance
  participant H as Host (IMDS in bcd)
  participant G as Gateway
  participant B as Workload broker (api)
  I->>H: PUT /latest/api/token (x-bc-metadata-token-ttl-seconds)
  H-->>I: session token, HMAC-bound to this instance
  I->>H: GET /latest/identity-token?audience=… (x-bc-metadata-token)
  H->>G: WorkloadTokenRequest (node-signed, one-time key)
  G->>B: relay
  B-->>G: WorkloadTokenResponse (JWE to the one-time key)
  G-->>H: relay
  H-->>I: the workload JWT"
/>

* **The metadata service.** The host serves each instance on its own interface and knows the instance from where
  the request arrived, never from a request field. `PUT /latest/api/token` with `x-bc-metadata-token-ttl-seconds`
  (1 to 21,600) returns a session token bound to that instance; `GET /latest/identity-token?audience=…` with
  `x-bc-metadata-token` returns the token. A session token from one instance fails for any other, and requests with
  `X-Forwarded-For` or `Forwarded` are refused, which blocks the usual SSRF relays.
* **The host holds no signing key.** It sends a `WorkloadTokenRequest` over its session, signed by its node key, and
  gets the token sealed to a one-time key, so the gateway only relays ciphertext.
* **The broker mints only when** the request is signed by the node's current certificate of role `node` (never
  `node-tenant`, a ramdisk, a site agent or a kind-restricted certificate) naming the session's node; the instance
  is live, placed on that node, with its live allocation there; its pool has `policy.workloadIdentity: true`; its
  tenant is neither suspended nor deleted (otherwise `disabled`, recorded with the reason `tenant-suspended` or
  `tenant-deleted`); the audience is the issuer's default or a configured one; the request id is new; and the instance is within its rate
  (30 tokens a minute by default). Tokens live at most 15 minutes.
* **Refusals are answers**, not errors: `not-bound`, `disabled`, `audience`, `rate-limited`, `access-denied` or
  `unavailable`. Each one publishes `security.workload.refused` (rate-limited per reason); each token issued writes
  the audit row `workload.token.issued` with its `jti`, never the token.

Turn it on with a workload issuer from `@better-compute/pki` and the pools that should use it:

```ts title="compute.ts"
import { createWorkloadIssuer } from '@better-compute/pki';

export const compute = betterCompute({
  // …
  workloadIdentity: {
    issuer: createWorkloadIssuer({
      issuer: 'https://compute.example.com/workload', // https on 443
      defaultAudience: 'https://iam.example.com',
    }),
    audiences: ['sts.example.com'], // besides the default
    ttlSeconds: 900, // default and maximum
    perInstancePerMinute: 30,
  },
});
```

`better-compute serve` publishes the issuer's discovery document and JWKS under the issuer URL's path, so relying
parties can verify the tokens. See [workload identity](/docs/operations/pki#workload-identity) for the keys and
their rotation.


# Networking (/docs/guides/vps/networking)

> How VPS instances get addresses, from one IPv4 range per hosted pool to what runtimes and providers report, and tenant networks (VPCs, subnets, firewall rules) on provider accounts.



A VM is not much use until you know its address. Better Compute keeps networking deliberately small: an address
plan per hosted pool, whatever the runtime or the provider reports, and, on provider accounts that support them,
[tenant networks](#tenant-networks) the provider builds. Software-defined networking of its own beyond bridges, VLANs
and IPAM is a non-goal for v1, as is live migration.

Where an instance's addresses come from depends on its pool:

| Pool                        | Addresses                                                                                                |
| --------------------------- | -------------------------------------------------------------------------------------------------------- |
| Hosted (`vps-host`)         | One IPv4 lease from the pool's range, taken at placement, plus any addresses the host's runtime reports. |
| Hosted, no range configured | Only what the runtime reports.                                                                           |
| Provider (`provider`)       | What the provider reports for the machine, replaced on every poll.                                       |

## Give a hosted pool a range [#give-a-hosted-pool-a-range]

Pool IPAM is one IPv4 range per hosted pool. You configure it with `setIpPool`, a function exported by
`@better-compute/vps`. There is no API method, CLI command or FleetConfig field for it in this release, so run it
from your own setup code against the compute database:

```ts title="setup-ipam.ts"
import { setIpPool } from '@better-compute/vps';

const range = await compute.context.sql.transaction((tx) =>
  setIpPool(tx, { poolId: 'pool_01jz…', cidr: '10.0.0.0/24', gateway: '10.0.0.1' }),
);
// → { poolId: 'pool_01jz…', cidr: '10.0.0.0/24', gateway: '10.0.0.1',
//     firstHost: '10.0.0.2', lastHost: '10.0.0.254' }
```

<TypeTable
  type="{
  poolId: { type: 'string', description: 'The hosted pool.', required: true },
  cidr: { type: 'string', description: 'The IPv4 network, such as 10.0.0.0/24. IPv6 ranges are refused.', required: true },
  gateway: { type: 'string', description: 'The gateway address. It is never leased, even when it falls inside the range.', required: true },
  firstHost: { type: 'string', description: 'The first address to lease.', default: 'the address after the gateway' },
  lastHost: { type: 'string', description: 'The last address to lease.', default: 'the address before the broadcast address' },
  dns: { type: 'string[]', description: &#x22;DNS servers for the guests (ip_pools.dns, migration vps/0019_ip_pool_dns; 387c22f). Empty: guests keep their image's resolvers.&#x22;, default: '[]' },
}"
/>

`firstHost` and `lastHost` must both lie inside `cidr`, with `firstHost` not after `lastHost`. Anything else,
including a malformed address, is `INVALID_INPUT` naming the pool. Because the default `firstHost` is the address after
the gateway, set it explicitly when the gateway sits at the end of the network (`10.0.0.254` in a `/24`).

Calling `setIpPool` again for the same pool replaces the range. Existing leases are kept as they are, even if they now
fall outside it.

## How leases work [#how-leases-work]

A hosted instance takes its lease in the same transaction that places it, right after capacity claims its resources on
a host (see [instances](/docs/guides/vps/instances)):

* The lease is the lowest free address from `firstHost` to `lastHost`, skipping the gateway.
* An instance holds at most one lease. Placing an instance that already holds one returns the same address.
* `(pool, address)` is the primary key of `ip_leases`, so two instances can never hold the same address. Concurrent
  placements that race for one address retry, up to 8 times, on the next free one.
* When the range is exhausted, placement fails with `CAPACITY_UNAVAILABLE` ("Pool … has no free IP address"). In
  `instances.create` the whole transaction rolls back, host claims included, and the caller gets the error. When the
  reconciler places an instance later (after a host loss), the instance moves to `error` with an `Unschedulable`
  condition.
* A pool without a range leases nothing, and placement succeeds without an address.

The lease is returned when the instance is deleted. After a host loss with `recreateOnHostLoss`, the instance returns
its lease with its allocation and takes a new one, again the lowest free address, when it is placed on another host.

For example, with `cidr: '10.0.0.0/29'` and `gateway: '10.0.0.1'`, the range is `10.0.0.2` to `10.0.0.6`: five
instances get `.2` to `.6` in order, and a sixth create fails with `CAPACITY_UNAVAILABLE` until one of them is deleted.

Operators can see the plan and the leases in the database:

```sql
SELECT cidr, host(gateway) AS gateway, host(first_host) AS first, host(last_host) AS last
  FROM ip_pools WHERE pool_id = $1;

SELECT host(address) AS address, instance_id
  FROM ip_leases WHERE pool_id = $1 ORDER BY address;
```

The v0.3 acceptance run (`tests/vps-churn.test.ts`) checks that after 200 instances churn on 3 hosts there is exactly
one lease per placed instance, none leaked and none shared.

## Addresses on an instance [#addresses-on-an-instance]

`Instance.addresses` starts with the lease, if any. After that:

* **Hosted pools.** Addresses in the host's `StatusReport.instances` (from the guest agent or DHCP leases, CIDR form
  on the wire) are added to the list. Reports only add; the lease stays first.
* **Provider pools.** Each poll replaces the list with the machine's addresses. Hetzner Cloud reports the public IPv4
  address, the public IPv6 network as Hetzner gives it (a `/64`), and private network addresses. Proxmox VE reports
  the guest agent's addresses while the VM runs, without loopback and link-local ones, and nothing without an agent.
  Proxmox clones are configured with `ipconfig0=ip=dhcp`, so the guest takes its address from your cluster network's
  DHCP.

## What reaches the host [#what-reaches-the-host]

A hosted instance's network reaches its host in `InstanceSpec.nics`, in the `instances` section of the
NodeSpec. vps sends one NIC per instance:

| Field         | Value                                                                                                         |
| ------------- | ------------------------------------------------------------------------------------------------------------- |
| `network`     | The host bridge: `br0`, or `vps({ hostNetwork })`                                                             |
| `mac`         | A stable locally administered MAC: `02` followed by the first 10 hex digits of the SHA-256 of the instance id |
| `addresses`   | The IPAM lease in CIDR form (the address with the range's prefix length), or none without a lease             |
| `gateway`     | The range's gateway, for a NIC with a lease (387c22f)                                                         |
| `nameservers` | The range's `dns`, for a NIC with a lease, when it has any (387c22f)                                          |

**The guest's network configuration.** Since 0791fc9 bcd writes a cloud-init `network-config` (version 2) into the
NoCloud seed of `libvirt-kvm` and `cloud-hypervisor` guests when a NIC has static configuration: each such NIC matched
by its MAC, its addresses static, a default route (`0.0.0.0/0`, or `::/0` for an IPv6 gateway) through its gateway, and
its nameservers. A NIC without addresses keeps DHCP, and with no static NIC at all bcd writes no `network-config`, so
the image's own fallback (DHCP on the first NIC) stays. bcd parses every value and quotes it again, so a malformed
lease fails the seed rather than adding YAML to it. A bcx runtime plugin gets each NIC's gateway and nameservers in its
spec.

The NodeSpec encoder keeps NIC order, since it is the guest's interface order. The spec also carries the instance's
`allocationId`, which the host names when it fetches the user data. The FakeDaemon's fake hypervisor creates a tap
per NIC. On a host build with `virsh`, bcd's `libvirt-kvm` runtime (362317c) attaches each NIC as a virtio device on
the named bridge with the spec's MAC, on a tap named `runtime.TapName`, so bcd's nftables rules for the NIC exist
before the tap does. bcd's `cloud-hypervisor` runtime (2598f0a) creates a persistent tap of the same name
(`ip tuntap add`), puts it on the named bridge and brings it up before the VMM starts, then gives it to the VM with
the spec's MAC (933c51b). A guest reboot reopens that same tap, a tap that cannot be attached means the VM does not
start, a NIC without a network is refused, and the taps are deleted when the VM stops or is destroyed. Both have been
tested against fake tools only, not on a real host. A VPS profile can already declare `network: { mode: 'bridge', ipam: 'pool' }`
(see [steps and profiles](/docs/extending/steps-and-profiles)), but `vps()` does not read profiles.

### bcd's nftables tables [#bcds-nftables-tables]

Every instance tap is named `bc<10 hex digits>n<nic>` (`runtime.TapName`), and bcd owns three nftables tables, which
it replaces together in one transaction before any domain starts:

| Table                  | Holds                                                                                                                                                                                                                                                                                                                                                                            |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bridge bcd_instances` | The per-tap rules: anti-spoofing (the NIC's MAC and addresses), the `private` and `none` network policies, sandboxes' SSH-from-the-host-only rule, and the drops every tap gets (DHCP server replies, router advertisements and redirects, the host's own addresses). See [sandbox networks](/docs/guides/vps/sandboxes#network)                                                 |
| `inet bcd_taps`        | A backstop (933c51b): `iifname "bc*" drop` on the input and forward hooks, and `oifname "bc*" drop` on forward and (since b12c217) output. Frames of a tap that is on its bridge reach these hooks as the bridge's, so only a tap off its bridge, which the bridge table cannot see, is dropped here. With no instance NIC on the host the table is deleted and not loaded again |
| `arp bcd_taps`         | The same backstop for ARP, which the inet hooks never see (b12c217): `iifname "bc*" drop` on input and `oifname "bc*" drop` on output. Present exactly when `inet bcd_taps` is                                                                                                                                                                                                   |

A NIC on a bridge whose name starts with `bc` would match every tap rule, so it fails its instance. bcd's 5-second
check counts the rules of all three tables against what it loaded, and under
[privilege separation](/docs/daemon#privilege-separation) the root helper accepts a ruleset that names these tables
and no other.

## Tenant networks [#tenant-networks]

On a provider pool whose provider supports networks (Google Compute Engine, DigitalOcean and AWS), a tenant builds its own private networks
through [networks](/docs/reference/api/networks) and [firewallRules](/docs/reference/api/firewall-rules), and the
provider runs them: a VPC per network, subnets in it, and firewall rules. The calls record what the tenant wants and
return at once; the `vps.networks` reconciler converges the provider.

```ts
const network = await compute.api.networks.create(
  { token },
  { tenantId, poolId, name: 'prod', idempotencyKey: 'net-prod' },
);
await compute.api.networks.addSubnet(
  { token },
  { tenantId, networkId: network.id, name: 'app', region: 'us-central1', cidr: '10.10.0.0/24' },
);
// Once the network has been ready once, instances can be created on it.
const vm = await compute.api.instances.create(
  { token },
  { tenantId, poolId, flavor, imageId, sshKeys, networkId: network.id, idempotencyKey: 'vm-1' },
);
await compute.api.firewallRules.create(
  { token },
  {
    tenantId,
    networkId: network.id,
    name: 'ssh',
    direction: 'ingress',
    rules: [{ protocol: 'tcp', ports: ['22'] }],
    sources: ['0.0.0.0/0'],
    targets: { instanceIds: [vm.id] },
  },
);
```

* **Whose it is.** A network is its tenant's (`compute-network`, a child of `compute-tenant`, decided on
  `compute:networks:*`), on a provider pool the tenant owns or is offered. Another tenant's network is `NOT_FOUND`
  in your tenant, and naming another tenant is `ACCESS_DENIED` before any decision. A rule targets only the tenant's
  own instances on that network, and `instances.create` takes only the tenant's own `networkId`: another tenant's
  is `INVALID_INPUT`, worded as for one that does not exist. The platform (root callers) may act on any tenant's
  networks. See [platform-issued compute](/docs/guides/vps/providers#platform-issued-compute).
* **Names.** A network's name is unique among the tenant's networks that are not deleted; a deleted network's name
  is free again. A retried create with the same `idempotencyKey` returns the same network for the same caller.
  At the provider every resource is named `bc-` and its lower-cased id, and each instance's machine carries the tag
  `bc-` and 16 hex digits of the sha256 of its instance id, which rules that name instances target.
* **States.** A network's `state` (`pending`, `ready`, `deleting`, `deleted`, `error`) covers the VPC and its
  subnets; each subnet and each rule has its own. A rule the provider refuses is `error` on the rule without taking
  the network out of `ready`. The reconciler publishes `network.*` events and re-applies drift every five minutes.
* **Tags.** The reconciler applies rules first, then tags the instances that own their machine (an attached one once
  claimed, never one that lost its machine); a tag set after its rule still matches. Tags set within the last hour
  are skipped, so a large network is tagged over several runs, and a tag removed by hand comes back within the hour.
* **Instances.** `instances.create` with `networkId` (and `subnetId`, or the network's only subnet in the offer's
  region) needs a network that has been provisioned (it reached `ready` once; one back in `pending` because a subnet
  or rule was added still takes instances), is not being deleted, and a `ready` subnet. A network not provisioned
  yet, or a subnet not ready yet, is `INVALID_STATE`, so retry. Attached machines keep their network and are linked to
  the tenant network their NIC is on.
* **Deletion.** `networks.delete` and `removeSubnet` are `INVALID_STATE` while instances are on them. The reconciler
  deletes rules, then subnets, then the VPC. A provider account with a network that is not deleted cannot be deleted.
* **Shared project quotas (Google Compute Engine).** All tenants' networks on one provider account live in that account's single project, so
  they share its quotas for VPC networks, subnetworks and firewall rules: what one tenant creates, the others on the
  account cannot. vps caps each tenant at 10 networks that are not deleted, and each network at 16 subnets and 100
  firewall rules (`QUOTA_EXCEEDED`, the cap in `details.limit`). Size the project's quotas for the tenants you offer
  the pool to, or give a tenant that needs more its own account (with its own key) and project.

### On DigitalOcean [#on-digitalocean]

DigitalOcean has no network object and no firewall priorities, so the
[DigitalOcean provider](/docs/guides/vps/providers#digitalocean) maps the same calls onto its VPCs, Cloud Firewalls and
tags. The tenant-facing behaviour above holds; these are the differences.

* **Subnet regions.** A DigitalOcean account has no zones, so a subnet's region must be one of the account's `regions`
  (vps reads `regions` as well as the zones of Google Compute Engine accounts). Any other region is `INVALID_INPUT`.
* **Provider-side names.** The provider names things by the provider-side names vps hands it (`bc-` and the lower-cased id of the
  network, subnet or rule), not by the tenant's display names. Below, "the network" and "the subnet" are those names.
* **A VPC per subnet.** DigitalOcean VPCs are regional, so each subnet is a VPC of its own, named by its provider-side
  name (`bc-<id>`), with the subnet's range and the description `bc:network=<network>` (the network's provider-side
  name). The range is an RFC 1918 IPv4 CIDR from /16 to /28 and
  may not overlap another VPC of the account. A VPC of that name that exists with another range, region or network is
  `INVALID_STATE` (`subnetDrift`). Deleting a subnet whose VPC still has members stays `pending` until they are gone.
* **The base firewall.** A network is its base firewall `bc-base-<net16>` (`net16` is 16 hex digits of the sha256 of
  the network's provider-side name). It targets the tag `bc-net-<net16>`, which every instance created on the network carries from its
  first boot, and has no inbound rule and an allow-all outbound rule: the "deny ingress, allow egress" of any other
  provider. The network is `ready` once that firewall exists, not once DigitalOcean has applied it. That a firewall
  on a tag covers a new instance before it serves traffic, and that DigitalOcean honours a rule's `deny`, are not
  verified against a real account yet.
* **A rule is a Cloud Firewall** named `<rule>.n<net16>.p<priority>`. DigitalOcean firewalls carry neither a network
  nor a priority, so the name does. A rule with no targets applies to the network's member tag; the tags a rule
  targets replace that. A firewall is applied in the background: the rule is `pending` until DigitalOcean reports it
  applied, and `error` (applied again on the next run) if it failed.
* **Priorities are refused when they can't be kept.** DigitalOcean applies every firewall that matches an instance, and
  a deny always wins, so a ruleset in which an allow has a lower priority number than an overlapping deny of the same
  network and direction can't be expressed. The provider checks each allow it applies against the denies DigitalOcean
  already holds; a deny is never checked and always applies. So in whichever order vps reconciles the pair, the deny
  is applied and the allow ends in `error` with `UNSUPPORTED` (reason `firewallPriority`) on its own reconcile: the
  ruleset is enforced more strictly than written, never more loosely. An allow that was applied before the deny
  arrived stays applied but is no longer updated; the deny wins at DigitalOcean for the traffic they share, so it
  opens nothing. Read the allow's error message, then fix or remove the inversion. Addresses on the wrong side of a rule
  (destinations on an ingress rule) and an `mtu` other than 1500 are `UNSUPPORTED` too.
* **An attached instance outside the network's VPCs.** DigitalOcean denies what no firewall allows. An attached
  Droplet that a rule targets, but that sits outside the network's VPCs, is covered by that rule's firewall alone,
  which allows only what the rule says, so it loses egress.
* **Only its own names are touched.** The provider replaces and deletes only firewalls whose names have exactly the
  shape above and VPCs whose description names a network. A `bc-r1.prod` firewall or a VPC made by hand is left alone.

### On AWS [#on-aws]

The [AWS provider](/docs/guides/vps/providers#amazon-ec2) maps a network onto an Amazon VPC. Everything it makes is
tagged (`better-compute:network`, `better-compute:subnet`, `better-compute:managed=true`) and found again by tag, so
nothing depends on remembered ids. The tenant-facing behaviour above holds; these are the differences.

* **A network is a VPC** with DNS support and hostnames, an internet gateway, and its own route table (a
  `0.0.0.0/0` route to the gateway) and network ACL, which every subnet better-compute makes in it uses. AWS's main
  route table and default ACL are left as they are. `networks.create` takes an optional `cidr`, the VPC's range: a private /16 to /28 (others are `INVALID_INPUT` at the API), default
  `10.64.0.0/16` (or a block from the account's `ipamPoolId`). Providers whose networks have no range ignore it.
* **Subnets are zone-scoped.** `networks.addSubnet` needs `zone`, one of the account's zones, in `region`
  (`us-east-1a` and the Local Zone `us-west-2-lax-1a` are in their region by AWS's own mapping); a subnet without one is `INVALID_INPUT`, and so is a zone on a region-scoped
  provider. A range outside the VPC's ranges first gets the subnet's /16 (or the range itself, when that /16 would
  overlap one) associated as a secondary range; AWS takes at most five per VPC. A subnet is /16 to /28, checked at the API. Instances go
  on a subnet in their offer's zone: naming one in another zone is `INVALID_INPUT`, and without `subnetId` the
  network's only subnet in that zone is used. Public IPs on launch follow the account's `associatePublicIp`. On an
  adopted VPC a subnet must fall inside the VPC's ranges (`error` otherwise): an adopted VPC is never changed.
* **Allow rules are security group rules.** A rule with no targets goes to the network's group `bc-net-<network>`; a
  rule that targets instances goes to the group named after each instance's tag (`bc-` and 16 hex digits). Every
  instance on the network carries the network's group and its own from its first boot, and the reconciler keeps them
  on its primary interface. Groups that better-compute did not make (by tag, not by name) stay on the interface.
  More groups than the account's per-interface quota (`securityGroupsPerInterface`, default 5) is `QUOTA_EXCEEDED`.
  Two rules that want the same security group rule share it, and the last one removed revokes it. A rule that names
  the other side's range (destinations on an ingress rule, sources on an egress one) is `error`.
* **Deny rules are network-wide.** Security groups only allow, so a deny becomes entries on the network's ACL,
  numbered `100 + priority/10` (taken numbers shift up); the ACL's allow-all is rule 32766, after every deny.
  A deny that targets instances is refused at `firewallRules.create` and `update` with `UNSUPPORTED`. A network-wide
  deny applies whatever the priority of a targeted allow, and ACLs are stateless: a deny on ingress ports can block the
  replies to outbound connections that use them. An ACL holds 20 entries per direction by default, and each deny
  entry's owner is a tag on the ACL (EC2 allows 50 per resource); a deny that does not fit either is `error`. Deny
  rules on an adopted VPC are `error`: its ACL is never edited.
* **Adopting.** `networks.adopt` takes a VPC id (`vpc-…`), or `default` (with `sharedTagScoped`): the account's
  default VPC in the region. An account without a default VPC answers `error` on its subnets and rules.
* **Private service access.** `networks.addSubnet` with `privateAccess: true` (Private Google Access on GCE) adds S3
  and DynamoDB gateway endpoints on the network's route table; they are removed with the network. An adopted network
  refuses `privateAccess` at the API (`INVALID_INPUT`): the endpoints would reroute those services for the whole VPC.
* **Deletion** removes endpoints, subnets, security groups, the gateway and the network's route table and ACL, then
  the VPC; anything still in use
  (`DependencyViolation`) keeps the delete `pending`, and machines still on a subnet keep their groups and gateway
  until they are gone. Only VPCs carrying the network tag are ever deleted. An
  instance's tag group goes once no machine has it and no rule targets it (swept every few minutes, adopted VPCs
  included), and the rest go with the network.
* **IAM.** `better-compute aws iam-policy --phases A1,A2` adds the network permissions: creates must tag what they make
  `better-compute:managed=true`, and changes and deletes need that tag, which nothing may write on an existing
  resource: routes, ACL entries and subnet associations change only on the network's own route table, ACL and
  subnets. A machine's network interface carries `better-compute:instance`, as the instance does; like attach's
  instance tag, the role may write it on any interface it may tag, which `--discover-tag` narrows.

What is not there at all: IPv6 ranges, more than one range or lease per instance, and an API to manage ranges. Tunnels
give you exec, SSH and single-port TCP access without routing to the instance network; see [tunnels](/docs/guides/access/tunnels).


# Volumes and addresses (/docs/guides/vps/provider-volumes)

> Block volumes, shared (NFS) volumes, their snapshots and reserved public addresses on a provider account, as typed resources next to instances, and what happens to them when an instance is deleted.



A tenant on a provider pool can keep data and an address that outlive any one instance: a **provider volume** is a
block volume on the pool's provider account (or a [shared NFS volume](#shared-nfs-volumes) many instances mount),
a **provider snapshot** is a point-in-time copy of one, and a **provider
address** is a reserved public IP. All three are typed resources of the `vps` module, listed and authorized like
instances, and converged on the provider by one reconciler, `vps.provider-resources`.

They are different from the [sandbox and host volumes](/docs/guides/vps/sandboxes) that `bcd` keeps on a hosted
machine's own disks: those live on a node, these live in the cloud account.

| Provider     | Volumes                                                                 | Snapshots                  | Addresses                                   |
| ------------ | ----------------------------------------------------------------------- | -------------------------- | ------------------------------------------- |
| DigitalOcean | Block Storage (`kind: block`) and Network File Storage (`kind: shared`) | Volume and share snapshots | Reserved IPv4 and IPv6                      |
| Others       | When the provider implements `ProviderImpl.volumes`                     | With volumes               | When it implements `ProviderImpl.addresses` |

A pool whose provider implements neither answers `UNSUPPORTED` to every create.

## How a request converges [#how-a-request-converges]

Every mutation records what the tenant wants (`desiredState`, the wanted attachment or assignment) in its own
transaction and returns at once, usually with `state: 'pending'`. The reconciler then calls the provider outside any
transaction and writes back what it saw:

* States are `pending`, `ready`, `updating` (a resize running), `deleting`, `deleted` and `error`. An `error` carries
  the provider's `message` and is retried by the next change or the periodic pass (every 5 minutes). A settled
  resource is read once an hour for drift, those reads spread across the hour so they leave the token's request
  budget to instance polling.
* Provider actions (attach, detach, resize, snapshot) never block a run. A running action is looked at again 5 seconds
  later; a rate-limited or unavailable provider 30 seconds later.
* A create sends the provider a name derived from the resource id, so a create whose answer was lost finds what it
  made instead of making a second one. A reserved IP has no name on DigitalOcean: a reserve whose answer was lost puts
  the address in `error` and is never sent again (see [lost reserves](#lost-reserves)).

Each create takes an `idempotencyKey`. The same key with the same body returns the same resource; with a different
body it is `CONFLICT`.

## Volumes [#volumes]

```ts title="volume.ts"
const volume = await compute.api.providerVolumes.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'pg-data',
  region: 'nyc3',
  sizeGiB: 100,
  filesystem: 'ext4',
  instanceId: 'inst_01jz…', // attach once both are ready
  idempotencyKey: 'pg-data-1',
});
// → { id: 'pvol_…', state: 'pending', attachment: { instanceId, state: 'pending', desiredState: 'present' }, … }
```

<TypeTable
  type="{
  name: { type: 'string', description: '1-42 lower-case letters, digits or -, starting with a letter. Unique per tenant among volumes that are not deleted.', required: true },
  region: { type: 'string', description: &#x22;One of the provider account's regions, such as nyc3.&#x22;, required: true },
  sizeGiB: { type: 'number', description: 'From 1 to 16384 GiB.', required: true },
  filesystem: { type: &#x22;'ext4' | 'xfs' | 'none'&#x22;, description: 'Formats the volume when it is created. Unset or none: a raw block device.' },
  instanceId: { type: 'string', description: 'An instance to attach it to, of the same tenant, provider account and region.' },
  deleteWithInstance: { type: 'boolean', description: 'Delete the volume when the instance it is attached to is deleted; otherwise deleting the instance only detaches it.', default: 'false' },
  description: { type: 'string', description: 'Free text kept with the volume.' },
  projectId: { type: 'string', description: 'The project the volume belongs to, for listing.' },
}"
/>

Once it is `ready`, `device` is the path the guest sees it at. On DigitalOcean that is
`/dev/disk/by-id/scsi-0DO_Volume_<name>`, where `<name>` is the provider name derived from the volume id.

better-compute does not mount it yet: the volume arrives in the guest without a reboot, as a block device (formatted
when you asked for a `filesystem`), and the guest mounts it, for example with an `/etc/fstab` line on that path with
`nofail`. Mounting by bcd, and a cloud-init mount for volumes attached at create time, are planned.

The other methods of `providerVolumes`:

| Method        | What it does                                                                                                                                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `get`, `list` | One volume, or the tenant's (filter by `poolId` or `projectId`; `includeDeleted` adds deleted ones).                                                                                       |
| `attach`      | Attaches it to an instance. A block volume is attached to one instance at a time: a second instance is `CONFLICT`, never a silent move. A [shared volume](#shared-nfs-volumes) takes many. |
| `detach`      | Detaches it (from `instanceId`, required for a shared volume). The volume stays.                                                                                                           |
| `resize`      | Grows it to `sizeGiB`. A smaller size is `INVALID_INPUT`: volumes only grow.                                                                                                               |
| `snapshot`    | Takes a snapshot (`name`, `idempotencyKey`). At most 25 snapshots that are not deleted per volume.                                                                                         |
| `delete`      | Deletes it at the provider. `INVALID_STATE` while it is attached: detach it first.                                                                                                         |

Attach refuses before any provider call: an instance of another tenant is `ACCESS_DENIED`; one on another provider
account or in another region is `INVALID_INPUT`; an instance being deleted is `INVALID_STATE`.

## Shared (NFS) volumes [#shared-nfs-volumes]

A volume with `kind: 'shared'` is a network file share: one filesystem that many instances mount at once over NFS.
On DigitalOcean it is a Network File Storage share. A share has no access control of its own beyond the network it is
reachable from, so a shared volume always names a [tenant network](/docs/guides/vps/networking) and is reachable only
from that network's instances.

```ts title="shared.ts"
const shared = await compute.api.providerVolumes.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'assets',
  region: 'nyc3',
  sizeGiB: 50,
  kind: 'shared',
  networkId: 'net_01jz…', // required: the share lives in this network's subnet in nyc3
  idempotencyKey: 'assets-1',
});
await compute.api.providerVolumes.attach(token, { tenantId: shared.tenantId, volumeId: shared.id, instanceId: 'inst_a…' });
await compute.api.providerVolumes.attach(token, { tenantId: shared.tenantId, volumeId: shared.id, instanceId: 'inst_b…' });
// get → { kind: 'shared', networkId, attachments: [{ instanceId: 'inst_a…', state: 'ready', … }, …],
//         device: '10.110.0.5:/2633050/…', mount: { source: '10.110.0.5:/2633050/…', type: 'nfs4' }, … }
```

How it differs from a block volume:

* **The network.** `networkId` must be a network of the tenant, on the pool's provider account, provisioned, with a
  subnet in the volume's region (`INVALID_INPUT` otherwise; `INVALID_STATE` while the network or that subnet is not
  ready yet, or the network is being deleted). The share is made in that subnet, never in the region's default VPC.
  `networkId` on a block volume is `INVALID_INPUT`. A network a shared volume is on is not deleted
  (`networks.delete` answers `INVALID_STATE` until the volume is deleted), and neither is a subnet of it in the
  volume's region (`networks.removeSubnet` answers `INVALID_STATE`, with the volumes in `details.volumeIds`).
  A provider that makes no shared volumes is `UNSUPPORTED` (`reason: 'sharedVolumes'`).
* **Many instances.** `attach` adds the instance (attaching one it is on returns the volume); the instance must be on
  the volume's network (`INVALID_INPUT` otherwise). The view lists every attachment in `attachments` (a block volume
  keeps the single `attachment`). `detach` names the instance, `{ volumeId, instanceId }` (`INVALID_INPUT` without
  it). `delete` is `INVALID_STATE` while any instance still has it.
* **No filesystem choice.** A share is a filesystem already: leave `filesystem` out (`none` is accepted and
  ignored). `deleteWithInstance` is always `false`: deleting an instance detaches it from the share and keeps the
  share.
* **Size.** It grows like a block volume and books `diskGiB` quota the same way. DigitalOcean shares are 50 to 16384
  GiB: a smaller size is `INVALID_INPUT` (`reason: 'size'`) at create, before any quota is booked.
* **Billing.** A shared volume bills `vps.share.gib.hours` (its GiB), not `vps.volume.gib.hours`, so a rate card
  prices shares apart from block storage.
* **Snapshots.** `snapshot` works as for a block volume; `providerSnapshots.restore` of a shared volume's snapshot is
  `UNSUPPORTED` (`reason: 'sharedRestore'`) for now.

On DigitalOcean, attaching tags the Droplet `bc-nfs-<16 hex>` (the record of which Droplets have the share) and adds
the Droplet's VPC to the share when it is not there yet; that VPC must be one of the same network. Detaching removes
the tag and takes the VPC off the share once no tagged Droplet is left in it; the share's last VPC stays. When an
instance on the share is deleted, the detach that follows finds its Droplet gone and takes off every VPC no tagged
Droplet is left in; deleting the share removes its tag. The hourly drift pass asks every ready
attachment of a settled shared volume again (an attach is idempotent), which puts back a VPC a racing detach took off.

### Mounting a shared volume [#mounting-a-shared-volume]

`device` (and `mount.source`) is the NFS source, `<host>:<path>`. better-compute does not mount it in the guest
yet; `sharedVolumeMount` from `@better-compute/vps` makes the lines for cloud-init or `/etc/fstab`:

```ts title="mount.ts"
import { sharedVolumeMount } from '@better-compute/vps';

const mount = sharedVolumeMount(shared.mount!.source, '/mnt/assets'); // { readOnly: true } mounts it ro
mount.cloudConfig; // ['10.110.0.5:/2633050/…', '/mnt/assets', 'nfs4', 'defaults,_netdev,nofail', '0', '0']
mount.fstab; //       '10.110.0.5:/2633050/… /mnt/assets nfs4 defaults,_netdev,nofail 0 0'
```

Put `cloudConfig` in the `mounts:` list of an instance's cloud-config user data (the image needs an NFS client, such
as `nfs-common` on Ubuntu). `_netdev` waits for the network and `nofail` lets the guest boot while the share is
unreachable. The helper refuses a source that is not `<host>:<absolute path>`, and a mount path that is relative, has
`.` or `..` segments or would cover part of the system (`/`, `/etc`, `/usr`, …), as for a sandbox volume's mount.
Mounting by bcd on pool members is planned, as for block volumes.

## Snapshots [#snapshots]

A snapshot outlives its volume. `providerSnapshots` has `get`, `list` (filter by `volumeId`), `delete` and `restore`:

```ts title="restore.ts"
const copy = await compute.api.providerSnapshots.restore(token, {
  tenantId: 'ten_01jz…',
  snapshotId: 'psnap_01jz…',
  name: 'pg-data-copy',
  idempotencyKey: 'restore-1',
  // optional: sizeGiB (not below the snapshot's), description, deleteWithInstance
});
// → a new provider volume, created from the snapshot
```

A restore waits until the snapshot is `ready`, and fails if the snapshot is deleted first. The restored volume keeps the
filesystem of the volume the snapshot was taken of: a restore takes no `filesystem` (`INVALID_INPUT`), since asking the
provider to format it could wipe the restored data.

## Addresses [#addresses]

```ts title="address.ts"
const ip = await compute.api.providerAddresses.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'ingress',
  region: 'nyc3',
  family: 'ipv4', // or 'ipv6'
  idempotencyKey: 'ingress-1',
});
await compute.api.providerAddresses.assign(token, { tenantId: ip.tenantId, addressId: ip.id, instanceId: 'inst_01jz…' });
// get → { address: '203.0.113.7', assignmentState: 'ready', instanceId: 'inst_01jz…', … }
```

`assignmentState` is `none`, `pending`, `ready`, `deleting` (an unassign running) or `error` (with
`assignmentMessage`). An address is on one instance at a time: assigning it to a second instance is `CONFLICT`, and to
an instance of another tenant `ACCESS_DENIED`. `unassign` releases it and keeps it reserved. `delete` is `INVALID_STATE`
while it is assigned.

The reconciler never takes an address over from a machine it did not put it on: if the provider reports it on another
machine, the assignment goes to `error` and nothing is moved.

Since a reserved IP carries no tag, the row is the ownership record, and one address is recorded by at most one live
row. Where the provider has one, the row also keeps the `fingerprint` from the reserve (on DigitalOcean a reserved
IPv6's reservation time). A reserved IPv4 has none: its project is shared by every reservation in the project, so it
cannot tell a re-reservation of the same IP apart.

An address released at the provider outside better-compute is lost for good. The reconciler notices it on its next
read (the IP is gone, or another reservation's fingerprint or region shows there), or when a later reserve of the same
account is handed that IP. The row then goes to `error` saying so, forgets the provider address and its instance, and
never calls the provider again: it is never re-adopted when the IP is handed out again, never assigned, and never
released. `assign` is `INVALID_STATE`; `delete` finishes the row (refunding its quota) without a provider call. The
provider refuses a mismatched address too (`INVALID_STATE`, reason `notManaged`).

A volume deleted at the provider outside better-compute is not made again either: once a volume was created, the
reconciler only reads it, and one the provider no longer has goes to `error` (its data is gone). Delete it.

### Lost reserves [#lost-reserves]

DigitalOcean's reserved IPs carry no name or tag, so a reserve whose answer never arrived (a timeout or a dropped
connection) cannot be found again. Rather than risk a second address, the address goes to `error` with a message
saying so, and is never reserved again. Delete it, and check the account's reserved IPs for one that is not assigned. A
reserve that was refused or rate-limited reserved nothing, and is retried.

## Deleting an instance [#deleting-an-instance]

Deleting an instance takes care of its volumes and addresses:

1. While the machine is being deleted, a volume with `deleteWithInstance` makes no provider call (an ensure by name
   could otherwise make again a volume the delete already took).
2. The machine delete names the instance's `deleteWithInstance` volumes. On DigitalOcean that is one
   `DELETE /v2/droplets/{id}/destroy_with_associated_resources/selective` naming only volumes that carry this
   deployment's managed tags and are attached to that Droplet. A tenant's own volume attached by hand is never named
   and survives. With nothing to name it is a plain `DELETE /v2/droplets/{id}`. Tag-scoped bulk deletes are never used.
3. When the delete finishes, the instance's attachments are no longer wanted, its `deleteWithInstance` volumes move to
   `deleting`, and every volume and address it had is woken. The reconciler settles the detach, deletes what is to go
   (already gone after a selective destroy; a provider without one deletes it after the machine) and unassigns its
   addresses, which stay reserved.

Once the instance's delete is asked, `detach` of a `deleteWithInstance` volume is `INVALID_STATE`: it goes with the
machine. Detach it before deleting the instance to keep it; a volume detached before the delete is never deleted with
it.

An instance that finishes as a **detach** (an attached machine with `onDelete: 'detach'`, or one whose ownership label
is gone) leaves its machine running. Its volumes are only detached, `deleteWithInstance` or not.

## Quota, billing and events [#quota-billing-and-events]

* A volume books its `sizeGiB` as `diskGiB` quota on (tenant, pool), the resource a provider instance's disk books;
  growing it books the difference. An address books one `addresses`. A tenant without a quota row for the resource
  gets nothing (`QUOTA_EXCEEDED`, fail closed). The booking is refunded once the provider has deleted the resource.
  Snapshots book no quota.
* On a metered pool, each resource bills from the moment it was `ready` until it is deleted, in whole-hour usage
  windows written by `vps.ancillary-usage`: a snapshot `vps.snapshot.gib.hours` (its GiB), a block volume
  `vps.volume.gib.hours` (its GiB), a shared volume `vps.share.gib.hours` (its GiB) and an address `vps.address.hours` (one per hour). One that never became ready never
  bills; a BYO account's pool (platform fee) bills none of them. Volumes, snapshots and addresses are all metered by
  the `vps.ancillary-usage` pass (usage source kinds `volume`, `snapshot` and `address`).
* Events: `providerVolume.*` (created, updated, ready, attached, detached, resized, error, deleting, deleted),
  `providerVolume.snapshot.*` and `providerAddress.*` (created, updated, ready, assigned, unassigned, error, deleting,
  deleted). See the [event catalog](/docs/reference/events).

## Permissions [#permissions]

| Group               | Read             | Everything else    | Resource                 |
| ------------------- | ---------------- | ------------------ | ------------------------ |
| `providerVolumes`   | `volumes:read`   | `volumes:manage`   | `provider-volume/{id}`   |
| `providerSnapshots` | `volumes:read`   | `volumes:manage`   | `provider-snapshot/{id}` |
| `providerAddresses` | `addresses:read` | `addresses:manage` | `provider-address/{id}`  |

`list` is decided on the tenant. A create is decided on the new resource, a restore on the snapshot and a snapshot on
its volume.

`providerVolumes.attach`, `providerVolumes.create` with an `instanceId` and `providerAddresses.assign` put the volume
or address on an instance, so they are also decided on that instance: the caller needs `instances:update` on
`instance/{instanceId}` as well, with its own credential. A role that manages volumes in one project cannot attach one
to an instance it cannot change.


# Providers (/docs/guides/vps/providers)

> Provider accounts, the defineProvider contract, the Hetzner Cloud, Proxmox VE, Google Compute Engine, DigitalOcean and Amazon EC2 providers, discovery and attach, provider reservations, platform-issued compute and its tenant isolation, billing (modes, attach billing, reservation grants, provider costs and pass-through), conformance cases, and secret references for credentials.



A provider pool runs its instances on someone else's infrastructure: a public cloud, or a hypervisor cluster you
operate but do not want to run bcd on. Better Compute splits this the way it splits every extension point: **code
registers kinds, data configures instances**.

* A provider is code: a `defineProvider()` definition such as `hetzner-cloud`, passed to
  `vps({ providers })`.
* A provider account is data: a row naming a provider, with a config validated by
  that provider's manifest. Its credentials are secret references, never values.
* A pool of class `provider` points at one account. Its instances are created, polled, powered and deleted through the
  account's client by the `vps.instances` reconciler (see [instances](/docs/guides/vps/instances)).

```ts title="compute.ts"
import { vps } from 'better-compute/vps';
import { hetznerCloud } from '@better-compute/provider-hetzner';
import { proxmoxProvider } from '@better-compute/provider-proxmox';
import { gcp } from '@better-compute/provider-gcp';
import { digitalocean } from '@better-compute/provider-digitalocean';

vps({ providers: [hetznerCloud, proxmoxProvider({ writeSnippet }), gcp, digitalocean] });
```

The provider packages are on npm like the rest of Better Compute, but sit outside the `better-compute` umbrella:
install them by their own names (`npm install @better-compute/provider-hetzner @better-compute/provider-proxmox @better-compute/provider-gcp @better-compute/provider-digitalocean`).

## Provider accounts [#provider-accounts]

### Credentials first [#credentials-first]

An account config never holds a credential. Fields declared with `field.secret()` accept only a `SecretRef`:
`secret://…`, `vault://…` or `env://…`. On top of the manifest, vps refuses any string field whose name ends in
`token`, `secret`, `password` or `key` that is not a reference, at any depth, with `INVALID_INPUT`.

Store the value first. [secrets.put](/docs/reference/api/secrets#put) (`compute:secrets:manage`) writes it to the
configured secret store and answers with its reference; nothing ever reads it back through the API:

```ts
const { ref } = await compute.api.secrets.put(
  { token },
  {
    tenantId: 'root',
    name: 'hcloud-token',
    value: process.env.HCLOUD_TOKEN!,
    idempotencyKey: 'hcloud-token-1',
  },
);
// ref: 'secret://root/sec_…'
```

References are resolved when a client needs them, through `vps({ secrets })` if you passed a resolver, otherwise the
server's secret broker, which routes each reference to the store serving its scheme. The default store, pki's
local-sealed store, serves `secret://` only. An `env://` or `vault://` reference resolves only if you configure a
`SecretStore` for that scheme in `betterCompute({ secrets })` (the list replaces the default) or pass your own
resolver to `vps()`.

### Whose references an account may name [#whose-references-an-account-may-name]

An account's references are scoped to its tenant, for every provider. An account of the root (platform) tenant may
name any `secret://`, `vault://` or `env://` reference. An account of any other tenant may name only
`secret://<its tenant>/…`, the secrets that tenant put with `secrets.put`, and the `vault://` paths you allow it:

```ts
vps({
  providers: [hetznerCloud, gcp],
  // An entry ending in / allows every path under it; any other entry allows that one path.
  tenantVaultPaths: { org_acme: ['kv/tenants/acme/'] },
});
```

`env://` stays the platform's: it reads the control plane's own environment. A tenant's account naming `env://`,
a `vault://` path it is not allowed, the platform's secrets or another tenant's would run on credentials that are
not the tenant's, so `providerAccounts.create` and `update` refuse it with `INVALID_CONFIG` (`details.path` names the
field, `details.reason` is `secretRefScope`). The rule holds again when a client resolves the account's references:
an account stored before it, or written around the API, fails that call with `ACCESS_DENIED` and nothing is read.
A tenant's references reach the store with the tenant as the resolve context (`resolve(ref, { tenantId })`): the
local-sealed store then serves only a secret stored for that tenant, and a `SecretStore` of your own for `vault://`
gets the same context to check against. The root tenant is the deployment's `rootTenantId`, given to
`betterCompute()` or named by the IAM port (`bridge.embedded(iam, { rootTenantId })`): modules see it either way.

### Create an account [#create-an-account]

```ts
const account = await compute.api.providerAccounts.create(
  { token },
  {
    tenantId: 'root',
    provider: 'hetzner-cloud',
    name: 'main',
    config: { token: ref, location: 'fsn1' },
  },
);
// → { id: 'pacct_…', tenantId: 'root', provider: 'hetzner-cloud', name: 'main',
//     config: { token: 'secret://root/sec_…', location: 'fsn1' }, version: 1 }
```

<TypeTable
  type="{
  tenantId: {
    type: 'string',
    description: 'The tenant that owns the account, usually the root (platform) tenant.',
    required: true,
  },
  provider: {
    type: 'string',
    description:
      'The id of a provider passed to vps({ providers }). An unknown id is INVALID_INPUT, with details.known listing the registered ones.',
    required: true,
  },
  name: {
    type: 'string',
    description:
      '1 to 63 lower-case letters, digits, dots, underscores or hyphens. Unique in the tenant (CONFLICT otherwise).',
    required: true,
  },
  config: {
    type: 'object',
    description:
      &#x22;Validated by the provider's manifest; defaults are filled in. Issues come back as INVALID_INPUT with details.issues [{ message, path }].&#x22;,
    default: '{}',
  },
  idempotencyKey: {
    type: 'string',
    description:
      'Makes the account id deterministic from the tenant, the caller and the key, as FleetConfig needs. A retry with the same provider and name returns the account; anything else under the key is CONFLICT.',
  },
}"
/>

The rest of the group:

| Method                                                                       | Action                     | What it does                                                                                                                                      |
| ---------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| [list](/docs/reference/api/provider-accounts#list)                           | `compute:providers:read`   | The tenant's accounts, by name.                                                                                                                   |
| [federationSetup](/docs/reference/api/provider-accounts#federationsetup)     | `compute:providers:read`   | The issuer and the exact subject to trust at the cloud for an account you are about to create; see [keyless GCP accounts](#keyless-gcp-accounts). |
| [get](/docs/reference/api/provider-accounts#get)                             | `compute:providers:read`   | One account; another tenant's is `NOT_FOUND`.                                                                                                     |
| [offers](/docs/reference/api/provider-accounts#offers)                       | `compute:providers:read`   | The provider's catalog through this account.                                                                                                      |
| [update](/docs/reference/api/provider-accounts#update)                       | `compute:providers:manage` | Replaces `config` (validated again) and bumps `version`; the next call builds a fresh client.                                                     |
| [delete](/docs/reference/api/provider-accounts#delete)                       | `compute:providers:manage` | Deletes the account, or `INVALID_STATE` while a non-archived pool or a non-deleted instance uses it.                                              |
| [bootstrap](/docs/reference/api/provider-accounts#bootstrap)                 | `compute:providers:manage` | Turns one of the account's machines into a bcd host of a pool; see [bootstrapping machines](/docs/extending/providers#bootstrapping-machines).    |
| [machines](/docs/reference/api/provider-accounts#machines)                   | `compute:providers:read`   | The machines discovery last saw in the account, free, attached or all; see [discovery and attach](#discovery-and-attach).                         |
| [discover](/docs/reference/api/provider-accounts#discover)                   | `compute:providers:manage` | Runs the account's next discovery pass now; `wait: true` holds the answer for at most 30 s.                                                       |
| [reservations](/docs/reference/api/provider-accounts#reservations)           | `compute:providers:read`   | The provider's capacity reservations with `free` and `pending`; see [provider reservations](#provider-reservations).                              |
| [bindReservation](/docs/reference/api/provider-accounts#bindreservation)     | `compute:providers:manage` | Makes a `ready` reservation a flavor of a provider pool.                                                                                          |
| [unbindReservation](/docs/reference/api/provider-accounts#unbindreservation) | `compute:providers:manage` | Removes the binding; the flavor is deleted, or archived while instances use it.                                                                   |

Clients are built on demand from the validated config and cached per account id and version. A failed build is not
cached.

`offers` takes an optional `query`:

<TypeTable
  type="{
  kind: { type: &#x22;'metal' | 'vps'&#x22;, description: 'Only offers of this kind.' },
  region: {
    type: 'string',
    description: &#x22;A provider region, such as fsn1. Providers default to the account's own.&#x22;,
  },
  gpuModel: { type: 'string', description: 'Only offers with this GPU model.' },
  minGpus: { type: 'number', description: '0 to 64.' },
  minVcpu: { type: 'number', description: '0 to 1024.' },
  minMemoryGiB: { type: 'number', description: '0 to 16384.' },
}"
/>

Each offer is `{ id, kind, region, vcpu, memoryGiB, diskGiB?, gpus?, pricePerHour?, currency?, available }`. Its `id`
is what a provider pool's [flavor](/docs/guides/vps/flavors) names as `offerId`.

### A pool on the account [#a-pool-on-the-account]

```ts
const pool = await compute.api.pools.create(
  { token },
  { tenantId: 'root', siteId, class: 'provider', providerAccountId: account.id },
);
// agent is 'none': a provider pool's machines are reached through the provider
```

A provider pool needs `providerAccountId`, and its `agent` must be `none`. Only provider pools take an account.

## Discovery and attach [#discovery-and-attach]

A provider that can list every machine of an account (`discover`: Google Compute Engine and DigitalOcean today) lets you hand a VM that
already runs to a customer tenant without recreating it.

* **The `vps.discovery` worker** lists each such account every `vps({ discovery: { every } })` (default `5m`) into
  `provider_machines`, and publishes `provider.machine.discovered`, `changed` and `gone`. It never changes a machine.
  [machines](/docs/reference/api/provider-accounts#machines) reads what it saw, and
  [discover](/docs/reference/api/provider-accounts#discover) runs a pass now (`wait: true` holds the answer for at
  most 30 s).
* **[instances.attach](/docs/reference/api/instances#attach)** turns a free machine into an instance of a tenant's
  provider pool backed by the same account. The instance is owned by `ownerIdentityId`, takes the machine's own shape
  as its flavor, and is charged to the tenant's quota. Only the platform attaches from accounts its own tenant does
  not own. The reconciler then writes the ownership label (`claim`) and adopts the machine as it is, running or
  stopped.
* **The tenant controls it** like any instance: `get`, `list`, `start`, `stop`, `reboot` and `delete`.
  `onDelete: 'detach'` (the default) removes the label and leaves the machine running; `'delete'` deletes it.
  [instances.detach](/docs/reference/api/instances#detach) always detaches.
* **Ownership is re-checked on every read.** A machine whose label is gone or names another instance fails its
  instance with `OwnershipLost` (`AttachConflict` when the first claim was refused). vps never powers, deletes or
  relabels it again, and deleting the instance finishes as a detach without a provider call. An instance whose
  machine id a live instance already holds fails with `DuplicateMachine` instead of retrying.
* **Networks.** An attached machine keeps its network. When its NIC is on one of the tenant's
  [networks](/docs/guides/vps/networking#tenant-networks) on the same account, the instance is linked to it; a
  network being deleted refuses the attach with `INVALID_STATE`.
* **Auto-attach** is opt-in (owners: see below). With `vps({ discovery: { autoAttach: { ownerIdentityId } } })`, a free machine labelled
  `better-compute-tenant=<tenant>` and `better-compute-pool=<pool>` is attached to that pool after each pass
  (`onDelete: 'detach'`), owned by the identity in its `better-compute-owner` label or else `ownerIdentityId`.

## Provider reservations [#provider-reservations]

A provider that can list capacity reservations (`listReservations`, Google Compute Engine today) has them detected and
offered as capacity that tenants create instances on.

* **The `vps.reservations` worker** asks each such account every `vps({ reservations: { every } })` (default `5m`)
  and publishes `provider.reservation.detected`, `changed`, `ready` and `gone`.
  [reservations](/docs/reference/api/provider-accounts#reservations) lists them with `free` (`total - inUse - pending`)
  and `pending` (creates admitted that the last scan may not have counted). `free` can be one pass behind the
  provider.
* **[bindReservation](/docs/reference/api/provider-accounts#bindreservation)** makes a `ready` reservation a flavor
  (default name `rsv-<name>`) of a provider pool of the account. A `scheduled` (future) reservation is listed and
  evented, and `provider.reservation.ready` fires when it is fulfilled, but it can be bound only once it is ready,
  because the provider fulfils it under a new id.
* **Admission.** `instances.create` on a bound flavor takes a free slot under a lock on the reservation, so concurrent
  creates never over-commit it; with none free it is `CAPACITY_UNAVAILABLE` with the counts in `details`. The provider
  stays the final referee: a create it refuses fails the instance with `ProvisionFailed`, and the slot is free again.
* **[unbindReservation](/docs/reference/api/provider-accounts#unbindreservation)** removes the binding. The flavor is
  deleted, or archived while instances use it; the worker does the same when a bound reservation goes away.

## Platform-issued compute [#platform-issued-compute]

The root (platform) tenant adds compute once and issues it to customer tenants, which then use and control what they
were given. The account, its credentials, the pool and its quotas stay the platform's.

1. **A root account.** The root tenant creates the provider account (`providerAccounts.create` in its own
   tenant). A root account may use the host's own credentials (ruling F1); a tenant's own account must name its key.
2. **A root pool, offered.** A `class: 'provider'` pool on that account (`pools.create`), flavors on it
   (`flavors.create`, or `providerAccounts.bindReservation` for a reservation), offered to each customer with a
   quota (`pools.offer`, `quotas.set`).
3. **Issue.** Attach a VM that already runs (`instances.attach` with the customer's `tenantId` and an
   `ownerIdentityId` of theirs; discovery lists the account's VMs), or create a new one (`instances.create` with
   the customer's `tenantId` and `ownerIdentityId`). The instance is the customer's: it shows in the owner's
   default `scope: 'mine'` list, owner-conditioned policies (`resource.ownerId`) apply to them, and it is
   charged to the customer's quota.
4. **The customer uses it.** With grants in its own tenant the customer lists, reads, starts, stops, reboots and
   deletes it (an attached VM's `onDelete: 'detach'` hands it back running), creates instances of its own on the
   offered pool (a reservation-backed flavor included), and builds
   [networks](/docs/guides/vps/networking#tenant-networks), subnets and firewall rules on it.

| Who                                                                           | Can                                                                                                                                                           | Cannot                                                                                                                                                                                                                                                                                                                                                                                             |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Root (root realm, root administrator, user session with MFA: `ROOT_OVERRIDE`) | Everything above, in any tenant, decided in the tenant that holds the resource                                                                                | Bypass state rules (an account still in use is `INVALID_STATE`)                                                                                                                                                                                                                                                                                                                                    |
| A customer tenant, with grants in its own tenant                              | `instances:*` on its instances, `networks:*` on its networks, its own provider accounts (with their own key), pools on those, attach and reservations on them | Read or change a root account (`get`, `offers`, `machines`, `discover`, `reservations`, `bindReservation`, `unbindReservation`, `bootstrap`, `update`, `delete`); build a pool on an account that is not its own; attach from a root account; change the root pool, its offers or flavors; set quotas; make anyone but itself the owner of an instance (`ownerIdentityId` on `create` or `attach`) |
| Another customer tenant                                                       | The same, for its own resources                                                                                                                               | See or act on the first tenant's instances, networks and rules; target them in its rules; put instances on its networks                                                                                                                                                                                                                                                                            |

IAM is the first gate: a tenant's grants are decided in its own tenant, on resources compute projects there. Compute
then holds these rules itself, so a tenant holding `compute:*` in its own tenant still cannot use them to reach
root-level resources or another tenant's:

* Naming another tenant's `tenantId`, the root tenant's included, is `ACCESS_DENIED` by the envelope before any IAM
  decision or database read. `quotas.set` is decided on the fleet, in the root tenant.
* Handlers read by `(id, tenantId)`: a root account, the management of the root pool (`pools.update`, `offer`,
  `unoffer`, `archive`, flavors) and another tenant's instances or networks are `NOT_FOUND` in the caller's own
  tenant.
* A provider pool runs only on an account of its own (owner) tenant: `pools.create` refuses another tenant's
  account as `NOT_FOUND`, worded as for an unknown one, and vps uses a pool's account only when the pool's owner
  owns it (instance creates and the provider reconciler, flavors, networks, reservation bindings). A pool offered to
  other tenants keeps working for them; they are not shown its `providerAccountId`.
* `instances.attach` from an account the caller's tenant does not own is the platform's only (`ACCESS_DENIED`).
  Only a root caller acting in another tenant makes anyone but itself the owner (`ownerIdentityId` on `create` and
  `attach`); when the IAM port can say (`IamPort.isMember`: the embedded better-iam bridge), the owner must be an
  active identity of that tenant (`INVALID_INPUT` otherwise). Ports without it check the id's shape only.
* A firewall rule's targets and an instance's `networkId` must be the tenant's own. Another tenant's are
  `INVALID_INPUT`, worded as for ones that do not exist.
* `providerAccounts.machines` on a tenant's own account does not name the instance or tenant holding a machine
  when that is another tenant (its pool was offered to it); the platform sees both.
* Label-driven auto-attach takes the `better-compute-owner` label only on a platform account, and only for an active
  member of the labelled tenant when the IAM port can say (else `autoAttachError` `AUTO_ATTACH_OWNER_REFUSED`, not
  attached). On a tenant's own account the label is the tenant's to write, so it is ignored: the configured
  `ownerIdentityId` gets the instance and `AUTO_ATTACH_OWNER_IGNORED` is recorded (with a
  `provider.machine.changed` event).
* A tenant's account names only its own secret references and never the host's ambient cloud credentials (see
  [Whose references an account may name](#whose-references-an-account-may-name) and the
  [Google Compute Engine](#google-compute-engine) section).

Not covered here: what a tenant may do in its own GCP project through its own key, and the platform's own operators,
who act with `ROOT_OVERRIDE` everywhere.

The IAM actions split the same way. Root-only by policy: `compute:providers:*` on root accounts,
`compute:instances:attach` from root accounts, `compute:quotas:manage`, and `compute:pools:share` and
`compute:flavors:manage` on root pools. Tenant-grantable: `compute:instances:*` (`attach` only from the tenant's
own accounts), `compute:networks:*` (on `compute-network`, a child of `compute-tenant` with `providerAccountId`,
`poolId` and `projectId` attributes) and `compute:providers:*` on the tenant's own accounts. An operator role
that controls what it was issued:

```json title="compute-operator (a tenant policy)"
{
  "version": 1,
  "statements": [
    {
      "effect": "allow",
      "actions": ["compute:instances:read", "compute:pools:read", "compute:networks:read"],
      "resources": ["compute-tenant/*"]
    },
    {
      "effect": "allow",
      "actions": ["compute:pools:read", "compute:flavors:read"],
      "resources": ["compute-pool/*"]
    },
    {
      "effect": "allow",
      "actions": ["compute:instances:create"],
      "resources": ["compute-instance/*"]
    },
    {
      "effect": "allow",
      "actions": ["compute:instances:read", "compute:instances:update", "compute:instances:delete"],
      "resources": ["compute-instance/*"],
      "conditions": { "StringEquals": { "resource.ownerId": "${principal.id}" } }
    },
    {
      "effect": "allow",
      "actions": [
        "compute:networks:create",
        "compute:networks:read",
        "compute:networks:update",
        "compute:networks:delete"
      ],
      "resources": ["compute-network/*"]
    }
  ]
}
```

## Billing [#billing]

The root tenant bills its customers for what it issues to them on its provider accounts: compute meters the VMs,
reservation slots, networks, disks and addresses, and Better IAM prices and invoices them. The whole model, prices
and budgets are in [Billing and usage](/docs/guides/iam/billing-and-usage); this is what a provider pool adds.

* **Billing modes.** A provider pool's `policy.billing.mode` is `metered` (vCPU, memory, GPU and per-VM lines, plus
  disks and public addresses), `platform-fee` (one `vm.managed.hours` line per running VM) or `none`. Root-owned
  pools default to `metered`; a pool on a tenant's own cloud account defaults to `platform-fee`, so a tenant is never
  charged compute it already pays its provider for. Only the platform sets the mode
  ([`pools.create`](/docs/reference/api/pools#create), `pools.update`).
* **What a VM records.** Usage comes from the instance's observed `running` and `stopped` spans, written with the
  lifecycle change: billing starts at the first observed `running` and ends at the observed stop, detach or delete,
  and a `lost` span is not billed. A stopped VM still bills its disks. A flavor with a root-set `billing.meter`
  (`flavor.<name>.hours`) bills one whole-shape line instead of the composed ones.
* **Attach.** [`instances.attach`](/docs/reference/api/instances#attach) takes `billing: 'metered' | 'none'`. Only
  the platform may write a VM off with `none`; a lent VM bills its tenant from its first observed `running` after
  the attach, never before.
* **Reservations.** A VM admitted against a bound reservation bills the `vm.reserved.*` meters. Root can commit
  slots of a reservation to one tenant for a term with
  [`reservationGrants.create`](/docs/reference/api/reservation-grants#create): the tenant pays
  `vm.grant.slot.hours` for every committed slot, used or not, gets priority on them, and its VMs inside the grant
  record no compute meters. While a grant is live, `unbindReservation` and `delete` on the account are
  `INVALID_STATE`.
* **Provider costs.** The `vps.costs` worker reads what the provider billed (Compute Engine: the BigQuery billing
  export named by `billingExport`) and attributes each row to a tenant by the `better-compute-instance` and
  `better-compute-tenant` labels. `billing.passThrough` rules on a root account bill costs compute does not meter
  itself, such as egress, at cost times a multiplier on a `reported` meter; the SKUs compute meters are refused as
  patterns and never passed through. Root reads its margin per tenant, pool, account or meter with
  `billing.margins`, and `billing.leak` flags a VM the provider billed for more hours than compute metered.
* **Budgets.** `instances.attach` (when billed) and `networks.create` (on a `metered` pool) are budget-checked like
  `instances.create`, against the target tenant's budgets when root issues into it. An offer with
  `onBudgetExceeded: 'suspend'` stops the tenant's running VMs in the pool and refuses its new work once its budget
  is spent.
* **Estimates.** Each item of `flavors.list` on a provider pool carries the hourly `estimate` for the caller's
  tenant, and `instances.estimate` answers it before a create.

## The provider contract [#the-provider-contract]

`defineProvider` checks a definition's shape up front and returns it frozen with `extension: 'provider'`. Every
refusal is `INVALID_CONFIG` with `details.path`:

```ts title="acme-cloud.ts"
import { ComputeError, field, manifest } from '@better-compute/core';
import { defineProvider } from '@better-compute/vps';

export const acmeCloud = defineProvider({
  id: 'acme-cloud', // lower-case letters, digits, hyphens
  version: '1.0.0',
  provides: ['vps'], // 'metal', 'vps' or both
  config: manifest({
    token: field.secret().describe('API token'),
    location: field.option(['fra', 'ams']).default('fra'),
  }),
  capabilities: ['power'],
  create: (config, context) => acmeClient(config, context), // returns a ProviderImpl
});
```

`create(config, context)` receives the validated config (secret fields still as references) and a
`ProviderContext` of `{ clock, secrets, log }`. Resolve the token with `context.secrets.resolve(config.token)` when
you call the API, not once up front.

The client is a `ProviderImpl`:

| Method                                             | Contract                                                                                                                                                                                                                                                          |
| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `listOffers(query)`                                | Offers matching the query; an impossible query returns `[]`.                                                                                                                                                                                                      |
| `create({ offerId, image, userData, name }, call)` | Idempotent by `call.idempotencyKey` (vps passes the instance id): a retried create returns the same machine.                                                                                                                                                      |
| `get(id, call)`                                    | The machine, or `undefined` when it does not exist.                                                                                                                                                                                                               |
| `delete(id, call)`                                 | A machine that is already gone counts as deleted: `NOT_FOUND` is success.                                                                                                                                                                                         |
| `power?(id, action, call)`                         | `on`, `off`, `graceful-off`, `reset`, `graceful-restart`, `cycle`, `nmi`. vps uses `on` and `graceful-off`; without `power`, a stop or start is `UNSUPPORTED` and fails the instance.                                                                             |
| `reinstall?`, `rescue?`, `list?`, `bootstrap?`     | Optional. `bootstrap` delivers a single-use join token to machines that will run bcd, and never through user data.                                                                                                                                                |
| `costs?(window, call)`                             | Optional ([billing](#billing)). The account's billed cost rows whose usage started in `[from, to)`, as `ProviderCostRecord`s in integer micros, each with a `sourceRowId` that stays the same when the provider re-exports the row. Read outside any transaction. |
| `listPrices?(query, call)`                         | Optional. The provider's list prices for `billing.proposePrices`, before any margin.                                                                                                                                                                              |

A machine is an `ExternalMachine`: `{ id, name?, state, offerId?, region?, addresses }`, with `state` one of
`pending`, `running`, `stopped`, `rescue`, `deleting`, `deleted`, `error` or `unknown`. Each call gets a
`CallContext` with `idempotencyKey`, an abort `signal`, `clock`, `secrets` and `log`.

A provider whose clients read costs also declares, on its definition, which of its SKUs compute meters itself:
`billing: { computeSku?, meteredSku? }`, two pure matchers over a cost row's `service`, `sku` and
`skuDescription`. `computeSku` (vCPU, memory, GPU, instance) is what leak detection counts; neither kind is ever
passed through, and a pass-through rule whose pattern either matches is refused. Pass-through rules are accepted
only for providers that declare them.

Throw `ComputeError`s. The reconciler records any failure as a `ProviderError` condition; `UNSUPPORTED`,
`INVALID_STATE` and `INVALID_CONFIG` fail the instance, and `CAPACITY_UNAVAILABLE` or `QUOTA_EXCEEDED` from `create`, or
from `get` before the machine has run, and `INVALID_INPUT` from `create`, fail it with a `ProvisionFailed` condition instead; everything else (use
`PROVIDER_ERROR` and `RATE_LIMITED`) is retried. User data from vps carries only the tenant's SSH keys or the tenant's own cloud-init, so a provider never has
to filter secrets out of it. [Providers](/docs/extending/providers) walks through writing one end to end.

## Conformance [#conformance]

`providerConformanceCases()` from `@better-compute/vps` turns the contract into cases you run with `runConformance`
from `@better-compute/core/conformance`, in any test runner, against a live account, a recording or an emulator:

```ts title="acme-cloud.test.ts"
import { runConformance } from '@better-compute/core/conformance';
import { providerConformanceCases } from '@better-compute/vps';

const report = await runConformance(providerConformanceCases(), () => ({
  provider: acmeCloud,
  config: { token: 'env://ACME_TOKEN' },
  offerId: 'small', // an offer create accepts
  image: 'ubuntu-24.04', // an image create accepts
  context: { secrets, clock },
  prefix: `ci-${runId}`, // concurrent runs never share idempotency keys
}));
expect(report.failed).toEqual([]);
```

The cases check that the definition builds a client with the four required methods; that `listOffers` returns
well-formed offers including the subject's and nothing for `minVcpu: 1_000_000`; that a retried `create` names the
same machine; that `get` follows it, `delete` removes it and a second `delete` fails only with `NOT_FOUND`; and that
`power`, when offered, stops and starts it. Set `powerObservable: false` for providers that can act on power but not
report it. Every case cleans up the machines it made.

## Hetzner Cloud [#hetzner-cloud]

`@better-compute/provider-hetzner` exports `hetznerCloud` (id `hetzner-cloud`, `provides: ['vps']`, capabilities
`power`, `reinstall`, `list`). Servers become the instances of provider pools.

<TypeTable
  type="{
  token: {
    type: 'SecretRef',
    description:
      'A Hetzner Cloud API token with read and write access. Resolved on every request.',
    required: true,
  },
  location: {
    type: &#x22;'fsn1' | 'nbg1' | 'hel1' | 'ash' | 'hil' | 'sin'&#x22;,
    description: 'Where servers are created.',
    default: &#x22;'fsn1'&#x22;,
  },
}"
/>

* **Idempotent create.** The Hetzner API has no idempotency keys, so every server carries the labels
  `better-compute.io/managed=true` and `better-compute.io/idempotency-key=<key>`. `create` looks the key up first; if a
  concurrent create wins the unique server name, it returns the winner.
* **Names** are hostname-safe: `inst_01…` becomes `inst-01…`, at most 63 characters.
* **Offers** come from `/server_types` and `/datacenters`, following pagination: gross hourly prices in EUR for the
  account's location (or the queried `region`), `available` from that location's datacenters, deprecated types
  skipped. GPU and metal queries return nothing.
* **Power:** `on` is `poweron`, `off` is `poweroff`, `graceful-off` is `shutdown`, `reset` and `cycle` are `reset`,
  `graceful-restart` is `reboot`; `nmi` is `UNSUPPORTED`. `reinstall` is `rebuild`. `list` yields only managed
  servers.
* **Errors:** a 404 is absent (`get` answers `undefined`, `delete` and `power` throw `NOT_FOUND`); a 429 is
  `RATE_LIMITED` with `retryAfterMs` from `RateLimit-Reset`; anything else is `PROVIDER_ERROR` with
  `details: { status, code }`, and an unreachable API is `PROVIDER_ERROR` too.

`hetznerCloudProvider({ fetch, endpoint, labelKey })` builds the same provider with another transport, endpoint
(default `https://api.hetzner.cloud/v1`) or label key. `tests/provider-hetzner.test.ts` passes an in-memory emulator
of the servers, server types, datacenters and actions endpoints as `fetch`, and runs the conformance cases plus the
Hetzner specifics against it; the same hook can record or replay real traffic.

The package also exports `hetznerRobot` (`provides: ['metal']`), which claims existing Hetzner Robot dedicated
servers and bootstraps them into bcd hosts. That is platform hardware, not tenant VMs: see
[bare metal](/docs/guides/bare-metal). `providerAccounts.bootstrap` drives it for a `vps-host` or `metal-host` pool;
see [bootstrapping machines](/docs/extending/providers#bootstrapping-machines).

## Proxmox VE [#proxmox-ve]

`@better-compute/provider-proxmox` exports `proxmox` and `proxmoxProvider(options)` (id `proxmox`,
`provides: ['vps']`, capabilities `power`, `reinstall`, `list`). It clones QEMU VMs from a template on your own
cluster.

<TypeTable
  type="{
  url: {
    type: 'string',
    description: 'The API endpoint, such as https://pve.example.com:8006.',
    required: true,
  },
  token: {
    type: 'SecretRef',
    description: 'A reference to a PVE API token user@realm!tokenid=secret.',
    required: true,
  },
  region: {
    type: 'string',
    description: 'The region offers and machines report.',
    default: &#x22;'pve'&#x22;,
  },
  node: { type: 'string', description: &#x22;The node to clone onto; default the template's node.&#x22; },
  pool: { type: 'string', description: 'A resource pool for new VMs.' },
  storage: { type: 'string', description: 'Target storage for full clones.' },
  fullClone: {
    type: 'boolean',
    description:
      'false for linked clones, which need the template on the same node and storage that supports them.',
    default: 'true',
  },
  disk: {
    type: 'string',
    description: &#x22;The boot disk an offer's size grows.&#x22;,
    default: &#x22;'scsi0'&#x22;,
  },
  vmidMin: { type: 'number', description: 'Lowest VMID the account uses.', default: '100000000' },
  vmidMax: {
    type: 'number',
    description: 'Highest VMID the account uses.',
    default: '999999999',
  },
  offers: {
    type: 'string[]',
    description:
      'The sizes offered, such as 2c-4g (vCPUs and GiB of memory) or 4c-16g-100d (plus GiB of boot disk). Each is an offer id.',
    default: &#x22;['1c-2g', '2c-4g', '4c-8g', '8c-32g']&#x22;,
  },
}"
/>

The image is a template's VMID or name. The token needs `VM.Clone` on the templates, plus `VM.Allocate`,
`VM.Config.*`, `VM.PowerMgmt`, `VM.Audit` and `Datastore.AllocateSpace` where VMs are created, and `Sys.Audit` for the
node headroom that offers report.

* **Idempotent create.** A key hashes (SHA-256) to a VMID slot in the account's range, and the clone records the key
  in its description. A slot held under another key probes the next one, up to 64. Retried and concurrent creates
  meet on one VMID, where PVE lets exactly one clone through; a clone interrupted before configuration is finished by
  the next create. No free slot is `CAPACITY_UNAVAILABLE`.
* **Configuration.** After cloning, create sets cores, memory and `ipconfig0=ip=dhcp`, grows the boot disk for sized
  offers (never shrinks it), tags the VM `better-compute` last and starts it. It reports `pending` until `get` sees it
  running.
* **Shared clusters.** `delete` and `power` act only on VMs tagged `better-compute` and refuse others with
  `ACCESS_DENIED`; `list` yields only tagged VMs; templates are never machines.
* **Tasks.** Clone, stop, delete and power wait for their task; a failed task is `PROVIDER_ERROR`. `taskPollMs`
  (default 1 s) and `taskTimeoutMs` (default 10 min) tune the wait.
* **Addresses** come from the QEMU guest agent while the VM runs; without an agent the list is empty.
* **Offers** are the configured sizes, `available` when an online node (or the pinned `node`) has the CPUs and free
  memory.
* **Power:** `on`, `off`, `graceful-off`, `reset`, `graceful-restart`, and `cycle` as stop then start; `nmi` is
  `UNSUPPORTED`. `reinstall` destroys the VM and clones the new template into the same VMID, keeping its key, name,
  size, disk size and cloud-init volume.

> **Proxmox needs writeSnippet under vps.** 
  The PVE API cannot upload cloud-init snippets, so without `proxmoxProvider({writeSnippet})` a
  create refuses any user data beyond an empty `#cloud-config` with `UNSUPPORTED`. vps always
  renders at least a hostname into user data, so an account used by vps needs `writeSnippet`;
  otherwise every instance ends in `error`.

```ts
import { proxmoxProvider } from '@better-compute/provider-proxmox';

export const proxmox = proxmoxProvider({
  // Write the snippet where the cluster reads it (SFTP, a shared mount), and return its volume id.
  writeSnippet: async (name, content, call) => {
    await snippets.write(name, content); // your transport
    return `local:snippets/${name}`;
  },
  // Self-signed cluster: a fetch bound to a dispatcher that pins the cluster CA.
  fetch: pinnedFetch,
});
```

`tests/provider-proxmox.test.ts` runs the conformance cases and the PVE specifics against an in-memory PVE emulator
passed as `fetch`.

## Google Compute Engine [#google-compute-engine]

`@better-compute/provider-gcp` exports `gcp` and `gcpProvider(options)` (id `gcp`, `provides: ['vps']`, capabilities
`power`, `list`, `discover`, `claim`, `reservations`, `networks`). Compute Engine VMs, GPU machine types included,
become the instances of provider pools. It also serves [discovery and attach](#discovery-and-attach),
[provider reservations](#provider-reservations) and [tenant networks](/docs/guides/vps/networking#tenant-networks),
and calls the Compute Engine REST v1 API over plain `fetch`.

```ts
await compute.api.providerAccounts.create(
  { token },
  {
    tenantId: 'root',
    provider: 'gcp',
    name: 'main',
    config: {
      projectId: 'my-project',
      zones: ['us-central1-a'],
      credentials: 'secret://gcp/sa-key',
    },
  },
);
```

<TypeTable
  type="{
  projectId: { type: 'string', description: 'The Google Cloud project id.', required: true },
  zones: {
    type: 'string[]',
    description: 'At least one zone. Offers, discovery and reservations are limited to these.',
    required: true,
  },
  credentials: {
    type: 'SecretRef',
    description:
      'A service-account key JSON. Absent: the host identity (the tokenSource option, else the metadata server), which only a root-tenant account may use.',
  },
  impersonate: {
    type: 'string',
    description: 'A service-account email to impersonate with the base credentials.',
  },
  federationAudience: {
    type: 'string',
    description:
      &#x22;Keyless sign-in: the tenant's workload identity pool provider, //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Never together with credentials; may be combined with impersonate. See keyless GCP accounts.&#x22;,
  },
  network: {
    type: 'string',
    description: 'The VPC that create uses for an instance on no tenant network.',
    default: &#x22;'default'&#x22;,
  },
  subnet: { type: 'string', description: 'The subnetwork that create uses with that network.' },
  imageProject: {
    type: 'string',
    description: 'Where image families resolve.',
    default: &#x22;'ubuntu-os-cloud'&#x22;,
  },
  imageProjects: {
    type: 'string[]',
    description:
      &#x22;The projects whose images VMs may boot, besides imageProject and the account's own project (at most 64). Absent: the public OS image projects (defaultImageProjects).&#x22;,
  },
  ownProjectImages: {
    type: 'boolean',
    description: &#x22;Whether VMs may boot images from the account's own project.&#x22;,
    default: 'true',
  },
  diskGiB: { type: 'number', description: 'Boot disk size.', default: '100' },
  discoverLabel: {
    type: 'string',
    description: 'key=value: discovery yields only VMs carrying this label.',
  },
  billingExport: {
    type: 'string',
    description:
      'project.dataset.table of the Cloud Billing detailed (resource-level) BigQuery export: the vps.costs worker reads provider costs from it.',
  },
  billingExportLocation: {
    type: 'string',
    description: &#x22;The export dataset's BigQuery location (US, EU, us-central1, …).&#x22;,
  },
}"
/>

The account's service account needs `roles/compute.instanceAdmin.v1` (instances, labels, tags and consuming
reservations; on the owner and consumer projects for shared reservations), `roles/compute.networkAdmin` (networks
and subnets), `roles/compute.securityAdmin` (firewalls) and `roles/compute.viewer` (discovery, reservations and
future reservations), plus `roles/iam.serviceAccountTokenCreator` on the target when `impersonate` is set.

* **Billing.** Disks are reported with their types, so attached VMs bill their real disks. With `billingExport`,
  the account's client reads costs (`costs`) from the detailed BigQuery export, and `listPrices` reads the Cloud
  Billing Catalog for `billing.proposePrices`. The export has to be turned on by hand in the Cloud Console, and the
  service account needs `roles/bigquery.dataViewer` on the export dataset and `roles/bigquery.jobUser` on its
  project; the account's project needs the Cloud Billing API for list prices. The provider-gcp README has the setup.
  Compute Engine's core, RAM, GPU and instance SKUs, persistent disk and Hyperdisk capacity, local SSD and external
  IP charges are the ones compute meters itself, so they are never passed through.

* **Ids.** A VM is `zone/name`, named `bc-` and the lower-cased instance id. An offer is `zone/machineType`, or
  `zone/machineType@reservation` for capacity a reservation holds, which create consumes by name.

* **Idempotent create.** GCE's `requestId` is a UUIDv5 of the idempotency key. create does not wait for the insert;
  the reconciler polls `get`, which answers for a VM GCE does not show yet from its insert operation: `pending` while
  the operation runs, its error once it failed.

* **Failed creates end.** A stockout (`ZONE_RESOURCE_POOL_EXHAUSTED`, mapped to `CAPACITY_UNAVAILABLE`) or a quota
  error, at once or after GCE accepted the insert, moves the instance to `error` with `ProvisionFailed` and publishes
  `instance.failed`. It is not retried, keeps its quota until it is deleted, and deletes without a VM.

* **Images.** Every image a VM boots comes from a project the account allows: `imageProjects` (default
  `defaultImageProjects`, the public OS projects such as `ubuntu-os-cloud`, `debian-cloud`, `rocky-linux-cloud`,
  `ubuntu-os-accelerator-images` and `deeplearning-platform-release`), `imageProject`, and the account's own project
  unless `ownProjectImages` is `false`. An `imageId` is a family name (in `imageProject`),
  `projects/<project>/global/images/[family/]<name>`, `global/images/[family/]<name>` (the own project) or a compute
  URL of the provider's own universe. Anything else, or another project, is `INVALID_INPUT` (`details.reason`
  `imageRef` or `imageProjectNotAllowed`) before anything is read, and the instance fails with `ProvisionFailed`. A
  platform account whose project holds private images sets `ownProjectImages: false`. `images.resolve` resolves
  families through `images.getFromFamily`, with a day's cache per account that never offers a deprecated image, and
  reports each image's architecture, minimum disk, guest OS features and login user.

* **Ownership labels.** create labels the VM `better-compute-managed=true` and `better-compute-instance=<id>`. Attach
  writes `better-compute-instance` only, and `better-compute-managed` just before an `onDelete: 'delete'` deletion.
  `delete` acts only on managed VMs and `power` only on VMs carrying the instance label.

* **Power:** `on` is start, `off` and `graceful-off` stop, `reset` and `cycle` reset, and `graceful-restart` is a
  stop (waited for, at most 20 s) then a start; `nmi` is `UNSUPPORTED`.

* **Credentials.** A key is exchanged for tokens at the OAuth endpoint and cached until a minute before expiry;
  without one the host's own identity is used: `gcpProvider({ tokenSource })` (a workload-identity client), else the
  metadata server. That identity is the control plane's, so only an account of the root (platform) tenant may rely on
  it: a tenant's account must name its own key in `credentials` or sign in keyless with `federationAudience` (see
  [keyless GCP accounts](#keyless-gcp-accounts)); `impersonate` alone is not enough, and without either
  `providerAccounts.create` and `update` refuse the account with `INVALID_CONFIG`. A `tokenSource` never overrides an account's
  `credentials`. The key must be the tenant's own (`secret://<tenant>/…`, see
  [above](#whose-references-an-account-may-name)). Tokens are never logged.

* **Pinned token endpoint.** The key JSON is the tenant's, and the control plane POSTs the signed JWT to its
  `token_uri`. So `token_uri` must be absent (the default), `https://oauth2.googleapis.com/token`, or its older alias
  `https://accounts.google.com/o/oauth2/token`, compared exactly; any other value, such as an address inside the
  platform's network, is `INVALID_CONFIG` (`details.reason` `tokenUri`) on the first call that needs a token, and
  nothing is fetched.

* **Errors** map to `ComputeError` codes (`NOT_FOUND`, `CONFLICT`, `RATE_LIMITED` with `retryAfterMs`,
  `QUOTA_EXCEEDED`, `ACCESS_DENIED`, `INVALID_INPUT`, `UNAVAILABLE`), with `details: { provider: 'gcp', status, reason }`.

### Keyless GCP accounts [#keyless-gcp-accounts]

A tenant need not hand over a service-account key. With `federationAudience`, the control plane signs in to the
tenant's own project through [workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation):
Better Compute is the OIDC issuer (`pki.workloadIssuer`, served at `{basePath}/workload/.well-known/openid-configuration`
and `{basePath}/workload/jwks.json`), and the tenant's workload identity pool provider trusts it. The host must run a
workload issuer; without one an account naming `federationAudience` is `INVALID_CONFIG` (`details.reason`
`noWorkloadIssuer`). An account names either `credentials` or `federationAudience`, never both
(`credentialsAndFederation`).

Every account has exactly one subject, `compute-account:<tenantId>/<provider>/<name>`, which vps alone builds. The
tenant's pool provider must trust that subject and no other: a broader condition, such as one matching every subject
that starts with `compute-account:`, would let one account, or one tenant, act as another. `providerAccounts.create`
and `update` check this before the account is stored (see the errors below).

1. **Ask for the setup.** Call `providerAccounts.federationSetup` with the `provider` and the `name` you will give
   `create`. It calls no provider and stores nothing, and answers with the exact values to use:

   ```ts
   const setup = await compute.api.providerAccounts.federationSetup(
     { token },
     { tenantId: 'org_acme', provider: 'gcp', name: 'gcp-main' },
   );
   // → {
   //   issuer: 'https://compute.example.com/api/compute/workload',
   //   jwksUri: 'https://compute.example.com/api/compute/workload/jwks.json',
   //   subject: 'compute-account:org_acme/gcp/gcp-main',
   //   attributeCondition: 'assertion.sub == "compute-account:org_acme/gcp/gcp-main"',
   //   allowedAudienceHint: 'https://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL>/providers/<PROVIDER>',
   // }
   ```

   An unknown `provider` is `INVALID_INPUT`, as in `create`; a host with no workload issuer is `INVALID_CONFIG`
   (`details.reason` `noWorkloadIssuer`). Give `create` the same `provider` and `name`: the subject is derived from
   them, so another name is another subject.

2. **Create a workload identity pool** in the tenant's project:

   ```sh
   gcloud iam workload-identity-pools create my-pool \
     --project=my-project \
     --location=global \
     --display-name="Better Compute"
   ```

3. **Create an OIDC provider** in the pool that trusts the `issuer`, maps the subject, and admits exactly the
   `attributeCondition` from step 1:

   ```sh
   gcloud iam workload-identity-pools providers create-oidc my-provider \
     --project=my-project \
     --location=global \
     --workload-identity-pool=my-pool \
     --issuer-uri=https://compute.example.com/api/compute/workload \
     --attribute-mapping=google.subject=assertion.sub \
     --attribute-condition='assertion.sub == "compute-account:org_acme/gcp/gcp-main"'
   ```

   Google fetches the issuer's discovery document and keys itself, so the issuer URL must be reachable from the
   internet. For an issuer Google cannot reach, download the keys from `jwksUri` (`curl -o jwks.json <jwksUri>`) and
   add `--jwk-json-path=jwks.json` to the command. Uploaded keys are a copy, and Google accepts no token signed by a
   key it was not given. The operator announces a rotation ahead of time and publishes the next key in `jwks.json`
   beside the current one before it signs anything: as soon as `jwks.json` lists a key your upload lacks, download
   it again and update the provider (`gcloud iam workload-identity-pools providers update-oidc` with
   `--jwk-json-path=jwks.json`) within the announced window. Uploading only after the issuer has switched keys leaves the account unable to
   authenticate until you do. The provider's default allowed audience is its resource name as an `https://` URL, the
   `allowedAudienceHint` with your project number, pool and provider filled in, and the control plane signs for it.

4. **Grant access to the account's principal**, which is the subject in the pool:
   `principal://iam.googleapis.com/projects/<n>/locations/global/workloadIdentityPools/<pool>/subject/<subject>`.
   Either let it impersonate a service account that holds the compute roles listed above:

   ```sh
   gcloud iam service-accounts add-iam-policy-binding vms@my-project.iam.gserviceaccount.com \
     --project=my-project \
     --role=roles/iam.workloadIdentityUser \
     --member="principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/my-pool/subject/compute-account:org_acme/gcp/gcp-main"
   ```

   or grant the compute roles to that principal directly (`gcloud projects add-iam-policy-binding` with the same
   `--member`).

5. **Create the account** with `federationAudience` (and `impersonate`, for the service account of step 4):

   ```ts
   await compute.api.providerAccounts.create(
     { token },
     {
       tenantId: 'org_acme',
       provider: 'gcp',
       name: 'gcp-main',
       config: {
         projectId: 'my-project',
         zones: ['us-central1-a'],
         federationAudience:
           '//iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/my-pool/providers/my-provider',
         impersonate: 'vms@my-project.iam.gserviceaccount.com',
       },
     },
   );
   ```

   The audience here starts with `//`, not `https://`. Two accounts cannot name the same `federationAudience`
   (`CONFLICT`): a pool provider belongs to one account, in any tenant.

`create` and `update` check the account before storing it, with no transaction open and a 10 second limit on each
call. They sign probe tokens for a subject of another tenant and for another account of the same tenant, and the
pool provider must refuse both; then they exchange the account's own token, impersonate if asked, and read the
project. Each `INVALID_CONFIG` below carries `details.reason`, and nothing is stored:

| `details.reason`             | Meaning                                                                                                                                | What the tenant does                                                                                                                                                                                                                            |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `federationTrustsAnySubject` | The pool provider accepted a probe (`details.probe` is `stranger` or `sibling`): its condition is broader than this account's subject. | Set the provider's `--attribute-condition` to exactly the `attributeCondition` from `federationSetup` (`gcloud iam workload-identity-pools providers update-oidc`).                                                                             |
| `federationRefused`          | Google STS refused the account's own token (`details.stsError`). The message quotes the condition, `assertion.sub == "<subject>"`.     | Check that the provider's `--issuer-uri` is the `issuer`, that the attribute mapping is `google.subject=assertion.sub`, that the condition matches the subject exactly (so the same `name`), and that `federationAudience` names that provider. |
| `impersonationRefused`       | The federated identity may not impersonate `impersonate`. The message names the principal.                                             | Grant that principal `roles/iam.workloadIdentityUser` on the service account.                                                                                                                                                                   |
| `projectAccess`              | The final identity cannot read the project (`compute.projects.get`), or the project id is wrong, or the Compute Engine API is off.     | Grant it `roles/compute.viewer` and the roles above, check `projectId`, and enable the API (`details.activationUrl` when Google gave one).                                                                                                      |

A throttled or unreachable Google (`RATE_LIMITED`, `UNAVAILABLE`) also stores nothing: retry with the same
`idempotencyKey`. Only a 400, 401 or 403 from Google counts as a refusal, so a malformed or throttled reply can never
pass as "refused". No token is logged. Check the gcloud commands above against your gcloud version.

`tests/provider-gcp.test.ts` runs the conformance cases and the GCE specifics against an in-memory emulator passed as
`fetch`, and `tests/gcp-end-to-end.test.ts` runs the provider and the emulator through vps.

## DigitalOcean [#digitalocean]

`@better-compute/provider-digitalocean` exports `digitalocean` and `digitaloceanProvider(options)` (id `digitalocean`,
`provides: ['vps']`, capabilities `power`, `reinstall`, `list`, `discover`, `claim`, `networks`). Droplets, GPU
Droplets included, become the instances of provider pools. It also serves [discovery and attach](#discovery-and-attach)
and [tenant networks](/docs/guides/vps/networking#tenant-networks), and calls the DigitalOcean API v2 over plain
`fetch`. It reads no capacity reservations (`listReservations` is absent).

```ts
await compute.api.providerAccounts.create(
  { token },
  {
    tenantId: 'root',
    provider: 'digitalocean',
    name: 'main',
    config: {
      token: 'secret://root/do-token',
      regions: ['nyc3'],
      sshKeyFingerprint: '3b:16:e4:bf:8b:00:8b:b8:59:8c:a9:d9:f4:61:21:ec',
    },
  },
);
```

<TypeTable
  type="{
  token: {
    type: 'SecretRef',
    description:
      'A personal access token with the scopes below. Resolved on the first request of a client. An account holds exactly one of token and refreshToken.',
  },
  refreshToken: {
    type: 'SecretRef',
    description:
      'An OAuth refresh token from connecting a DigitalOcean account; rotated on every exchange (see Operations below).',
  },
  regions: {
    type: 'string[]',
    description:
      'At least one region slug. Offers, discovery, creates, claims and subnets are limited to these.',
    required: true,
  },
  vpc: {
    type: 'string',
    description:
      &#x22;A VPC name or uuid that create uses for an instance on no tenant network. Absent: the region's default VPC.&#x22;,
  },
  sshKeyFingerprint: {
    type: 'string',
    description:
      'The MD5 fingerprint of an account SSH key that is added to every Droplet. A fingerprint, not a secret.',
  },
  monitoring: {
    type: 'boolean',
    description: &#x22;Install DigitalOcean's metrics agent on new Droplets.&#x22;,
    default: 'true',
  },
  ipv6: { type: 'boolean', description: 'Give new Droplets an IPv6 address.', default: 'false' },
  allowSpot: {
    type: 'boolean',
    description:
      'Offer and create Spot GPU capacity (region/size@spot) where DigitalOcean reports a Spot price. While the Spot API shape is unverified, previews must be on too.',
    default: 'false',
  },
  discoverTag: {
    type: 'string',
    description:
      'Discovery yields only Droplets carrying this tag. Lower-case [a-z0-9:_-], at most 255 characters; anything else is INVALID_CONFIG (discoverTag).',
  },
  previews: {
    type: 'boolean',
    description:
      'Products DigitalOcean still marks as previews. Today: Spot (with allowSpot), whose API shape better-compute has not verified yet.',
    default: 'false',
  },
  costs: {
    type: 'boolean',
    description: 'Read the invoices into provider costs (vps.costs). Needs billing:read.',
    default: 'false',
  },
  projectPerTenant: {
    type: 'boolean',
    description:
      'Assign each new Droplet to the project bc-<tenant>, made on first use. Needs project:read,create,update.',
    default: 'false',
  },
}"
/>

The token needs the custom scopes `account:read`, `regions:read`, `sizes:read`, `image:read`, `actions:read`,
`droplet:read,create,update,delete` (the read also covers autoscale pool membership, which discovery reads),
`ssh_key:read,create`, `tag:read,create,delete`, `vpc:read,create,update,delete` and
`firewall:read,create,update,delete`. A missing scope is `ACCESS_DENIED` with `details.reason` `scope`.

* **Ids.** A Droplet is its decimal id, named `bc-` and the lower-cased instance id (at most 63 characters). An offer is
  `region/size`, one per size and configured region, with its hourly price in USD. GPU Droplets are listed separately
  by DigitalOcean (`type=gpus`, which can't be combined with a tag filter); the provider reads both lists for `list`
  and `discover` and for the idempotency lookup of a GPU size. A GPU size's `gpu_info.model` becomes the pool's
  `gpuModel` string (`nvidia_h100` is `H100`).
* **Idempotent create by tag.** DigitalOcean has no idempotency key, so the Droplet carries a hashed
  `better-compute-key` tag; create looks for it before the POST and again after, and when two creates raced on one key, keeps the
  oldest Droplet and deletes the rest. See
  [Idempotent create without idempotency keys](/docs/extending/providers#idempotent-create-without-idempotency-keys).
* **Ownership tags.** A Droplet is marked with flat tags `better-compute-managed:true`, `better-compute-instance:<id>`
  and `better-compute-deployment:<hash>` (the `digitaloceanProvider({ deployment })` name). `delete`, `reinstall` and
  the duplicate reaper need the managed tag and this deployment's; `power` needs the deployment's tag and an instance
  tag. The tag-scoped bulk endpoints of DigitalOcean are never called: every delete, action and tag write names one
  Droplet. A create label whose key has a colon is `INVALID_INPUT` (`label`): it could forge a reserved tag.
* **An SSH key is always sent.** A Droplet created without one may make DigitalOcean email a root password, so create
  registers the instance's keys in the account (`bc-<fingerprint prefix>`, never deleted) and adds the key named by
  `sshKeyFingerprint`. An instance with no key and an account without that field is `INVALID_CONFIG` (`noSshKey`).
* **Failed creates end.** DigitalOcean accepts a create with a 202 and runs it as an action. While the Droplet is
  `new`, `get` reads that action: an `errored` one is `CAPACITY_UNAVAILABLE`, and a size, region or capacity refusal at
  the POST is the same, so the instance ends in `error` with `ProvisionFailed`. The Droplet limit is
  `QUOTA_EXCEEDED`.
* **Attach.** `discover` yields every Droplet of the configured regions, GPU Droplets and members of a DigitalOcean
  autoscale pool included (these carry the label `better-compute-autoscale-pool` and are never claimed). `claim`
  writes the instance and deployment tags and verifies them, giving them back with `CONFLICT` when another instance's
  tag appeared. A managed claim of a Droplet that carries another deployment's tag is `INVALID_STATE`
  (`otherDeployment`). `release` removes the managed tag (unless another instance tag is on the Droplet), its own
  instance tag and, once no instance tag is left, the deployment tag. It keeps no record of what its claim wrote, so a
  managed tag that was on the Droplet before the claim goes too.
* **Power:** `on` is `power_on`, `off` is `power_off`, `graceful-off` is `shutdown`, `graceful-restart` is `reboot`,
  `cycle` and `reset` are `power_cycle`; `nmi` is `UNSUPPORTED`. `reinstall` is `rebuild`. Actions are started, not
  awaited.
* **Images.** An image is a slug, a numeric image id, or a catalog `name@version`. A catalog image is imported as a
  DigitalOcean custom image from a short-lived URL the host's `digitaloceanProvider({ catalogImages })` source hands
  over, tagged with its digest so retries and racing creates reuse one import, and copied to other regions on demand.
  A catalog ref is unique only within a tenant: vps resolves it for the instance's tenant (`vps({ images })`), refuses one
  the tenant can't see (`NOT_FOUND`), and passes the resolution (`catalogImage: { tenantId, sha256 }`) to the provider,
  whose source must look the image up by it. While it imports or copies, the create is `UNAVAILABLE` (`imageImporting`,
  `imageTransferring`) and vps retries it; a copy that errors three times or is still running a day later, or an import
  still pending a day later, fails it (`INVALID_STATE`). A failed import is deleted before the survivor is chosen, so an
  available duplicate still boots.
  An AI/ML image (`gpu-…`) on a size without GPUs or of the other GPU vendor is `INVALID_INPUT` (`gpuImage`).
* **Spot.** With `allowSpot` (and, while the wire shape is unverified, `previews`), a size that reports a Spot price
  gets a `region/size@spot` offer at that price; its Droplets carry `better-compute-capacity:spot`. Without `allowSpot`
  an `@spot` offer is `UNSUPPORTED` (`spot`), without `previews` `UNSUPPORTED` (`spotPreview`), and a Spot create
  for a size without a Spot price is refused before any request (the size catalog is loaded if need be). The Spot wire
  shape is not in DigitalOcean's published specification yet, so the provider keeps its assumptions in one place
  (`spotShape`) for the live suite to settle. The provider declares the `spot-offers` capability: only then does vps
  read an `@spot` offer as Spot (elsewhere it names a reservation called `spot`). A Spot instance whose machine
  vanishes is `lost` with a `Preempted` condition and `instance.preempted`; a pool policy `replacePreempted: true`
  recreates it on a new machine (its provider calls then carry the key `<id>:r<generation>`), at most three times in a
  row without reaching `running`, backing off from the second.
* **Bare Metal GPUs.** `digitaloceanMetalProvider({ machines, ssh })` (id `digitalocean-metal`, `provides: ['metal']`,
  platform accounts only) claims operator-listed machines over SSH with an atomic claim file, bootstraps bcd with the
  join token over SSH stdin, and reboots them (a detached `systemd-run` timer, so the exit status comes back; a
  non-zero exit or a dropped connection is `PROVIDER_ERROR`, never a retry); DigitalOcean's API has no Bare Metal
  resource. Every machine pins its
  SSH host key (`hostKeySha256`, required); calls act only on the caller's own claim; a login other than `root` runs
  each command through `sudo -n`; a release removes bcd's install and state before the claim.
* **Droplet metadata.** On a Droplet, bcd reads the metadata service into the inventory system section's `cloud`
  (Droplet id, region, tags, interfaces; never user data, keys or tokens). It is inventory only: the service is
  unsigned, so Droplets join with tokens, and the control plane reads it with `cloudInstanceOf`. A read that fails
  (a timeout, a non-200) reuses `bcd node`'s last good answer for up to six hours, so one blip does not change the
  section's hash and upload a spurious inventory change.
* **Rate budget.** DigitalOcean allows 250 requests a minute and 5,000 an hour per token, shared by every reconciler
  that uses it. The provider keeps each token under both limits: discovery and listing may spend at most 20% of a
  window, single reads 30%, and writes whatever is left, so discovery defers first. Over its share a call is
  `RATE_LIMITED` with `retryAfterMs`.
* **Errors** map to `ComputeError` codes (`INVALID_CONFIG` for a refused token, `ACCESS_DENIED`, `NOT_FOUND`,
  `CONFLICT`, `RATE_LIMITED`, `CAPACITY_UNAVAILABLE`, `QUOTA_EXCEEDED`, `INVALID_INPUT`, `UNAVAILABLE`), with
  `details: { provider: 'digitalocean', status, reason }`. Never the token.

### Operations [#operations]

Billing, projects, monitoring, security and OAuth. Each is opt-in and needs its own read scope (`billing:read`,
`project:read,create,update`, `monitoring:read`, `security:read`). Called with the fourth argument
`{ operations: true }`, `probeAccount` reports which of them a token has.

* **List prices.** DigitalOcean prices a Droplet size as a whole, so `listPrices` reports one `instance` price per
  size and account region, naming its offer, and `billing.proposePrices` prices a flavor meter whose flavors share
  that offer at its price plus the margin. No vCPU, memory or GPU price is derived from a size price.
* **Provider costs** (`costs: true`). The `vps.costs` worker reads the invoices: a closed month from its final
  invoice, the current one from the month-to-date preview. An invoice item is a month aggregate, so it is read by
  every window it overlaps and keeps its row id as it grows; the final invoice replaces the preview's amounts.
  Items carry no tags, so a Droplet item takes the instance and tenant tags of this deployment's Droplet with that id
  and is attributed to the instance; a Droplet already deleted is left to the account. `Droplets` and
  `GPU Droplets` are compute SKUs and volumes and reserved IPs metered ones, so pass-through never bills them.
* **Leaks.** An invoice item is a month aggregate whose duration is hours, so the provider declares
  `billing: { periodRows: true, billsStopped: true }`: the leak check compares an item's hours with the hours the
  instance's running and stopped intervals cover of the item's span (DigitalOcean bills a powered-off Droplet), and a
  gap over `max(leakToleranceHours, 5%)` raises `billing.leak` once per instance and month. `billing.leaks` lists them.
* **Billing summary.** `billingSummary(config, context)` reads the balance, invoices, month to date and newest
  history entries, in micros of USD.
* **Projects** (`projectPerTenant`). Each new Droplet goes to the project `bc-<tenant>`: grouping in DigitalOcean's
  console and invoices, never authority. A failed assignment is logged and never fails the create.
* **Droplet metrics.** `doMetricsSource(config, context)` is an `externalMetrics` source for attached Droplets that
  run no bcd. The host pulls it through `externalMetricsCollector` (`@better-compute/observability`), which serves
  `better_compute_external_host_*{source, instance}` gauges.
* **Security findings.** `readSecurityFindings` reads the latest posture scan, read-only. `providerAccounts.findings`
  answers it per account as findings (`security:<rule>`, severity `critical`, `warning` or `info`); a token without
  `security:read` answers one `security:unreadable` finding instead of an empty list. The admin panel's Provider
  accounts page and attention queue show findings and billing leaks.
* **Account check.** `providerAccounts.check` (CLI `better-compute provider-accounts check`, or
  `better-compute digitalocean check|offers|discover <account>`) runs the probe with the operations scopes on a
  stored account, read-only: `ok` is false when the account is not `active` or a scope Droplets need is
  `missing`; the operations scopes it lacks are `optional`, and `details` holds the account and per-family results.
* **OAuth.** An account may hold `refreshToken` instead of `token`. The host passes
  `digitaloceanProvider({ oauth: { clientId, clientSecret, redirectUris, rotate: rotateInto(store) } })`, where
  `rotateInto` writes each rotated refresh token over the account's own secret ref in the host's `SecretStore`; a
  host without `oauth` refuses OAuth accounts. A refresh token DigitalOcean no longer accepts is `INVALID_CONFIG`
  (`oauthGrant`): connect the account again.
* **Connect a DigitalOcean account.** With at least one registered `redirectUris` entry, the console's Provider
  accounts page offers it: `providerAccounts.connectSetup` answers DigitalOcean's authorization URL (scope
  `read write` by default, `oauth.scope` to change it) for a registered redirect URI and the console's state, and
  the callback page, after comparing the state with the one its tab kept, calls `providerAccounts.connect` with the
  code. Compute trades the code, seals the refresh token through the secret broker and stores the account with it as
  `refreshToken`; a refused account check deletes the sealed token again, and the state keys the call, so a retry
  replays instead of spending the code twice. Register the console's `/provider-accounts/connect/callback` URL with
  the OAuth app and in `redirectUris`; any other redirect URI is `INVALID_INPUT`.

Alert policies, Insights rules and uptime checks are cloud resources (a later phase), not provider-account settings.

`tests/provider-digitalocean.test.ts` and `tests/provider-digitalocean-networks.test.ts` run the conformance cases and
the DigitalOcean specifics against an in-memory emulator passed as `fetch`, and `tests/do-end-to-end.test.ts` runs the
provider and the emulator through vps. A few DigitalOcean behaviours are not in its specification (whether a tag
filter lists GPU Droplets, the text of some 422 messages); an opt-in live suite, `tests/live/digitalocean.live.test.ts`
(`BC_DO_LIVE`, `BC_DO_TOKEN`, `BC_DO_SSH_KEY_FINGERPRINT`), covers them and has not been run yet. Neither it nor
the emulator checks yet that DigitalOcean honours a rule's `deny`, that a firewall on a tag covers a new Droplet before
it serves traffic, or that a `PUT` renames a firewall; see the package README.

## Amazon EC2 [#amazon-ec2]

`@better-compute/provider-aws` exports `aws` and `awsProvider(options)` (id `aws`, `provides: ['vps']`, capabilities
`power`, `reinstall`, `list`, `discover`, `claim`). EC2 instances, GPU instances included, become the instances of
provider pools, and it serves [discovery and attach](#discovery-and-attach). It calls AWS over plain `fetch` through
`@better-compute/aws-core` (SigV4, the EC2 protocol, STS); there is no AWS SDK. One provider account is one AWS account
in one region: a fleet in several regions is several accounts.

```json title="fleet.json"
{
  "version": 1,
  "providerAccounts": [
    {
      "name": "aws-use1",
      "provider": "aws",
      "config": {
        "region": "us-east-1",
        "zones": ["us-east-1a", "us-east-1b"],
        "assumeRoleArn": "arn:aws:iam::222222222222:role/better-compute"
      }
    }
  ]
}
```

<TypeTable
  type="{
  region: {
    type: 'string',
    description:
      'The AWS region. It must match the AWS region grammar: it is the only config text that reaches a host name.',
    required: true,
  },
  zones: {
    type: 'string[]',
    description:
      'Availability Zone names as this account sees them, or Local Zones (us-west-2-lax-1a). Offers and creates are limited to these.',
    required: true,
  },
  credentials: {
    type: 'SecretRef',
    description:
      'An access key JSON { accessKeyId, secretAccessKey }. Allowed; a role is better.',
  },
  assumeRoleArn: {
    type: 'string',
    description:
      &#x22;The IAM role to assume, in the region's partition. better-compute sends a platform-assigned ExternalId.&#x22;,
  },
  webIdentityRoleArn: {
    type: 'string',
    description:
      &#x22;Keyless: the IAM role the account's own workload identity assumes with AssumeRoleWithWebIdentity (audience sts.amazonaws.com). Excludes credentials and assumeRoleArn.&#x22;,
  },
  subnetId: {
    type: 'string',
    description: &#x22;The subnet create uses (else the zone's default subnet).&#x22;,
  },
  securityGroupIds: { type: 'string[]', description: 'The security groups create uses.' },
  instanceProfile: {
    type: 'string',
    description: 'The instance profile ARN passed to RunInstances.',
  },
  rootVolumeGiB: { type: 'number', description: 'Root EBS volume size.', default: '100' },
  rootVolumeType: {
    type: &#x22;'gp3' | 'gp2' | 'io2'&#x22;,
    description: 'Root EBS volume type.',
    default: &#x22;'gp3'&#x22;,
  },
  kmsKeyId: {
    type: 'string',
    description: &#x22;The KMS key for EBS encryption (else the account's default).&#x22;,
  },
  metadataHopLimit: { type: 'number', description: 'The IMDSv2 hop limit.', default: '1' },
  associatePublicIp: {
    type: 'boolean',
    description: 'Give instances a public IPv4 address.',
    default: 'true',
  },
  discoverTag: {
    type: 'string',
    description: '&#x22;key=value&#x22;: discovery yields only instances carrying this tag.',
  },
  spot: { type: 'boolean', description: 'Offer and launch Spot capacity.', default: 'false' },
  quotas: {
    type: 'boolean',
    description: 'Cap offers at the EC2 vCPU quota headroom.',
    default: 'true',
  },
  pricing: {
    type: &#x22;'off' | 'on-demand' | 'all'&#x22;,
    description: 'Offer prices from the AWS Price List; all adds Spot prices.',
    default: &#x22;'off'&#x22;,
  },
  allowPurchases: {
    type: 'boolean',
    description:
      'Allow buying Capacity Blocks and new ODCRs in this platform account (spends money).',
    default: 'false',
  },
  costs: {
    type: &#x22;'off' | 'region' | 'account'&#x22;,
    description:
      &#x22;Read this account's costs from Cost Explorer, which charges per request: region (only this region, for several regional accounts on one AWS account) or account (every region). Absent or off reads none.&#x22;,
  },
  efa: {
    type: &#x22;'off' | 'auto'&#x22;,
    description:
      'auto launches EFA-capable types with one EFA interface per network card (needs subnetId or a network).',
    default: &#x22;'off'&#x22;,
  },
}"
/>

* **Credentials.** A platform (root-tenant) account may sign in as the host's own identity (environment, web
  identity, container or IMDSv2 credentials, or `awsProvider({ credentials })`). A tenant's account names an
  `assumeRoleArn` (or a static key): without either it is `INVALID_CONFIG` (`ambientCredentials`), because ambient
  credentials are the platform's. Every AssumeRole sends the ExternalId `better-compute:<providerAccountId>`, assigned
  by the platform and never configurable, so a tenant naming another tenant's role is refused by STS. A role in one of
  `awsProvider({ platformAccounts })`, or in the platform identity's own account, is refused, and after AssumeRole the
  identity must be the role's own account.
* **Onboarding a tenant.** `better-compute aws iam-policy` prints the role's permissions policy and
  `better-compute aws trust-policy --platform-principal <arn> --external-id better-compute:<providerAccountId>` its
  trust policy; `better-compute aws onboarding-template --account <providerAccountId> --platform-principal <arn>`
  renders both as one CloudFormation template. `providerAccounts.get` shows the account's `onboarding`: its `mode`,
  and for a role the `externalId` and (with `awsProvider({ platformPrincipalArn })`) the principal to trust; a keyless
  account shows its `subject`, `audience` and the host's workload `issuer` URL. The
  commands ship with the provider; add `awsIamPolicyCommand()`, `awsTrustPolicyCommand()`,
  `awsOnboardingTemplateCommand()` and `awsWorkloadRoleCommand()` to your configuration's `commands`.
* **Keyless accounts.** As with GCP's `federationAudience`, `webIdentityRoleArn` signs the account in with no key:
  its workload identity (subject `compute-account:<tenantId>/aws/<name>`, audience `sts.amazonaws.com`) is exchanged
  with `AssumeRoleWithWebIdentity`, so it needs a workload issuer but never the platform's AWS identity. Before it is
  stored, the provider checks that the role accepts the account's own subject (`federationRefused`) in the role's
  account, then refuses a stranger's and a sibling's with `AccessDenied` (`federationTrustsAnySubject`; any other
  STS answer to a probe is `UNAVAILABLE`, `federationProbeInconclusive`), and can read EC2 (`ec2Access`). `aws onboarding-template --issuer <url> --tenant <id> --name <name>` renders the IAM OIDC
  provider and that role.
* **Workload roles.** An instance of a hosted pool with `policy.workloadIdentity: true` may name an IAM role as
  `workloadIdentity.awsRoleArn`, if the pool lists it in `policy.workloadRoles.aws` (ARN globs; `INVALID_INPUT`,
  reason `awsRoleNotAllowed`, otherwise). The host gets it as `InstanceSpec.awsRole` while the pool allows it, and bcd
  serves the role's credentials to the guest from the instance's workload token. `aws workload-role --tenant <id> --issuer <url>` renders the role, trusting `sub` like `t:<tenantId>:instance:*` and the session tag `bc-role`
  equal to the role itself (`sts:TagSession` allowed). The workload broker mints a `sts.amazonaws.com` token only for
  an instance naming a role its pool still allows (refusal `audience`, reason `aws-role-not-allowed`), and tags it
  with that role, so the token assumes no other workload role of the tenant, even in the guest's hands. A
  hand-written role needs the same tag condition. `pools.update` recomputes the affected hosts' specs when the
  pool's workload settings change; credentials already issued stay valid until they expire. IAM Roles Anywhere for
  non-AWS nodes is deferred.
* **Tags and the guard.** Instances and root volumes are tagged `better-compute:managed=true` and
  `better-compute:instance=<id>`. `delete` needs the first, `power` and `reinstall` the second, and the generated
  policy makes AWS enforce the same rules (ABAC).
* **Offers and images.** An offer is `<zone>/<instanceType>` (bare metal is not offered). An image is an `ami-…`, a `resolve:ssm:<parameter>`, or one of the aliases
  `ubuntu-24.04`, `ubuntu-22.04`, `al2023`, `dlami-gpu` and `eks-al2023-<k8s>`, resolved for the instance type's
  architecture. The image comes from the tenant, so a raw `ami-…` must be public, owned by `amazon` or
  `aws-marketplace`, or owned by an account in `awsProvider({ imageOwners })`, and a raw `resolve:ssm:` must name a
  public `/aws/service/` parameter or one under `awsProvider({ ssmImagePrefixes })`; anything else is
  `INVALID_INPUT` (reason `image`). `reinstall` reads an SSM image to its AMI id first, since root-volume replacement
  takes only AMI ids.
* **Hardened defaults.** IMDSv2 only, hop limit 1, instance metadata tags off, an encrypted root volume, and a
  shutdown from inside the instance stops it.
* **Creates.** `RunInstances` carries a `ClientToken` derived from the instance id, so a retried create is the same
  instance. `get` rides out EC2's eventual consistency after a launch (by id, then by client token, then `pending`
  for two minutes). User data over 16 KiB is gzipped, and refused (`user-data-too-large`) if still over. An instance
  EC2 ends for capacity after accepting it is `CAPACITY_UNAVAILABLE`, and the instance ends in `error` with
  `ProvisionFailed`.
* **Capacity markets.** ODCRs and Capacity Blocks are provider reservations (`cr-…`, `cb:cr-…`) offered as
  `<zone>/<type>@<cr-id>` with their free slots, so [provider reservations](#provider-reservations) admit creates
  against them; a block launch sets `MarketType=capacity-block`. With `spot`, `:spot` offers launch one-time Spot
  Instances, and a reclaimed one reads as `deleted` with an `interruption`. Offers carry the vCPU quota headroom as
  `count` and, with `pricing`, Price List and Spot prices. `providerAccounts.reservationOfferings` lists capacity to
  buy, and `providerAccounts.purchaseReservation` buys it: both root-only, the purchase with `confirm: true`, exactly
  once, and only with `allowPurchases` in a platform account (otherwise refused before anything is recorded). A
  purchase whose attempts ran out while one may have bought is `unknown`, never `failed`.
* **Attach.** `discover` yields every instance that is not terminated; `claim` writes the ownership tag and re-reads
  it, turning a lost race into `CONFLICT`; deleting an attached instance detaches it and leaves it running.
* **Networks.** Tenant networks are VPCs with zone-scoped subnets, allow rules as security group rules and deny rules
  as network ACL entries (network-wide only); see [tenant networks on AWS](/docs/guides/vps/networking#on-aws). The
  account fields `ipamPoolId` (VPC ranges from Amazon VPC IPAM) and `securityGroupsPerInterface` (default 5) apply to
  them, and `aws iam-policy --phases A1,A2` prints the permissions they need.
* **EBS volumes.** `volumes` serves the provider volumes contract with EBS: volumes and snapshots tagged
  `better-compute:managed=true` plus their name, always encrypted (the account's `kmsKeyId` or default key), created
  with a `ClientToken` from the name, in the zone the spec names (else the account's only zone: a bare region on an
  account with several zones is `INVALID_INPUT`, reason `zone-required`). Types `gp3` (default), `gp2`, `io2`,
  `io1`, `st1`, `sc1` and `standard`. Attach only to an instance of this deployment in the volume's zone (`INVALID_INPUT`, reason `zone-mismatch`, otherwise), at the next free `/dev/sd[f-p]`; the guest
  sees `/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_<volume id without the dash>`. Resize grows only, and inside
  EBS's six-hour modification cooldown it is `RATE_LIMITED` with `retryAfterMs`. Restores use only this deployment's
  completed snapshots. Every change refuses a volume or snapshot without the ownership tags.
* **Costs and list prices.** An account with `costs` set has `costs` (Cost Explorer `GetCostAndUsage`, daily, by
  the `better-compute:instance` tag provider-aws writes, and the `better-compute:tenant` tag where an operator tagged
  a resource with it, filtered to the caller's own linked account), which [provider costs](#billing) read. Cost
  Explorer groups only by active cost allocation tags: activate them in the payer account with
  `better-compute aws cost-allocation-tags --activate --yes`, or nothing is attributed to instances and budgets track
  nothing. Costs are read in partition `aws` only, and need `vps({ costs: { lookbackDays } })` of 3 or more (the
  default): a shorter window is refused. `listPrices` reads the AWS Price List (partition `aws`), splitting each family's instance price into vCPU
  and GiB prices that compose back to it, for price proposals. `aws iam-policy --costs --prices` adds the permissions.
* **Governance.** These commands run as the operator's own AWS identity; `aws governance-policy` prints its IAM
  policy, with each action that changes AWS opt-in. `aws organization-accounts` renders FleetConfig provider accounts
  for an organization's ACTIVE member accounts (not its management account, unless `--include-management`), `aws quotas` and `aws quota-request --yes` read and raise EC2 vCPU quotas,
  `aws tag-policy` renders an Organizations tag policy for the better-compute keys, and `aws budget-template` an AWS
  Budgets template over everything tagged `better-compute:managed=true` (it refuses `--tenant` and `--pool`:
  provider-aws does not write those tags).
* **ML fabric.** A create with a `placement` (`cluster`, `spread` or `partition`) launches into that EC2 placement
  group, made on first use and tagged `better-compute:managed=true`; a group better-compute did not make, or one with
  another strategy, is `CONFLICT`. With `efa: 'auto'` an EFA-capable type launches with one EFA interface per network
  card. Running EFA-family machines carry `topology.aws/network-node-1..N` labels from `DescribeInstanceTopology`.
  Trainium and Inferentia devices are the accelerator models `TRN1`, `TRN2`, `INF1` and `INF2`, and an AMI of the
  wrong architecture for a Graviton (or x86) type is refused before launch (`image-arch`). `aws iam-policy --phases
  A1,A10` prints the permissions.

`tests/provider-aws.test.ts`, `tests/provider-aws-credentials.test.ts`, `tests/provider-aws-attach.test.ts`,
`tests/provider-aws-cli.test.ts`, `tests/provider-aws-networks.test.ts` and `tests/provider-aws-volumes.test.ts` run
the conformance cases and the EC2, VPC and EBS specifics against an in-process AWS emulator passed as `fetch`, and
`tests/aws-end-to-end.test.ts`, `tests/aws-networks-end-to-end.test.ts` and `tests/aws-volumes-end-to-end.test.ts` run
the provider through vps; `tests/aws-account-federation.test.ts` and `tests/vps-workload-aws-role.test.ts` cover
keyless accounts and workload roles. No live AWS suite has been run yet.


# Sandboxes (/docs/guides/vps/sandboxes)

> Short-lived isolated VMs from per-tenant templates, with a time-to-live, warm pools, snapshots, exec over SSH certificates and one IAM decision per sandbox.



A sandbox is a short-lived, isolated place to run code you do not fully trust: a build, a test suite, an AI agent's
tool call, a customer's notebook. In Better Compute a sandbox is one hosted VPS instance
made from a per-tenant **template**, with a **lifetime** after which it is deleted, optionally started from a **warm
pool** of instances the template keeps booted. It has no placement, quota or tenancy path of its own: the instance
it drives is admitted, placed, charged, addressed, metered and deleted exactly as one from `instances.create`, so the
guarantees of [instances](/docs/guides/vps/instances) hold unchanged. It is its own VM sized by a
flavor, with no join token, no `bcd` and no IAM key inside, and
its boot data sealed through the secret broker.

`vps()` serves three API groups for it (v0.5, beta): [sandboxTemplates](/docs/reference/api/sandbox-templates),
[sandboxes](/docs/reference/api/sandboxes), whose `exec` method the tunnels module contributes, and
[sandboxSnapshots](/docs/reference/api/sandbox-snapshots). All 15 methods are served. Since 2a09202 a fourth group,
[sandboxVolumes](/docs/reference/api/sandbox-volumes) (v1.1, agent memory), keeps persistent disks that sandboxes
attach across runs; see [volumes](#volumes). Migrations `vps/0004_sandboxes` to `vps/0020_volumes` add their tables,
columns and indexes, the `vps.sandboxes` reconciler expires sandboxes, keeps warm pools filled and settles snapshots,
and `vps.volumes` finishes volume deletions.

> **The control plane is done; the host runtime is untested on real hardware.** 
  Sandboxes run only on hosted (`vps-host`) pools, and a hosted instance is booted by a runtime inside `bcd` on its
  host. bcd has the `instances` converger, `instance.snapshot`, `restore_from` and the template `network` rules, and a
  host build runs the built-in `libvirt-kvm` runtime when `virsh` is installed (362317c) and `cloud-hypervisor` when
  the `cloud-hypervisor` binary is (2598f0a), each with pause and resume, snapshots with memory, and restores. Both
  have been tested only against fake tools (a fake `virsh`, a fake cloud-hypervisor API, a fake `qemu-img`): no sandbox
  has booted through either on a real KVM host yet. A host with neither and without a bcx runtime plugin fails a non-empty `instances` section with `this host runs no instance runtime`, so there a sandbox's instance is
  placed and holds its allocation and IP lease, but never boots, and the sandbox stays `starting`. fakehw builds of bcd
  run instances on a file-backed runtime, and the TypeScript FakeDaemon's `FakeHypervisor` implements pause, resume,
  snapshots and restores for the end-to-end suites. See [where sandboxes run today](#where-sandboxes-run-today).

The examples on this page use a `cloud-hypervisor` pool, the microVM runtime PLAN.md pairs with sandboxes. Security
review 12 of that runtime was fixed in 933c51b: a sandbox's taps are created on their bridge before its VMM starts,
so a guest reboot never takes them out of the template's `network` rules; each VMM runs in its own confined
`bcd-ch-<id>.service` unit that sees only its instance directory and base image; tenant images and restored
snapshots that name a backing file or an external data file are refused on both runtimes; and a memory snapshot
restores only into a sandbox of the same vCPUs and memory. See [the runtimes](/docs/daemon/delivery#the-runtimes).

## Templates [#templates]

A template belongs to one tenant and fixes what its sandboxes run. Create one with
[sandboxTemplates.create](/docs/reference/api/sandbox-templates#create) (`compute:sandboxes:manage` on the tenant):

```ts title="template.ts"
const template = await client.sandboxTemplates.create({
  tenantId: 'acme',
  name: 'py-small',
  poolId: sandboxPool, // a hosted (vps-host) pool offered to the tenant
  flavor: 'sbx.small',
  imageId: 'ubuntu-24.04',
  userData: '#cloud-config\n…', // optional
  env: { OPENAI_API_KEY: 'secret://acme/sec_01k…' }, // secret references only
  network: 'private', // default 'egress'
  defaultTtlSeconds: 1800, // default 3600
  maxTtlSeconds: 7200, // default 86400
  warm: { min: 2, max: 4 }, // default: no warm pool
});
// { id: 'sbxt_…', runtime: 'cloud-hypervisor', network: 'private', warm: { min: 2, max: 4 }, version: 1, … }
```

<TypeTable
  type="{
  name: { type: 'string', description: &#x22;1 to 63 of a-z 0-9 . _ -, starting with a letter or digit; unique among the tenant's live templates (CONFLICT otherwise). A deleted template's name is free again.&#x22;, required: true },
  poolId: { type: 'string', description: 'A vps-host pool the tenant owns or is offered, not archived. A provider pool is INVALID_INPUT: sandboxes run on hosted pools.', required: true },
  flavor: { type: 'string', description: 'A flavor of that pool, not archived (NOT_FOUND otherwise).', required: true },
  imageId: { type: 'string', description: 'Resolved with vps({ images }) for the tenant (NOT_FOUND when it does not resolve; INVALID_CONFIG without images).', required: true },
  runtime: { type: 'string', description: &#x22;Optional. When given it must equal the pool's runtime (INVALID_INPUT otherwise). The template always stores the pool's runtime, checked once here.&#x22;, default: &#x22;the pool's&#x22; },
  userData: { type: 'string', description: 'At most 32 KiB of cloud-init, or secret://<tenantId>/<id> for a secret of this tenant; any other URI is INVALID_INPUT. With vps({ sandboxExec }), a secret:// reference needs the secret broker (INVALID_CONFIG otherwise), since vps resolves it into the user data it renders. update takes null to remove it.' },
  env: { type: 'Record<string, string>', description: 'Up to 64 variables, named A-Z 0-9 _ (not starting with a digit, at most 64 characters), each a secret://<tenantId>/<id> reference to a secret this tenant wrote with secrets.put; a value is never accepted, and a platform-minted secret is INVALID_INPUT. Needs the secret broker (INVALID_CONFIG otherwise). update replaces the whole map.', default: '{}' },
  network: { type: &#x22;'egress' | 'private' | 'none'&#x22;, description: &#x22;What the sandbox's NICs may reach: anything the pool's network allows, the pool's IPAM subnet and the host, or the host only. private and none need the pool's IP range (INVALID_STATE otherwise).&#x22;, default: &#x22;'egress'&#x22; },
  defaultTtlSeconds: { type: 'number', description: 'A sandbox lifetime when create gives none. 60 s to 30 days.', default: '3600' },
  maxTtlSeconds: { type: 'number', description: 'The longest lifetime, extensions included. 60 s to 30 days, at least defaultTtlSeconds.', default: 'max(86400, defaultTtlSeconds)' },
  warm: { type: '{ min: number; max?: number }', description: 'Warm instances to keep, 0 to 64 each; max defaults to min and must be at least min.', default: '{ min: 0, max: 0 }' },
  volumes: { type: '{ volumeId | name, mountPath, readOnly? }[]', description: &#x22;Up to 8 sandbox volumes each sandbox attaches, in order. Resolved to ids when written: the tenant's own, in the same pool, ready. A template with volumes keeps no warm pool. See Volumes below.&#x22;, default: '[]' },
}"
/>

The checks are those of a hosted `instances.create`: the pool must be visible to the tenant, and its runtime must boot
the image's format and pass through the flavor's GPUs. Unknown fields are `INVALID_INPUT`. A sandbox takes its
flavor's [placement](/docs/guides/vps/flavors#placement) with the rest of its shape (9d55b36): huge-page-backed memory,
a NUMA node, pinned vCPUs.

A template that names secrets, in `env` or as a `secret://` user data, hands them to every sandbox's guest, which can
read them, while secrets are otherwise write-only. So those references must be secrets the tenant wrote itself with
`secrets.put` (never platform-minted ones such as join tokens, BMC credentials or sealed user data, which are
`INVALID_INPUT`), nor fleet secrets (`secrets.put` with `scope: 'fleet'`, recorded in `secret_refs.scope`; also
`INVALID_INPUT`). `create` and `update` also decide `compute:secrets:manage` on the tenant, with the caller's own
credential, before the write (`ACCESS_DENIED` with `reason: 'credential-required'` without one). In the root tenant,
where fleet secrets written before `core/0025_secret_scope` still read `tenant`, naming any secret also decides
`compute:secrets:manage` on the fleet (`compute-fleet/global`). An `update` that changes what the guest runs (flavor,
image, user data, `env` or `network`) is decided on the resulting template's secrets even when it names none itself,
since it hands them to new code. Creating an instance from a template checks its secrets against `secret_refs` again,
so a template written before the rule never resolves a platform-minted secret. The decisions use the deployment's IAM
type prefix (`ModuleContext.typePrefix`), falling back to `vps({ typePrefix })`.

[update](/docs/reference/api/sandbox-templates#update) changes the flavor, image, user data, `env`, `network`, TTL
bounds or warm pool and raises `version`; the pool, name and runtime stay, and a new flavor or image is checked again,
including that the pool still runs the template's runtime. The secrets check reads the template before the write's
transaction, so an update that changes what the guest runs records the secret references it was decided on, and the
write refuses it with `CONFLICT` and `details.reason: 'secrets-changed'` when the template, read under its lock,
names any other (someone added one meanwhile); retry it (7b01053). A change to the flavor, image, user data, `env` or
`network` also raises the template's internal `instance_version`: warm instances made from an older one are never
claimed and are drained, so a claim always gets an instance of the current shape. Existing sandboxes keep their
instances. [delete](/docs/reference/api/sandbox-templates#delete) hides the template from `get`, `list`, `update` and
`sandboxes.create` and frees its name; its sandboxes live on until they end, and the reconciler deletes its warm
instances. [list](/docs/reference/api/sandbox-templates#list) returns the tenant's live templates ordered by name, and
[get](/docs/reference/api/sandbox-templates#get) one of them. None of these publishes an event. Templates can also
be kept in a [FleetConfig](/docs/guides/fleet-config) document.

### Environment [#environment]

Each `env` value is resolved by the secret broker for the template's tenant when an instance is created, and written
to `/etc/sandbox/env` (owner `root`, mode 0600) as one `NAME='value'` shell assignment per line, inside the instance's
user data. vps renders that user data and seals it through the broker before the instance row is inserted, so resolved
values never reach the `instances` table, its WAL or the NodeSpec in the clear. The values never appear in the
template, the API or an event. Without a broker, a template with `env` (or, with exec trust configured, one whose user
data is a `secret://` reference) is `INVALID_CONFIG`, both when it is written and when an instance is made from it.
A warm instance holds the values resolved when it was made; changing `env` drains it, and since 2c7072b so does
rotating or deleting one of its secrets: it is never claimed again and the next pass replaces it (see
[warm pools](#warm-pools)). A running sandbox keeps the values it booted with. The broker releases nothing for a suspended or deleted tenant. Source the file from your own user data or command, for example `set -a; . /etc/sandbox/env`.

### Network [#network]

`network` becomes the instance's `InstanceSpec.network_policy`, which bcd enforces with host nftables rules on the
instance's taps, whatever the runtime (6db49c8):

| `network`          | The guest may reach                                                                                                                                                                   |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `egress` (default) | Anything the pool's network allows; no rules. Sent as the wire default, so older specs are unchanged                                                                                  |
| `private`          | Only its NICs' IPAM subnets and the host on them (the metadata service included); nothing routed beyond them, nothing from outside them                                               |
| `none`             | Only the host: DHCP, ARP, replies, and connections the host opens from its address on the subnet (bcd's tunnel dial). No other instance, nothing routed. `sandboxes.exec` still works |

bcd owns one bridge-family table, `bcd_instances`, and replaces it in a single nftables transaction whenever the
policies change, before any domain starts; the same transaction loads the backstop table `inet bcd_taps`, which drops
traffic from or to an instance tap that is not on its bridge (see
[networking](/docs/guides/vps/networking)). Taps are matched by name, so the rules exist before a runtime creates the
tap. It fails closed: a policy that cannot be expressed (a NIC without IPAM addresses) or rules that fail to load put
the instance in `error` and stop it if it runs.

Because the rules are written on the NICs' addresses, `private` and `none` need the pool's IP range (`setIpPool`): a
template with either on a pool without one is `INVALID_STATE` at create and update, and so is creating an instance
from it if the range was removed since, so vps never starts an instance bcd would have to stop. The FakeDaemon
records the policy without enforcing it.

An older bcd drops `network_policy` as an unknown field and would give the instance full egress, so a bcd that
enforces it advertises the capability `instance.network-policy` (on hosts with a runtime), which the gateway writes as
the node label `bc.dev/cap.instance.network-policy=1`. An instance with a `private` or `none` policy
(`instances.network_policy`) is placed only on nodes carrying that label, with any value (cfdb918): where no such host
has room it waits `Unschedulable` rather than run unisolated. A warm instance of a `private` or `none` template on a
host without that label (placed before the rule, or on a downgraded bcd) is never claimed, and the reconciler drains
it (34f4084). The FakeDaemon advertises the capability on VPS hosts.
`private` means the pool's whole IPAM subnet, so on a pool offered to several tenants it includes their instances
too. A `private` or `none` NIC needs IPAM addresses and a valid MAC, or the instance fails.

Whatever the policy, every instance tap, guarded or not, may not act as a DHCP server or a router (DHCP server
replies from UDP port 67 or 547, router advertisements and redirects are dropped), and may never send from or claim
the host's own addresses on the instances' bridges (6827df8). bcd also guards every tap whose NIC has a MAC and IPAM
addresses (278296e): frames must carry the NIC's MAC, IP packets its own addresses (plus `0.0.0.0` and `::` for
DHCP, and its own EUI-64 link-local address, derived from its MAC, for neighbour discovery) and ARP its own IPv4
addresses, so one guest cannot pose as another. A guest that uses a stable-privacy link-local address instead of the
EUI-64 one loses IPv6 neighbour discovery. Each guarded address, the link-local one included, is also bound to its
instance's tap: no other bridge port (an unguarded tap, a NIC without addresses, a port bcd does not manage) may
send from it, claim it in ARP or name it in a neighbour advert, and a deleted instance keeps its guard until its
domain is gone. Neighbour adverts from a guarded tap name only its own addresses (6639335). The metadata service and
instance tunnels, which name an instance by its address, trust only addresses the loaded rules bind to an instance,
never one two instances claim. Every 5 seconds bcd counts the rules in its table against the ruleset it loaded, so an
nftables reload, `flush ruleset`, or a flushed table or chain ends that trust at once, and bcd loads the ruleset
again right away rather than waiting for the next converge pass.

**SSH only from the host.** Every sandbox instance, warm ones included, has `InstanceSpec.ssh_from_host_only` set
(5346523), whatever its `network`, and bcd enforces it in the same table (474d51f, security review 13 L5). For each
of the instance's taps, tcp/22 forwarded from another bridge port (another guest, an uplink) is dropped, and tcp/22
the host sends into the tap is dropped unless its source is one of the host's own addresses on the instances'
bridges, where the exec tunnel's dial comes from; with no host address of a family, that family's tcp/22 is dropped
outright. Both rules sit ahead of every accept. So an exec certificate is usable only through a tunnel, which the
control plane re-authorizes and closes on revocation, and guests on one pool cannot reach each other's sshd. An
ordinary instance is unaffected. A bcd from before 474d51f would ignore the field, so since 82f4f51 a bcd that
enforces it advertises the capability `instance.ssh-host-only` (on hosts with a runtime), which the gateway writes as
the node label `bc.dev/cap.instance.ssh-host-only=1`, and vps places sandbox and warm instances only on nodes that
carry it; where no such host has room the instance waits in `scheduling`. `tests/vps-sandboxes.test.ts` checks that a
sandbox lands only on such a host (3b9e81d).

## Create a sandbox [#create-a-sandbox]

```ts title="sandbox.ts"
const sandbox = await client.sandboxes.create({
  tenantId: 'acme',
  templateId: template.id,
  ttlSeconds: 900, // default: the template's defaultTtlSeconds
  idempotencyKey: `job-${jobId}`,
  wait: { timeoutMs: 20_000 }, // optional: hold the answer until it leaves 'starting' (at most 30 s)
});
// { id: 'sbx_…', state: 'ready', instanceId: 'inst_…', warm: true, expiresAt: 1790000000000, … }
```

The same call is `POST /api/compute/sandboxes/create`, or
`better-compute sandbox create --template sbxt_… --ttl 15m` from the CLI (see [the CLI](#from-the-cli)). See
[sandboxes.create](/docs/reference/api/sandboxes#create).

### How a create is admitted [#how-a-create-is-admitted]

A sandbox is admitted like an instance: it exists as a draft before anyone decides on it.

      ### Transaction 1: the draft [#transaction-1-the-draft]

    The sandbox id (`sbx_` and 52 characters) derives from the tenant, the caller's identity and the idempotency key. An
    admitted sandbox with that id is returned as it is (a replay); a different request under the same key is `CONFLICT`
    with `details.sandboxId`. `wait` is not part of the compared request, but the caller's agent and delegation are
    (7b01053): an agent replaying its person's key, or the person replaying the agent's, is `CONFLICT` rather than a
    takeover of a sandbox attributed to someone else. A person's own requests hash as before. vps reads the template (`NOT_FOUND` when it is
    missing or deleted), bounds `ttlSeconds` by its `maxTtlSeconds` (`INVALID_INPUT`), checks that its flavor still
    exists (`INVALID_STATE`), checks a `fromSnapshot` (see [restore](#restore)), and writes the draft and its
    authorization row.
  
      ### The decision [#the-decision]

    Better IAM decides `compute:sandboxes:create` on `compute-sandbox/<draft id>`. The row's parent is the template's
    pool, its owner is the caller, and it carries `templateId`, `vcpu`, `memoryGiB`, `gpus` (the template's flavor),
    `ttlSeconds`, `warm` (true when the template keeps a warm pool and the create is not a restore) and, on a restore,
    `fromSnapshot`. A refusal deletes the draft; an agent under
    [delegated confirmation](/docs/guides/iam/delegated-confirmation) gets `CONFIRMATION_REQUIRED` and retries with the
    same key. After an allow, the pool's budget check runs outside any transaction, as for `instances.create`: a spent
    Better IAM budget answers `SPEND_LIMIT_REACHED` and keeps the draft. See
    [budgets](/docs/guides/iam/billing-and-usage#budgets).
  
      ### Transaction 2: claim or create [#transaction-2-claim-or-create]

    Unless the create is a restore, vps looks for the template's oldest warm instance that has settled (`stopped` or
    `paused`, not being deleted) and was made from the template's current `instance_version`, with `FOR UPDATE SKIP
    LOCKED`, so concurrent creates never wait on each other and each gets a different instance. A claimed instance moves
    to the sandbox: its owner becomes the caller, it takes the sandbox's agent and delegation attribution,
    `desiredState` becomes `running`, `claimed_at` is stamped (its warm
    metering ends there), and `instance.starting` is published. Nothing new is placed or charged; it was placed and
    charged when it was warmed. Without a claimable instance, vps creates one from the template and schedules it with the
    decision's admission context, exactly like a hosted `instances.create`: placement, quota (`QUOTA_EXCEEDED`), an IP
    lease (`CAPACITY_UNAVAILABLE`) and sealed user data. When no host fits it waits in `scheduling` and the sandbox stays
    `starting`.

    The TTL starts here, at admission: `expiresAt` is now plus `ttlSeconds`, whether the instance was warm or cold, so
    boot time counts against it. `sandbox.created` is published with `warm` and `expiresAt`.
  
`wait: { timeoutMs }` makes the call async-with-wait: the server re-reads the sandbox until it leaves `starting` and
answers with the last read when the time runs out, which is not an error. The answer's `warm` says which path was
taken. A template whose warm instances have not settled yet (still `provisioning`) gets a cold create.

A policy can bound what callers start, for example short CPU-only sandboxes:

```json title="policy statement"
{
  "sid": "ShortCpuSandboxes",
  "effect": "allow",
  "actions": ["compute:sandboxes:create", "compute:sandboxes:update"],
  "resources": ["compute-sandbox/*"],
  "conditions": {
    "NumericLessThanEquals": { "resource.ttlSeconds": 3600, "resource.gpus": 0 }
  }
}
```

## The sandbox [#the-sandbox]

<TypeTable
  type="{
  id: { type: 'string', description: 'sbx_ and 52 characters.', required: true },
  state: { type: &#x22;'starting' | 'ready' | 'failed' | 'deleting' | 'deleted'&#x22;, description: 'Derived from the instance: ready while it runs, failed in error or lost, deleting once delete or expiry asked, deleted when the instance is gone; starting otherwise.', required: true },
  instanceState: { type: 'InstanceState', description: &#x22;The instance's own state.&#x22;, required: true },
  instanceId: { type: 'string', description: 'The instance it drives.', required: true },
  templateId: { type: 'string', description: 'The template it was made from.', required: true },
  poolId: { type: 'string', description: &#x22;The template's pool.&#x22;, required: true },
  addresses: { type: 'string[]', description: &#x22;The instance's addresses.&#x22;, required: true },
  nodeId: { type: 'string', description: 'The host it is placed on.' },
  ttlSeconds: { type: 'number', description: 'The lifetime: as asked, then from createdAt to the new expiry after an extend.', required: true },
  expiresAt: { type: 'number', description: 'Epoch ms; deleted then unless extended.', required: true },
  warm: { type: 'boolean', description: &#x22;Claimed from the template's warm pool.&#x22;, required: true },
  readyAt: { type: 'number', description: 'When its instance first ran.' },
  expiredAt: { type: 'number', description: 'Set when the sandbox ended by reaching expiresAt.' },
  fromSnapshot: { type: 'string', description: 'The snapshot it was restored from.' },
  identityRebound: { type: 'boolean', description: &#x22;A memory restore: false while it still answers to its source's SSH identity (exec refuses with restore-shares-identity), true once the host installed its own. Absent for other sandboxes.&#x22; },
  snapshots: { type: 'SandboxSnapshot[]', description: 'Its snapshots, newest first, deleted ones left out. Only sandboxes.get returns them.' },
  ownerIdentityId: { type: 'string', description: 'Who created it; list defaults to these.', required: true },
  idempotencyKey: { type: 'string', description: 'The key of the create.', required: true },
  createdAt: { type: 'number', description: 'Epoch ms.', required: true },
}"
/>

<Mermaid
  chart="stateDiagram-v2
  [*] --> starting: sandboxes.create (warm claim, cold create or restore)
  starting --> ready: instance running (sandbox.ready)
  starting --> failed: instance error or lost
  ready --> failed: instance error or lost
  starting --> deleting: sandboxes.delete, or expiry (sandbox.expired)
  ready --> deleting: sandboxes.delete, or expiry (sandbox.expired)
  failed --> deleting: sandboxes.delete, or expiry (sandbox.expired)
  deleting --> deleted: instance deleted (sandbox.deleted)
  deleted --> [*]"
/>

A sandbox is `ready` the moment its instance reaches `running`: the instance reconciler stamps `readyAt` and publishes
`sandbox.ready` in the same transaction, and the sandbox reconciler's pass is only the fallback. A `failed` sandbox is
not replaced; it ends with its TTL or a delete. If its pool sets `recreateOnHostLoss`, a lost instance is recreated
like any other and the sandbox goes back to `starting`.

A sandbox whose host's bcd cannot start its instance (host admission reports it stopped with `ResourcesExhausted:`)
stays `starting` while vps places the instance elsewhere (564f543): after the refusal has stood a minute, a cold
create's instance still in `provisioning`, or a claimed warm instance whose sandbox has not become ready, releases its
booking and address and is scheduled away from every host that refused it. The sandbox keeps its `expiresAt`: the
TTL counts from the create, not from the new placement. After three re-placements the next refusal fails the
instance, and the sandbox with it. See
[hosts that cannot start an instance](/docs/guides/vps/instances#hosts-that-cannot-start-an-instance).

## Read, extend and delete [#read-extend-and-delete]

* [get](/docs/reference/api/sandboxes#get) returns one sandbox of the tenant with its snapshots, deleted ones
  included; drafts and other tenants' sandboxes are `NOT_FOUND`.
* [list](/docs/reference/api/sandboxes#list) returns `{ items, cursor? }`, newest first. `scope` is `mine` (the
  default, the caller's own) or `all`, and each item is then checked for `compute:sandboxes:read`. Deleted sandboxes
  appear only with `deleted: true`; `templateId` filters by template; `limit` is 1 to 200 (default 50), and `cursor`
  is the last sandbox id of the previous page.
* [extend](/docs/reference/api/sandboxes#extend) moves `expiresAt` later. Give exactly one of `until` (the new expiry,
  epoch ms) or `by` (added to the current expiry: `30m`, `8h`, `7d`, or milliseconds, at most 30 days). It is decided
  as `compute:sandboxes:update` with a pending `ttlSeconds` equal to the whole lifetime after the change, counted from
  `createdAt`, so a policy that caps `ttlSeconds` caps extensions too and cannot be bypassed in small steps. A lifetime
  above the template's `maxTtlSeconds` (`INVALID_INPUT`, even for a template deleted since) or past the host
  certificate is refused before the pending lifetime is written, so a refused extend leaves nothing behind for later
  decisions (the handler checks again in case the template changed in between). The handler applies the absolute expiry that was decided, not "current plus `by`" again, so two concurrent
  `by` extends cannot add up to a lifetime no decision covered. An `until` not later than the current expiry returns
  the sandbox unchanged without a decision; a repeated `by` extends again. A sandbox being deleted is `INVALID_STATE`.
  After an allow, the sandbox pool's budget check runs as for `create` and `reservations.extend` (f3b7f9e): where the
  pool's policy sets `admission.budgetCheck: 'meter'` or the tenant's offer stops new work on a spent budget (`stop-new`, `drain`), a
  spent budget is `SPEND_LIMIT_REACHED` and clears the pending lifetime, so a tenant over budget cannot keep a
  sandbox alive by extending it. If the handler throws after an allow (the sandbox or its template changed between
  the transactions), the admission's `abandoned` hook clears the pending lifetime in a transaction of its own, so it
  is not merged into later decisions; a retry with the same key decides again (a2288ec). Only one extend at a time
  stages a pending lifetime, since the authorization projection keeps one pending value per resource (f2faa81,
  migration `vps/0014_sandbox_extending`): while another extend holds the sandbox (`extending_request`), an extend
  is `CONFLICT` with `details.reason: 'extend-in-flight'` and `retryAfterSeconds`, a hold older than 60 seconds
  (measured with `compute_now()`) is taken over, the handler applies only while its hold still owns the sandbox, and
  a refusal, an abandon or a budget refusal clears the pending lifetime only when the hold is its own. The hold is a
  nonce the server mints per extend (`newId('hold')`, on the parsed input, which callers cannot set), never the
  caller's `x-request-id`: two concurrent extends sharing one request id would otherwise share the hold and reopen
  the `BoundedExtend` bypass (e68182d). The extended sandbox's
  authorization attributes come from its instance's own shape, not from its template's current flavor, which may
  have changed or gone (7fb1e01). It publishes `sandbox.extended`.
* [delete](/docs/reference/api/sandboxes#delete) marks the sandbox `deleting`, sets its instance's `desiredState` to
  `deleted` (publishing `instance.deleting` with `sandboxId`) and wakes the reconciler. It is idempotent. The
  `vps.instances` reconciler releases the allocation, IP lease, quota and sealed user data as for any instance. When
  it finishes deleting a sandbox's instance it wakes `vps.sandboxes` in the same transaction (df4b371), so the sandbox
  becomes `deleted` right after its instance, not at the next 15-second pass.

```ts
await client.sandboxes.extend({ tenantId: 'acme', sandboxId: sandbox.id, by: '30m' });
await client.sandboxes.delete({ tenantId: 'acme', sandboxId: sandbox.id });
```

## Run code inside [#run-code-inside]

There is no platform agent in a sandbox, as in any tenant VM. [sandboxes.exec](/docs/reference/api/sandboxes#exec)
gets you in over plain SSH instead: one call, decided as `compute:sandboxes:exec` on the sandbox, returns a
single-use `ssh` tunnel ticket for the sandbox's instance and a short OpenSSH user
certificate for your own public key from the deployment's sandbox SSH CA.

<Mermaid
  chart="sequenceDiagram
  participant U as You (better-compute sandbox exec)
  participant A as api (tunnels module)
  participant G as gateway
  participant D as bcd on the host
  participant S as sandbox sshd
  U->>A: sandboxes.exec (publicKey, ttlSeconds)
  A->>A: decide compute:sandboxes:exec on the sandbox
  A-->>U: ticket, path, certificate for sandbox@inst_…
  U->>G: GET /api/compute/tunnels/{ticket} (WebSocket)
  G->>D: OpenTunnel (op ssh, the instance)
  D->>S: TCP to the instance's own address, port 22
  U->>S: SSH with the certificate, over the tunnel"
/>

### How the guest trusts it [#how-the-guest-trusts-it]

With `vps({ sandboxExec: { caPublicKey, account } })`, every sandbox instance, warm ones included, gets a user data
MIME multipart whose first part is vps's own cloud-config and whose second is the template's user data (typed by its
first line; a `secret://` reference is resolved first). vps's part:

* creates `account` (default `sandbox`, a locked-password account with `/bin/bash`);
* writes the CA key to `/etc/ssh/bc_sandbox_ca.pub` and the principal `<account>@<instanceId>` to
  `/etc/ssh/bc_sandbox_principals/<account>`;
* adds `/etc/ssh/sshd_config.d/10-bc-sandbox.conf` with `TrustedUserCAKeys` and `AuthorizedPrincipalsFile`, and
  reloads sshd;
* with `certifyHostKey` (below), also writes a P-256 host key to `/etc/ssh/bc_sandbox_host_key` (0600) and its host
  certificate beside it, and adds `HostKey` and `HostCertificate` to the drop-in.

`certifyHostKey` closes the other direction: without it the guest's own SSH host key is new at every boot and a
client can only accept it on first use. With it, vps generates a host key for each sandbox instance and asks the hook
for a host certificate naming the instance id, valid for 30 days; the host wires it to the same sandbox CA's
`issueHostCertificate`. The private key travels only in broker-sealed user data, so the hook without a secret broker
is `INVALID_CONFIG`. `instances.host_certified` records it (migration `vps/0011_host_certificates`) and
`instances.host_cert_valid_before` until when the certificate holds (`vps/0012_host_cert_validity`, on snapshots too).
A memory restore inherits both through its snapshot and mints no host key of its own. Nothing renews a host
certificate, so a sandbox never outlives its guest's: a warm claim takes only instances whose certificate covers the
new TTL, the reconciler recycles warm instances whose certificate no longer covers the template's default TTL (at
most 15 days), a fresh sandbox's expiry is capped at its certificate, and a memory restore or an `extend` past it is
refused, naming how long is left (`INVALID_INPUT`, or `INVALID_STATE` once less than a minute remains).
`sandboxes.exec` returns `hostPrincipal`, `hostCertificate` (`valid`, `expired` within a minute of its end, or `none`
for a guest never certified) and `hostCertified` (`hostCertificate === 'valid'`),
so a client can pin the guest with `@cert-authority <hostPrincipal> <sandbox CA key>`. `better-compute sandbox exec`
does (see [the CLI](#from-the-cli)).

The principal names the instance, so a certificate for one sandbox opens no other, and a warm instance trusts the CA
before any sandbox exists. A memory restore is the exception: it resumes the guest without re-running cloud-init, so
its sshd still accepts the principal of the instance the memory was captured on. Snapshots record that instance, a
restore carries it in `restore_from.principalInstanceId`, and `sandboxes.exec` signs for it. A snapshot belongs to
its sandbox's owner, whoever took it, and only that owner can restore it, so only the owner's own lineage shares a
principal. Within that lineage a certificate for one guest would open the others, so today `sandboxes.exec` refuses
while more than one of them is live; see [restore](#restore).

The CA must be one `ssh-ed25519` or `ecdsa-sha2-nistp256` key line, with an optional comment of up to 64 printable
characters that may contain spaces, as pki's are (`INVALID_CONFIG` otherwise). It is one CA for the deployment, not
one per tenant as PLAN.md first described. It is separate from the [break-glass SSH](/docs/guides/access/ssh) CA, so
a sandbox certificate never opens a node; the tunnels module refuses a `tunnels.sandboxSsh` equal to `tunnels.ssh`.

### Wire it [#wire-it]

The tunnels module signs with `tunnels.sandboxSsh` and logs in as `tunnels.sandboxAccount` (default `sandbox`, which
must match vps's `account`); vps renders the same CA's public key. A [PKI bundle](/docs/operations/pki#the-sandbox-ssh-ca)
holds that CA, and `tunnelsOptions(bundle, { issuer })` returns the exec ticket issuer with both SSH CAs:

```ts title="compute.ts"
import { loadPkiBundle } from 'better-compute/cli';
import { tunnelsOptions } from 'better-compute/pki';
import { vps } from 'better-compute/vps';

const bundle = await loadPkiBundle('.better-compute/pki.json', process.env);
const sandboxCa = bundle.sandboxSshCa!;

export const compute = betterCompute({
  // database, secret, iam, pki ...
  tunnels: tunnelsOptions(bundle, { issuer: 'https://compute.example.com' }), // issuer, ssh, sandboxSsh
  modules: [
    vps({
      images,
      sandboxExec: {
        caPublicKey: sandboxCa.publicKey, // guests trust it for user certificates
        // optional: guests present host keys the same CA certifies, so clients pin them
        certifyHostKey: async ({ instanceId, principals, publicKey, validBefore }) =>
          sandboxCa.issueHostCertificate({ publicKey, principals, keyId: `host/${instanceId}`, validBefore })
            .certificate,
      },
      snapshots: snapshotStore, // see below
    }),
  ],
});
```

A bundle made before sandboxes lacks the CA: add it with `better-compute pki add-sandbox-ca`. Without
`tunnels.sandboxSsh`, `sandboxes.exec` is `UNSUPPORTED`; without `sandboxExec`, guests do not trust the CA and SSH
fails at login.

### The call [#the-call]

<TypeTable
  type="{
  tenantId: { type: 'string', description: 'The tenant acting.', required: true },
  sandboxId: { type: 'string', description: 'The sandbox (sbx_…).', required: true },
  publicKey: { type: 'string', description: 'Your OpenSSH public key line, Ed25519 or ECDSA P-256, at most 4096 characters.', required: true },
  ttlSeconds: { type: 'number', description: &#x22;Certificate lifetime, 60 to 900, and never past the caller's session or the sandbox's expiresAt.&#x22;, default: '600' },
}"
/>

It answers `{ sandboxId, instanceId, nodeId, tunnelId, ticket, path, expiresAt, certificate, serial, keyId, principal,
account, validBefore, hostPrincipal, hostCertified, hostCertificate }`; `hostPrincipal` is the name the guest's host
certificate carries (its instance, or the original one after a memory restore until the host gives the restore its
own identity), `hostCertificate` is `valid`,
`expired` or `none`, and `hostCertified` says whether it is `valid`. Open `GET {basePath}/{path}` as a WebSocket before `expiresAt` (the ticket is single use and
lives 60 s) and speak SSH over it as `account`, with the certificate saved beside your key as `<key>-cert.pub`. The
key id is `bc:<identityId>:<sandboxId>:ssh_<id>`.

| Situation                                                                                                                                                                                                                                                                      | Error                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ |
| No sandbox SSH CA configured                                                                                                                                                                                                                                                   | `UNSUPPORTED`                                                |
| An API key rather than a signed-in session                                                                                                                                                                                                                                     | `ACCESS_DENIED`, `details.reason: 'session-required'`        |
| The session ends within a minute                                                                                                                                                                                                                                               | `ACCESS_DENIED`, `details.reason: 'session-ending'`          |
| The sandbox is not `ready` (its instance not `running`), or is being deleted                                                                                                                                                                                                   | `INVALID_STATE`                                              |
| The sandbox expires within a minute (or already has): extend it first                                                                                                                                                                                                          | `INVALID_STATE`, `details.reason: 'sandbox-expiring'`        |
| Another live guest of the tenant shares the sandbox's SSH identity: its memory snapshot's source, a sibling restore, or a memory restore of this sandbox, until the host gives the restore its own (87ff2c5; see [a restore's own SSH identity](#a-restores-own-ssh-identity)) | `INVALID_STATE`, `details.reason: 'restore-shares-identity'` |
| The node is not connected, or its daemon does not advertise `tunnel.instance-dial`                                                                                                                                                                                             | as for `tunnels.open`                                        |
| More tunnels than the per-identity limit, shared with `tunnels.open`                                                                                                                                                                                                           | `RATE_LIMITED`                                               |
| An RSA or malformed key                                                                                                                                                                                                                                                        | `INVALID_INPUT`                                              |

The typed client does not list `exec`, since the tunnels module adds it at run time; call it by path, then open the
tunnel with the client's `connectTunnel`:

```ts
import { connectTunnel, tunnelUrl } from '@better-compute/client';

const grant = await client.$request<{ path: string; certificate: string; account: string }>('sandboxes/exec', {
  tenantId: 'acme',
  sandboxId: sandbox.id,
  publicKey: 'ssh-ed25519 AAAA… ci@runner',
});
const tunnel = connectTunnel({ url: tunnelUrl('https://compute.example.com', '/api/compute', grant.path) });
// bridge a local socket to `tunnel` and point your SSH client at it
```

The CLI's `better-compute sandbox exec sbx_… -- cmd` does all of it with your OpenSSH client; see
[the CLI](#from-the-cli).

### Revocation [#revocation]

Guests have no agent to receive a revocation list, so the certificate's short lifetime (at most 15 minutes, never
past your session or the sandbox) is the revocation control. The transport is re-authorized: the exec ticket names
the sandbox (`sbx`), the gateway keeps it as `TunnelInfo.sandboxId` and hands it back at each 60-second re-check, and
tunnel re-authorization (`tunnelAuthorizer` from `better-compute/iam`) re-decides `compute:sandboxes:exec` on that
sandbox, never on the instance, so withdrawn access closes an open session while a session that is still allowed
stays open. A gateway's control API can also close or re-check the tunnels into one sandbox (`sandboxId` in its
tunnel filter). Each call publishes `tunnel.issued` (subject `instance/<id>`, with `sandboxId`) and
`sandbox.exec.issued`.

`sandboxes.exec` is the only way in. `tunnels.open` refuses a sandbox's instance, and a warm one, for every op
(`exec`, `ssh`, `tcp`) with `INVALID_STATE` and `details.reason: 'sandbox-managed'` ("managed by sandbox …; use
sandboxes.exec"), so `compute:instances:exec` on the instance, from an API key or a pool-wide grant, cannot bypass the
session requirement and the `compute:sandboxes:exec` decision. `better-compute exec --instance` and `forward --instance` hit the same refusal. See [tunnels](/docs/guides/access/tunnels).

## Volumes [#volumes]

A sandbox volume (v1.1, 2a09202) is agent memory: a persistent disk on one host of a pool that the tenant's sandboxes
attach across runs, so an agent keeps its memory, caches and checkouts after each sandbox is gone. PLAN.md §3.8 counts
it as the third kind of memory, beside booked host memory and memory snapshots.

```ts title="volume.ts"
const volume = await client.sandboxVolumes.create({
  tenantId: 'acme',
  poolId: sandboxPool, // a vps-host pool
  name: 'agent-memory',
  sizeGiB: 50,
  idempotencyKey: 'agent-memory',
});
// { id: 'vol_…', name: 'agent-memory', sizeGiB: 50, serial: 'bcv-…', nodeId: 'node_…', state: 'ready', … }

await client.sandboxTemplates.update({
  tenantId: 'acme',
  templateId: template.id,
  warm: { min: 0, max: 0 }, // a template with volumes keeps no warm pool
  volumes: [{ name: 'agent-memory', mountPath: '/workspace' }],
});
```

| Method                                                                | Decided as                                                                         |                                                                                                                     |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| [`sandboxVolumes.create`](/docs/reference/api/sandbox-volumes#create) | `compute:sandboxes:manage` on the volume it would create, with `sizeGiB` (6cb30e9) | `poolId`, `name`, `sizeGiB` (1 to 16,384), `idempotencyKey`                                                         |
| [`sandboxVolumes.list`](/docs/reference/api/sandbox-volumes#list)     | `compute:sandboxes:read` on the tenant                                             | `scope` (`mine`, the default, or `all`; 77bf9f5), optional `poolId` and `includeDeleted`; at most 500, oldest first |
| [`sandboxVolumes.get`](/docs/reference/api/sandbox-volumes#get)       | `compute:sandboxes:read` on the volume                                             |                                                                                                                     |
| `sandboxVolumes.resize`                                               | `compute:sandboxes:manage` on the volume, with the new `sizeGiB` pending (6cb30e9) | `sizeGiB`, grow only (8cbece0)                                                                                      |
| [`sandboxVolumes.delete`](/docs/reference/api/sandbox-volumes#delete) | `compute:sandboxes:manage` on the volume                                           |                                                                                                                     |

A volume is the IAM resource `sandbox-volume`, whose parent is its pool, with the attribute `sizeGiB`. Since 6cb30e9
(security review 2) a policy can bound how big a volume gets:

* **Create** is decided on the volume itself, `compute-sandbox-volume/<id>`, not on the tenant. The id derives from
  the tenant, the caller and the idempotency key, and before the decision the projection gets that volume's row,
  under its pool and owned by the caller, with the requested `sizeGiB`. A key already used returns its volume without
  a new decision, and a refusal removes the staged row, so nothing is left behind. A policy that granted
  `compute:sandboxes:manage` on the tenant still reaches the volume through its pool.
* **Resize** is decided with the new size as the volume's pending `sizeGiB`, as `sandboxes.extend` decides a
  lifetime. Refusals come first (not found, not `ready`, a shrink), and the current size returns the volume
  unchanged. One resize of a volume is decided at a time: another one meanwhile is `CONFLICT`
  (`details.reason: 'resize-in-flight'`, `retryAfterSeconds: 60`), and a hold older than 60 seconds is taken over
  (migration `vps/0021_volume_resize_hold`). On success the volume's `sizeGiB` becomes the new size; a refusal, or a
  resize that fails with `QUOTA_EXCEEDED` or `CAPACITY_UNAVAILABLE`, clears the pending value.

**List** returns the caller's own volumes by default (`scope: 'mine'`, the volumes it created, as for snapshots), and
the whole tenant's with `scope: 'all'` (77bf9f5). Without a caller, `mine` matches nothing.

**Create.** vps books the volume's `diskGiB` on one host of the pool through the capacity module, in an allocation
owned by the volume (`owner_kind` `volume`, migration `core/0028_volume_allocations`), open-ended, and charged to the
tenant's `diskGiB` quota (`QUOTA_EXCEEDED` past it). No host with room is `CAPACITY_UNAVAILABLE`, and a provider pool
is `INVALID_INPUT`: volumes live on hosted pools. The name is unique among the tenant's volumes that are not deleted
(`CONFLICT`). A retry with the same idempotency key returns the same volume; the key with another request is
`CONFLICT`. The volume's serial is `bcv-` and the last 16 characters of its id, and its host's NodeSpec lists it, so
bcd creates it there as a sparse raw file (see [volumes on the host](/docs/daemon/delivery#persistent-volumes)). The
create publishes `sandbox.volume.created`.

Sparse does not mean overcommitted (851016e): bcd creates or grows a volume only when its data directory's filesystem
can hold every volume's full size, keeping a reserve of 5% (at least 2 GiB) for the instances' overlays and its own
files. Otherwise the volume keeps its size and the sandboxes that attach it report why. Hosts report the numbers in
`StatusReport.volume_storage` (37d6d6d, b8e7271), and since 131f5ee (security review 2) the control plane books
against them, since `diskGiB` is the sum of the node's disks, overcommitted, and says nothing about the filesystem the
volumes live on:

* **What is kept.** vps's status handler (`volumeStorageStatusHandler`, which `serve` and the dev stack compose)
  keeps each host's last report in `host_volume_storage` (migration `vps/0022_volume_storage`), written when a value
  moved by at least 1 GiB, the committed or outstanding bytes changed, or the stored report is 5 minutes old.
* **Headroom.** bcd's own rule: available less reserved less outstanding (the volumes' unwritten, sparse space),
  less what is booked on the host that bcd has not reported yet (the live volumes' sizes beyond its reported
  committed bytes). A host that has not reported its store has no headroom and takes no volume.
* **Create** books only on the pool's hosts with headroom for the whole size, their rows locked so concurrent
  bookings on a host are decided one after the other, else `CAPACITY_UNAVAILABLE` with
  `details.reason: 'volume-storage'`. **Resize** checks the volume's own host first: `volume-storage` with `freeGiB`
  and `neededGiB`, or `volume-storage-unknown` for a host that has not reported. `diskGiB` and quota are charged on
  top, as before.
* **Doctor** (1617d84). vps's self-check warns `vps.volumes.storage-unreported` for a host that keeps volumes
  (`bc.dev/cap.instance.volumes`) but never reported its store (an older bcd), and `vps.volumes.storage-stale` for a
  connected host whose report is over an hour old (the gateway does not compose vps's status handlers, or the host
  sends no status), and fails `vps.volumes.storage-full` for a host whose booked volumes exceed what its store can
  hold, for example after the filesystem filled up.

**Attach.** A template names up to 8 volumes, each by `volumeId` or by `name` (not both), with a `mountPath` and
`readOnly` (default false). They are resolved to ids when the template is written: each must be the tenant's own, in
the template's pool, and `ready`. A mount path is absolute, without `.` or `..` segments, never a system path (`/`,
`/bin`, `/boot`, `/dev`, `/etc`, `/lib`, `/lib32`, `/lib64`, `/proc`, `/root`, `/run`, `/sbin`, `/sys`, `/tmp`, `/usr`,
`/var`) or inside `/proc`, `/sys`, `/dev` or `/boot`; two volumes never share a path or nest. A template with volumes
keeps no warm pool (`warm.max` must be 0), since a warm instance would hold them.

A template hands its volumes to every sandbox it starts, so since 77bf9f5 (security review 2) naming a volume takes a
decision on that volume, not only the right to manage templates: `compute:sandboxes:manage` on
`compute-sandbox-volume/<id>` for a read-write mount, `compute:sandboxes:read` for a read-only one, each on the
caller's own credential (`ACCESS_DENIED` with `reason: 'credential-required'` without one), before the write and
outside its transaction. `sandboxTemplates.create` decides every volume it names; `update` decides a new `volumes`
list, and the current one whenever the update changes what the guest runs (flavor, image, user data, `env` or
`network`); a TTL or warm pool change decides nothing more. A name re-pointed at another volume, or a mount made
writable, between the decision and the write is `CONFLICT` (`reason: 'volumes-changed'`): retry, and it is decided
again. Every sandbox of the template then:

* attaches the volumes before it is placed, and one volume takes one read-write attachment on its own, or any number
  of read-only ones, never both: otherwise `CONFLICT` with `details.reason: 'volume-in-use'`;
* is placed only on the volumes' host (`PlacementRequest.onlyNodes`, 8cbece0), and only while that host carries
  `bc.dev/cap.instance.volumes` (82f4f51); otherwise it waits in `scheduling`. Volumes on different hosts are
  `INVALID_STATE`;
* gets cloud-config that formats each writable volume once as ext4 (`fs_setup` with `overwrite: false`, so existing
  data stays; a read-only volume is never formatted), mounts every volume by `/dev/disk/by-id/virtio-<serial>` with
  `defaults,nofail` (`ro` for a read-only one, and a missing volume does not stop the boot), and runs `resize2fs` on
  each writable volume at first boot.

A sandbox with a read-write volume cannot be snapshotted with its memory (see [take one](#take-one)).

**Resize.** `sandboxVolumes.resize` grows a volume and never shrinks it (a smaller size is `INVALID_INPUT`). The growth
is charged to quota and booked on the volume's host; a host without room is `CAPACITY_UNAVAILABLE`, with the quota
refunded and nothing changed. The host's spec carries the new size and bcd grows the file, telling a running guest at
once (b5415a2); the filesystem grows at the next sandbox's first boot. It publishes `sandbox.volume.resized`.

**Delete.** `sandboxVolumes.delete` is refused (`CONFLICT`) while a live instance attaches the volume or a live
template names it. Otherwise the volume becomes `deleting` and leaves its host's spec, so bcd destroys it, but its disk
stays booked until the host has applied a later spec generation (or the host is gone or retired). The `vps.volumes`
reconciler then releases the booking, which gives the quota back, and marks the volume `deleted`. Since df4b371 the
status report in which the host's applied generation passes the deletion wakes `vps.volumes` for that volume in the
same transaction (when the report carries the host's volume store), so the deletion finishes then rather than at the
next 30-second periodic pass, which remains the fallback. The events are
`sandbox.volume.deleting` and `sandbox.volume.deleted`. Tenant teardown deletes a tenant's volumes after its instances.

**Metering.** `vps.volume.gib.hours`, the size per hour, from the booking until it is released, by the `vps.usage`
job.

> **Where volumes are exposed.** 
  Besides the API and the TypeScript client, the CLI has `sandbox-volume create|list|get|resize|delete` and
  `sandbox-template create|update --volumes` (f181f43; see [from the CLI](#from-the-cli)), the MCP server has four
  volume tools (`list_sandbox_volumes`, `create_sandbox_volume`, `resize_sandbox_volume`, `delete_sandbox_volume`;
  e6759df; see [agents over MCP](#agents-over-mcp)), and a [FleetConfig](/docs/guides/fleet-config) sandbox template
  mounts up to 8 existing volumes by name (94b8b70). The console's **Volumes** page (50add82) lists, creates, grows and
  deletes them (see [in the console](#in-the-console)). Every client caps a size at the API's 16,384 GiB, and every
  list takes the per-caller scope: the CLI's `--all`, the MCP tool's `scope`, the console's **Everyone's volumes**
  (e68aba7).

## Snapshots and restore [#snapshots-and-restore]

`sandboxes.snapshot` saves a sandbox's state, disk and (by default) memory, as a tenant-private snapshot, and
`sandboxes.create({ fromSnapshot })` starts a new sandbox from it.

### The store [#the-store]

Snapshots are uploaded by the host straight to object storage you run. Give vps a `SnapshotStore`:

```ts title="snapshot-store.ts"
import type { SnapshotStore } from 'better-compute/vps';

export const snapshotStore: SnapshotStore = {
  // A presigned https PUT for the new object, and the URL restoring hosts download it from.
  async presign({ tenantId, sandboxId, snapshotId }) {
    const key = `sandboxes/${tenantId}/${snapshotId}`;
    return { uploadUrl: await presignPut(key), uploadHeaders: {}, url: await presignGet(key) };
  },
  // Optional: called after a snapshot's row is deleted (failures are logged, never retried).
  async delete({ tenantId, snapshotId, url }) {
    await removeObject(`sandboxes/${tenantId}/${snapshotId}`);
  },
};
```

Without `vps({ snapshots })`, `sandboxes.snapshot` is `INVALID_CONFIG`; an `uploadUrl` that is not https is
`INVALID_CONFIG` too. The download `url` is stored but never shown to tenants.

### Take one [#take-one]

```ts
const snap = await client.sandboxes.snapshot({
  tenantId: 'acme',
  sandboxId: sandbox.id,
  includeMemory: true, // default: a running state; false for disk only
  keep: false, // default: deleted with the sandbox
  idempotencyKey: 'nightly-42', // optional: a retry returns the same snapshot
});
// { id: 'snap_…', state: 'pending', includeMemory: true, keep: false, runtime: 'cloud-hypervisor', … }
```

It is decided as `compute:sandboxes:update` on the sandbox. The sandbox's instance must be `running` or `paused` on a
node (`INVALID_STATE` otherwise, and for a sandbox being deleted), on a runtime whose `RuntimeTraits.snapshot` is true:
`libvirt-kvm`, `cloud-hypervisor` and `firecracker` in `builtinRuntimeTraits` declare it, and any other runtime
(`containerd`, a bcx runtime that does not declare it) is `UNSUPPORTED` before any command reaches the host. vps enqueues the `instance.snapshot` command (lane
`bulk`, not retry-safe, timeout 1 h) on the instance's node with the upload URL, keyed
`vps.sandboxes:<snapshotId>:snapshot`, and records the snapshot `pending`. With an `idempotencyKey`, the snapshot id
derives from the key and a retry returns the same snapshot; the key used for another sandbox, or with another
`includeMemory` or `keep`, is `CONFLICT` (7b01053). Without
one, every call takes a new snapshot.

The host's runtime captures the instance and uploads it with its SHA-256. When the command settles, the command
sweeper or the gateway wakes `vps.sandboxes` with the snapshot id:

| Command outcome                                                          | Snapshot                                                                                                                        | Event                                                                              |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Succeeded with a valid `InstanceSnapshotResult`                          | `ready`, with `sha256`, `sizeBytes`, the runtime and, for a memory snapshot, the host's `cpuModel` as a `cpu.model` label value | `sandbox.snapshot.ready` with `snapshotId`, `includeMemory`, `sha256`, `sizeBytes` |
| Failed, rejected, expired or cancelled, or a result that does not decode | `failed`, with `reason` (the command's reason or state, or `bad-result`)                                                        | `sandbox.snapshot.failed` with `snapshotId`, `includeMemory`, `reason`             |

`sandboxes.get` lists a sandbox's snapshots (`id`, `state`, `includeMemory`, `keep`, `runtime`, `sha256`,
`sizeBytes`, `cpuModel`, `reason`, `createdAt`, `readyAt`).

Two kinds of sandbox cannot be snapshotted with their memory, and since 2a09202 vps refuses them up front with
`INVALID_STATE`, before any command reaches the host:

| Sandbox                                   | `details.reason`               | Why                                                                   |
| ----------------------------------------- | ------------------------------ | --------------------------------------------------------------------- |
| With passthrough GPUs                     | `memory-snapshot-with-gpus`    | The GPU's state lives in the device, and no VMM saves it              |
| Attaching a [volume](#volumes) read-write | `memory-snapshot-with-volumes` | The saved memory would not match the disk the volume goes on changing |

Both runtimes refuse the same on the host too (be4de2a, 550da91). Take a disk-only snapshot (`includeMemory: false`)
of such a sandbox instead.

### Restore [#restore]

`sandboxes.create({ ..., fromSnapshot: 'snap_…' })` restores one of the caller's own `ready` snapshots: a snapshot
holds its sandbox's memory and `/etc/sandbox/env`, so it belongs to the sandbox's owner, whoever took it (someone
allowed `compute:sandboxes:update` on another user's sandbox can snapshot it, but the snapshot is still the owner's),
and only that owner may boot it. An agent acting for the owner restores only snapshots of sandboxes that agent
created itself (7b01053, security review 13): snapshots record the agent of their sandbox (`sandbox_snapshots.agent_id`,
migration `vps/0015_snapshot_agent`, backfilled from the sandboxes), so an agent never boots its person's other
memory, shell history or `/etc/sandbox/env`. Another user's snapshot, even in the same tenant, another agent's,
another tenant's, or a deleted one, is `NOT_FOUND`; one still `pending` or `failed` is `INVALID_STATE`. A restore
runs under the template its snapshot was taken under: `templateId` must be that template (`INVALID_INPUT` otherwise),
so memory captured under a `none` network never resumes under an `egress` one. One taken on a runtime other than the
template's is `INVALID_INPUT` too. The decision sees `fromSnapshot`, and a restore never
claims a warm instance: it creates one whose spec carries `restore_from` (the URL, the SHA-256 and the snapshot id). A
memory snapshot places only on nodes whose `cpu.model` label equals the one it was taken on, because memory state
needs the same CPU features. bcd downloads the object over https only into `data_dir/snapshots/<sha256>`, verifies
the digest before using it, fails the instance for a snapshot from another runtime or a memory snapshot from another
CPU model, and deletes the file once no spec restores from it.

### A restore's own SSH identity [#a-restores-own-ssh-identity]

A memory restore resumes the guest with its source's sshd identity: the principal of the instance the memory was
captured on, and its host key. A certificate for one of those guests would therefore open the others. So
`sandboxes.exec` (and `better-compute sandbox exec`) refuses with `INVALID_STATE` and
`details.reason: 'restore-shares-identity'` while another live guest of the tenant (any state but `deleting` or
`deleted`) shares the sandbox's identity (87ff2c5, security review 13 H2):

* exec into a memory-restored sandbox is refused while its source, or a sibling restored from the same snapshot, is
  still live; restores of restores carry the original principal and count too;
* exec into the source is refused while a memory restore of it is live.

Disk-only snapshots (`includeMemory: false`, `better-compute sandbox snapshot --no-memory`) are not affected: their
restores boot fresh, and exec signs for the restored instance itself.

The refusal lasts until the host gives the restored guest its own identity (5346523, ca08d5c):

<Mermaid
  chart="sequenceDiagram
  participant V as vps
  participant B as bcd (host)
  participant G as Restored guest
  V->>V: seal { v: 1, account, principals, hostKey?, hostCertificate? }
  V->>B: InstanceSpec.identity_rebind_ref
  B->>V: SecretRequest (allocation_id)
  V-->>B: the bundle, sealed to a one-time key
  B->>G: hot-add a 1 MiB disk, serial bc-rebind
  G->>G: install principals and host key, reload sshd
  G-->>B: BCREBOUND nonce, in the disk's last 4 KiB
  B->>B: remove the disk, zero and delete its file
  B->>V: InstanceStatus.identity_rebound_ref
  V->>V: principal and host certificate become the restore's own"
/>

1. **vps seals it.** A memory restore on a deployment with `vps({ sandboxExec })` and a secret broker gets its own
   bundle: the principals file for `<account>@<instance>` and, with `certifyHostKey`, a fresh host key and its
   certificate, as JSON `{ v: 1, account, principals, hostKey?, hostCertificate? }`. The reference goes in
   `instances.identity_rebind_ref`, the certificate's expiry in `rebind_cert_valid_before` (migration
   `vps/0017_identity_rebind`), and the spec carries it as `InstanceSpec.identity_rebind_ref`. The bundle is deleted
   with the instance's sealed user data.
2. **bcd delivers it.** Once the instance runs, the host's agent fetches the bundle with a `SecretRequest` bound to
   the instance's allocation, checks it (version 1, the account name, principal lines, and a PEM host key with an
   OpenSSH certificate line, or neither), and hands it to the runtime off the converge pass. `cloud-hypervisor` and
   `libvirt-kvm` hot-add a raw 1 MiB disk with virtio serial `bc-rebind` holding a tar (`nonce`, `account`,
   `principals`, and `host_key` with `host_key-cert.pub` when there is a host key), and wait up to 2 minutes for the
   guest's `BCREBOUND <nonce>` at the start of the disk's last 4 KiB. Either way bcd then removes the disk and zeroes
   and deletes its file. See [identity rebind](/docs/daemon/delivery#identity-rebind).
3. **The guest installs it.** A udev rule starts `bc-rebind@<dev>.service`, whose script copies the principals and
   host key into `/etc/ssh`, reloads sshd and writes the acknowledgement.
4. **vps re-points exec.** bcd records the applied reference in `data_dir/instances-rebound.json` and reports it as
   `InstanceStatus.identity_rebound_ref`. When it equals the spec's, vps sets the instance's principal to the instance
   itself and its host certificate to the bundle's: exec then signs for that restore alone, the grant's
   `hostPrincipal` names it, and it no longer counts for `restore-shares-identity`.

`sandboxes.get` and `list` show the state as `identityRebound` (96435d6): `false` while a memory restore still
answers to its source's identity, `true` once the host installed its own, and absent for other sandboxes. Readiness
does not wait for it: a restore is `ready` when its instance runs.

The guest's side is bcd's contract in `daemon/README.md` ("Identity rebind"): the udev rule
`/etc/udev/rules.d/90-bc-rebind.rules`, the oneshot `/etc/systemd/system/bc-rebind@.service` and the script
`/usr/local/sbin/bc-rebind`. With `sandboxExec` configured, vps's sandbox user data writes all three verbatim at first
boot (owner root, the script 0755) and runs `systemctl daemon-reload` and `udevadm control --reload` before reloading
sshd (3e25c78). A memory restore resumes without cloud-init, so it inherits them from its source. A test compares the
rendered files byte for byte with the README block, so the two cannot drift apart.

> **When exec stays refused.** 
  A guest without the guest half never acknowledges, so bcd gives up after 3 tries 5 minutes apart and reports
  nothing. That fails closed: the restore keeps its source's identity, and exec stays refused while the source or a
  sibling is live. It applies to restores of snapshots whose source booted before 3e25c78 (or without `sandboxExec`),
  to a runtime that cannot hot-add the disk (a bcx runtime plugin), and to a bundle the control plane refuses to
  release or bcd finds malformed (not retried). Delete the other guests to exec into such a restore. None of this has
  run on a real KVM host: the runtimes' rebind is tested against fake guests.

### Lifetime [#lifetime]

Snapshots are tenant data that include memory. They are never promoted to shared images, and when their sandbox is
deleted they are deleted with it unless taken with `keep: true`; the store's `delete` then removes the object, outside
any transaction. A deleted snapshot keeps its object and its authorization row until its upload has settled and no
live instance restores from it (a recreate after host loss downloads it again); the reconciler then purges it (the
`purged` column), so a late upload never outlives its row. Restores read the snapshot `FOR SHARE`. A kept snapshot
outlives its sandbox and can still be restored by its owner until it is deleted, and a tenant teardown deletes every
snapshot, kept ones included.

### Manage snapshots [#manage-snapshots]

Each snapshot is its own IAM resource, `compute-sandbox-snapshot/<id>`, owned by its sandbox's owner, with the attributes
`sandboxId`, `templateId` and `includeMemory`, so a kept one stays reachable after its sandbox is gone. The
[sandboxSnapshots](/docs/reference/api/sandbox-snapshots) group manages them:

| Method                    | Action                                     | Does                                                                                                                                                                                                                                                                             |
| ------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sandboxSnapshots.list`   | `compute:sandboxes:read` on the tenant     | The tenant's snapshots, newest first, deleted ones left out and kept ones of deleted sandboxes included. `scope` is `mine` (the default) or `all` (each item checked); `sandboxId` filters; `limit` 1 to 200 (default 50); `cursor` is the last snapshot id of the previous page |
| `sandboxSnapshots.get`    | `compute:sandboxes:read` on the snapshot   | One snapshot; a deleted or unknown one is `NOT_FOUND`                                                                                                                                                                                                                            |
| `sandboxSnapshots.delete` | `compute:sandboxes:delete` on the snapshot | Deletes it at once for reads and restores, publishes `sandbox.snapshot.deleted` and wakes the reconciler, whose pass deletes the stored object once its upload settled and no live instance restores from it, so no network call runs in the request. Idempotent                 |

```ts
const { items } = await client.sandboxSnapshots.list({ tenantId: 'acme', sandboxId: sandbox.id });
await client.sandboxSnapshots.delete({ tenantId: 'acme', snapshotId: items[0].id });
```

Snapshot views carry `ownerIdentityId`, since only the owner can restore one.

## The reconciler [#the-reconciler]

`vps.sandboxes` runs every 15 seconds (`shardBy: 'none'`, one worker at a time), and `sandboxes.delete`, settled
snapshot commands and the deletion of a sandbox's instance (df4b371) wake it. Each pass does six things:

1. **Snapshots.** Pending snapshots whose command ended are settled, as above.
2. **Expiring.** A sandbox about to expire publishes `sandbox.expiring` with its `expiresAt`, once per expiry
   (`sandboxes.expiring_at`, migration `vps/0013`; f169205), so an agent driving it can extend in time instead of
   polling. The lead is five minutes (`sandboxExpiringLeadSeconds`), or a quarter of the lifetime when that is
   shorter, but never under two passes (`sandboxExpiringMinLeadSeconds`, 30 s) and never more than the lifetime
   (7fb1e01). Sandboxes whose instance is already deleting are skipped. `sandboxes.extend` clears the mark, so the new
   expiry is warned again.
3. **Expiry.** Up to 100 sandboxes whose `expiresAt` has passed and whose deletion has not started are locked with
   `SKIP LOCKED`. Each gets `expiredAt`, publishes `sandbox.expired`, and then goes through an ordinary instance delete,
   the same as `sandboxes.delete`. A sandbox can outlive its `expiresAt` by up to one period.
4. **Ready.** A sandbox whose instance is `running` and has no `readyAt` yet gets one and publishes `sandbox.ready`
   (the fallback for the instance reconciler's own stamp).
5. **Deleted.** A sandbox whose instance is `deleted` (or gone) is marked deleted, leaves the authorization
   projection, drops its snapshots that are not kept, and publishes `sandbox.deleted` with `expired` telling whether
   expiry ended it. Deleted snapshots are purged once their upload settled and no live instance restores from them.
6. **Warm pools**, below.

`vps().selfCheck` adds four sandbox findings to [`doctor`](/docs/operations/observability#doctor):

| Check                    | Severity | Means                                                                                                                                                                                                                                                                                     |
| ------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vps.sandboxes.expiry`   | error    | Sandboxes more than two passes (30 s) past `expiresAt` and not deleting: the reconciler is not running, or failing                                                                                                                                                                        |
| `vps.sandboxes.deleting` | warning  | Sandboxes deleting for over 10 minutes: their instance is stuck (its host must report it gone, or be declared lost)                                                                                                                                                                       |
| `vps.sandboxes.warm`     | warning  | Templates below `warm.min` settled instances with no refill for 10 minutes (quota, capacity, a vanished flavor or image), or warm instances not settled within 10 minutes                                                                                                                 |
| `vps.sandboxes.network`  | error    | Open sandboxes with a `private` or `none` network whose host lacks the `bc.dev/cap.instance.network-policy` label (placed before the capability constraint, or its bcd since downgraded), so their guests have full egress. Upgrade bcd on those hosts, or delete the sandboxes (bd18e35) |

The [metrics](/docs/reference/metrics) include `better_compute_sandbox_ready_seconds` by source (warm or
cold), `better_compute_sandbox_expiry_lag_seconds` and the warm-pool gauges.

### Warm pools [#warm-pools]

For each live template with `warm.min > 0`, the reconciler keeps `warm.min` warm instances of the template's current
`instance_version`. It creates at most 4 per template per pass, each in its own transaction, owned by the template's
author (`instances.warm_template_id` set). Each is placed and charged to the tenant's quota when it is created, like
any instance, and publishes `instance.created` with `warmTemplateId`. Refills are best effort: a quota refusal, a
flavor or image that is gone, or no fitting host stops that template's refill for the pass, logs `vps: warm sandbox
instance not created`, and the next pass tries again.

Warm instances are metered to the tenant, so warm pools follow its budget (4080816). Where the template's pool checks
budgets (`admission.budgetCheck: 'meter'`, or the tenant's offer has `onBudgetExceeded` `stop-new` or `drain`), the
pass asks `IamPort.billingCheck({ tenantId })`, with no caller, only when a refill or a drain depends on it: a spent
budget skips refills, and a `drain` offer drains the pool to 0. The answer is kept a minute per tenant (7fb1e01), so a
drain offer with warm instances does not cost an IAM call every pass; a failed check is logged (`vps: warm pool budget
check failed`) and leaves the pool as it is. Pools that check no budget refill without asking. Each refill re-reads its template in its own transaction (`FOR
SHARE`: not deleted, same `instance_version`), so a template deleted or reshaped since the pass began, such as by a
tenant teardown, is skipped rather than refilled. Warm instances in `error` or `lost` do not count toward
`warm.min`: they are drained and replaced.

A warm instance's `desiredState` depends on the runtime its template stores:

| Runtime `RuntimeTraits.pause`                                                       | Warm instance                                         | A claim then                                     |
| ----------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------ |
| `true` (`libvirt-kvm`, `cloud-hypervisor`, `firecracker` in `builtinRuntimeTraits`) | Booted, then `paused`: vCPUs stopped, memory resident | Resumes it (`paused` to `running`)               |
| absent or `false` (`containerd`, or a runtime you declare without it)               | `stopped`                                             | Starts it (`stopped` to `starting` to `running`) |

The lifecycle walks a paused warm instance `provisioning`, `starting`, `running`, then `paused` as the host reports
each; it becomes claimable only once it is `paused` (or `stopped`). See
[paused instances](/docs/guides/vps/instances#paused-instances).

**Rotated secrets** (2c7072b, security review 13). A warm guest holds the template's secrets as they were resolved
when it was made, so each warm instance records the `secret_refs.version` of every secret it was given
(`instances.warm_secret_versions`, migration `vps/0016_warm_secret_versions`). The versions are read before the
secrets are resolved, so a rotation that races the creation reads as stale, never as current. A warm instance whose
recorded versions differ from the current ones (a secret was rotated with `secrets.put`, or deleted) is stale:

* a claim retires a stale candidate and tries the next, up to 4 in one claim, and falls back to a cold create when
  none is current;
* each pass first retires every stale warm instance of a template that names secrets, then refills to `warm.min`.

Templates without secrets record nothing and pay for no check.

Warm instances of an older `instance_version` are drained first, then instances beyond `warm.max`, newest first since
claims take the oldest; a deleted template's are all drained. Draining sets `desiredState: 'deleted'`. Warm instances
are hidden from `instances.list` and refused by instance methods.

**Metering.** Until a sandbox claims it, a warm instance accrues `vps.sandbox.warm`: its flavor's vCPUs per hour,
billed to the tenant with no identity, so you can price idle capacity. From `claimed_at` on it accrues the ordinary
`vps.vcpu.hours`, `vps.memory.gib.hours` and GPU lines for the sandbox's owner; if the claimed instance's allocation
is recreated after a lost host, it is billed from that allocation's own start, so nothing is counted twice.
`better-compute iam setup --meters` creates the `vps.sandbox.warm` meter ("Warm sandbox vCPU hours") with the other
root meters; it must exist in Better IAM billing before a template keeps a warm pool, or its windows are
dead-lettered. See [usage](/docs/guides/capacity/quotas-and-usage).

## The instance behind a sandbox [#the-instance-behind-a-sandbox]

`instances.sandbox_id` marks the instance a sandbox drives and `instances.warm_template_id` a warm one; never both.

* `instances.start`, `stop`, `resize` and `delete` refuse both with `INVALID_STATE` ("managed by sandbox …" or "is a
  warm instance of sandbox template …"). Only the sandbox and its template move them.
* `instances.get` returns both; `instances.list` shows a sandbox's instance and hides warm instances.
* Everything else is the instance's: placement, quota, IPAM, sealed user data, host loss and recreation, the leak
  diff, `vps.usage` metering, and [workload identity](/docs/guides/vps/instances#workload-identity) when the pool
  enables it.

See [sandbox and warm instances](/docs/guides/vps/instances#sandbox-and-warm-instances).

## Events [#events]

Sandbox events have subject `sandbox/<id>`. The lifecycle events carry `templateId`, `instanceId` and `poolId`:

| Event                                               | When                                                                                                                                   | Extra `data`                                                                          |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `sandbox.created`                                   | `create` admitted it                                                                                                                   | `warm`, `expiresAt`                                                                   |
| `sandbox.ready`                                     | Its instance first ran                                                                                                                 | `warm`                                                                                |
| `sandbox.extended`                                  | `extend` moved the expiry                                                                                                              | `expiresAt`                                                                           |
| `sandbox.expiring`                                  | Once per expiry, shortly before it (see [the reconciler](#the-reconciler)); an `extend` re-arms it                                     | `expiresAt`                                                                           |
| `sandbox.expired`                                   | The reconciler found `expiresAt` passed                                                                                                | `expiresAt`                                                                           |
| `sandbox.deleted`                                   | Its instance is gone                                                                                                                   | `expired`                                                                             |
| `sandbox.withdrawn`, `sandbox.reassigned`           | Its agent or delegation ended, or its owner left (see [withdrawn agents and departing owners](#withdrawn-agents-and-departing-owners)) | `reason`; `from`, `to` for a reassignment                                             |
| `sandbox.exec.issued`                               | `exec` issued a certificate                                                                                                            | `instanceId`, `tunnelId`, `serial`, `keyId`, `principal`, `identityId`, `validBefore` |
| `sandbox.snapshot.ready`, `sandbox.snapshot.failed` | A snapshot settled                                                                                                                     | `snapshotId`, `includeMemory`, and `sha256` and `sizeBytes`, or `reason`              |
| `sandbox.snapshot.deleted`                          | `sandboxSnapshots.delete` deleted a snapshot                                                                                           | `snapshotId`, `includeMemory`                                                         |

The instance publishes its own `instance.*` events alongside, with `sandboxId` or `warmTemplateId` in `data`. Both
`sandbox.*` and `sandbox.snapshot.*` are in the events catalog, so webhooks can subscribe to them like any other
family. See [event types](/docs/guides/events#event-types).

## Authorization [#authorization]

| Method                            | Action                                                                                                                                                                                                                                                                                                                                   | Decided on                                              |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| `sandboxTemplates.create`         | `compute:sandboxes:manage`, and `compute:secrets:manage` when it names secrets; for each volume it names, `compute:sandboxes:manage` (read-write) or `compute:sandboxes:read` (read-only) on that `compute-sandbox-volume`                                                                                                               | `compute-tenant`                                        |
| `sandboxTemplates.list`           | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-tenant` + items                                |
| `sandboxTemplates.get`            | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-sandbox-template`                              |
| `sandboxTemplates.update`         | `compute:sandboxes:manage`, and `compute:secrets:manage` on the tenant (and the fleet, in the root tenant) when the resulting template names secrets and the update changes `env`, user data, flavor, image or `network`; the per-volume decisions for a new `volumes` list, or for the current one when the update changes those fields | `compute-sandbox-template`                              |
| `sandboxTemplates.delete`         | `compute:sandboxes:manage`                                                                                                                                                                                                                                                                                                               | `compute-sandbox-template`                              |
| `sandboxes.create`                | `compute:sandboxes:create`                                                                                                                                                                                                                                                                                                               | `compute-sandbox` (the draft)                           |
| `sandboxes.list`                  | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-tenant` + items                                |
| `sandboxes.get`                   | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-sandbox`                                       |
| `sandboxes.extend`, `snapshot`    | `compute:sandboxes:update`                                                                                                                                                                                                                                                                                                               | `compute-sandbox`                                       |
| `sandboxes.exec` (tunnels module) | `compute:sandboxes:exec`                                                                                                                                                                                                                                                                                                                 | `compute-sandbox`, and again at tunnel re-authorization |
| `sandboxes.delete`                | `compute:sandboxes:delete`                                                                                                                                                                                                                                                                                                               | `compute-sandbox`                                       |
| `sandboxSnapshots.list`           | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-tenant` + items                                |
| `sandboxSnapshots.get`            | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-sandbox-snapshot`                              |
| `sandboxSnapshots.delete`         | `compute:sandboxes:delete`                                                                                                                                                                                                                                                                                                               | `compute-sandbox-snapshot`                              |
| `sandboxVolumes.create`           | `compute:sandboxes:manage`, with the requested `sizeGiB`                                                                                                                                                                                                                                                                                 | `compute-sandbox-volume` (the volume it would create)   |
| `sandboxVolumes.list`             | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-tenant`                                        |
| `sandboxVolumes.get`              | `compute:sandboxes:read`                                                                                                                                                                                                                                                                                                                 | `compute-sandbox-volume`                                |
| `sandboxVolumes.resize`, `delete` | `compute:sandboxes:manage` (resize with the new `sizeGiB` pending)                                                                                                                                                                                                                                                                       | `compute-sandbox-volume`                                |

A `compute-sandbox` has the pool as its parent, so pool relations apply. The example `StopWhenOverBudget` statement
denies `compute:sandboxes:create` and `compute:sandboxes:update` (extensions and snapshots) once the caller's budget is
spent. See [actions and resources](/docs/guides/iam/actions-and-resources) and [policies](/docs/guides/iam/policies).

## Tenant teardown [#tenant-teardown]

When an IAM tenant is deleted, the built-in `tenancy` reconciler calls `vps().tenantDeleted` until it answers `DONE`.
In one transaction per call it deletes every snapshot of the tenant (kept ones too; their objects go through the
store's `delete` afterwards, once no live instance restores from them), deletes its templates (their warm pools drain), marks its open sandboxes `deleting`,
drops sandbox and instance drafts (removing the authorization rows of the deleted templates and dropped drafts), and
sets up to 100 live instances, sandboxes' and warm ones included, to
`desiredState: 'deleted'`, each publishing `instance.deleting` with `reason: 'tenant deleted'`. It answers
`SYNC_AGAIN` until every instance of the tenant is `deleted`; the `vps.sandboxes` reconciler then publishes each
sandbox's `sandbox.deleted`. See [tenancy](/docs/guides/iam/tenancy).

## Withdrawn agents and departing owners [#withdrawn-agents-and-departing-owners]

PLAN.md §12.8's IAM reactions have their vps half (9ecd28f), level-triggered and meant to run in the IAM inbox
transaction beside capacity's reservation reactions:

* `withdrawAgentSandboxes(tx, context, { tenantId, agentId | delegationId | allAgents, reason })`, for an agent
  suspended or a delegation revoked, deletes the open sandboxes they created after publishing `sandbox.withdrawn`
  with the `reason`. A sandbox is the agent's own workspace, so unlike a started reservation it does not run on;
  kept snapshots stay.
* `reassignSandboxes(tx, context, { tenantId, identityId, successorId?, reason })`, for an owner who is gone, moves
  the owner's open sandboxes and their instances, snapshots and the templates they wrote (with their warm instances)
  to the successor, authorization rows included (`sandbox.reassigned`). It clears the departed owner's agent and
  delegation from those sandboxes, so a later event about that agent cannot delete the successor's work, and it
  rewrites their authorization attributes from each instance's own shape. Without a successor it deletes the owner's
  open sandboxes and, since 7b01053, all their snapshots, kept ones included (as `sandboxSnapshots.delete` does: at
  once for reads, each publishing `sandbox.snapshot.deleted`, the objects on a later reconciler pass), since nobody
  may restore them any more and they hold memory and secrets. Their templates stay for the tenant's administrators.

Events can be missed, or race a create, so the IamReconcile sweep re-reads every identity that holds a sandbox (the
`sandboxHolders` it is given) and hands each one that is no longer active to the identity reaction as if its event
had arrived. An agent also goes to the agent reaction as a synthetic `agent:suspend` (b011ac6), so a sandbox or
reservation an agent created after its suspension is withdrawn too.

`better-compute serve` runs them when the host configuration lists them as a module reaction, right after capacity's
reservation reactions in the default agent and identity handlers (a throw retries the IAM event); the examples do:

```ts title="better-compute.config.ts"
import { defineHost } from 'better-compute/cli';
import { sandboxIamReactions } from 'better-compute/vps';

export const host = defineHost(async () => ({
  // { agent: withdrawAgentSandboxes, identity: reassignSandboxes }; also used with dev --sim's embedded IAM,
  // and host.iam.reactions takes the same shape
  iamReactions: [sandboxIamReactions],
}));
```

## From the CLI [#from-the-cli]

The `sandbox` commands (group `Sandboxes`) call the API as `BETTER_COMPUTE_TOKEN`, in process or with `--url`:

```bash
better-compute sandbox-template create --name python --pool pool_01k5w… --flavor small --image img_01k5w… \
  --warm-min 2 --max-ttl 4h                              # compute:sandboxes:manage
better-compute sandbox-template update sbxt_01k5w… --warm-min 4   # only the fields given; --no-user-data clears it
better-compute sandbox templates --format table          # the tenant's templates and warm pools
better-compute sandbox create --template sbxt_01k5w… --ttl 30m
better-compute sandbox create --template sbxt_01k5w… --from-snapshot snap_01k5w…
better-compute sandbox list --all --format table
better-compute sandbox extend sbx_01k5w… --by 30m        # or --until 2026-10-01T12:00:00Z, or --until +2h
better-compute sandbox snapshot sbx_01k5w… --keep        # --no-memory for disk only
better-compute sandbox exec sbx_01k5w… -- uname -a       # or no command for a login shell
better-compute sandbox delete sbx_01k5w…
```

`sandbox exec` reads the public key beside `--key` (default `~/.ssh/id_ed25519`, so `~/.ssh/id_ed25519.pub`; the
private key never leaves your machine), asks `sandboxes.exec` for a certificate of `--minutes` (default 10, 1 to 15),
listens on a random `127.0.0.1` port for exactly one connection, and runs your OpenSSH client (`--ssh`, default
`ssh`) against it with `-F none` (neither your nor the system's `ssh_config`), the certificate, `IdentitiesOnly=yes`
and a known-hosts file for this one connection, used as both the user and the global known-hosts file. When
the grant says `hostCertified`, that file holds `@cert-authority <hostPrincipal> <sandbox CA key>` (the key that
signed your user certificate in the same authenticated answer, so there is no new trust input) and ssh runs with
`HostKeyAlias=<hostPrincipal>` and `StrictHostKeyChecking=yes`: nothing that took the local port, and nothing on the
node's network, can pose as the guest; a certified answer whose host principal is malformed fails
(`INVALID_RESPONSE`) instead of falling back. A guest without a valid host certificate (created before
`certifyHostKey`, on a deployment without it, or whose certificate expired) is refused with `INVALID_STATE` before ssh
runs, unless you pass `--accept-host-key`: then that one connection gets an empty file,
`StrictHostKeyChecking=accept-new` and `HostKeyAlias` set to the sandbox id, and the CLI says it trusts the key on
first use. The certificate lives in a temporary directory that is
removed afterwards, and the command exits with ssh's status. It needs `--url` and a signed-in session.

Because the guest is untrusted and the command may come from an agent or a model, `sandbox exec` puts `--` before the
host (a command such as `-oProxyCommand=…` runs in the guest, never as an ssh option on your machine) and turns off
the parts of your ssh configuration that would hand the guest anything: agent and X11 forwarding (`-a -x`), all port
forwardings, `ProxyCommand` and `ProxyJump`, connection sharing (`ControlMaster`, `ControlPath`) and
`PermitLocalCommand`. When `sandboxes.exec` refuses with `restore-shares-identity`, `sandbox exec` adds a hint:
delete the sandbox it was restored from or its sibling restores, or wait until the host gives it its own identity,
which `sandbox get` shows as `identityRebound: true` (9958865). `sandbox-template create|update|delete` manage
templates from a terminal.

[Volumes](#volumes) have their own commands (f181f43), in the same `Sandboxes` group:

```bash
better-compute sandbox-volume create --pool pool_01k5w… --name agent-notes --size 20   # GiB, 1 to 16,384
better-compute sandbox-volume list --format table        # yours; --all for the tenant's, --pool ID, --deleted
better-compute sandbox-volume get vol_01k5w…
better-compute sandbox-volume resize vol_01k5w… --size 40                            # grow only
better-compute sandbox-volume delete vol_01k5w…          # wipes the data
better-compute sandbox-template update sbxt_01k5w… --warm-min 0 --warm-max 0 \
  --volumes agent-notes:/workspace,datasets:/data:ro
```

`sandbox-volume create` sends a fresh idempotency key unless you pass `--key KEY`, so repeat it with the same `--key`
to retry safely. `get`, `resize` and `delete` take a volume id and refuse anything that is not `vol_…` before calling
the API. `--volumes` on `sandbox-template create` and `update` lists `NAME:/mount/path` or `NAME:/mount/path:ro`
entries, comma-separated and in order; an entry that looks like a volume id (`vol_…`) is sent as `volumeId`, anything
else as `name`, and a malformed entry (no absolute path, a mode other than `ro`) is a usage error. `--volumes ""`
removes every volume from the template. The server's checks in [attach](#volumes) still apply, so a template with
volumes also needs `--warm-min 0 --warm-max 0` (`--warm-max` needs `--warm-min`). See
[the CLI guide](/docs/guides/clients/cli#sandboxes) and the [reference](/docs/reference/cli#sandbox-create).

## Agents over MCP [#agents-over-mcp]

The [MCP server](/docs/guides/clients/mcp#the-tools) has eight sandbox tools: `list_sandbox_templates`,
`create_sandbox` (`templateId`, `ttlSeconds`, `idempotencyKey`, `waitSeconds` default 20; it answers
`{ idempotencyKey, sandbox }` and turns `CONFIRMATION_REQUIRED` into a result the model can retry), `list_sandboxes`,
`get_sandbox`, `extend_sandbox` (`sandboxId`, `by`), `delete_sandbox`, `list_sandbox_snapshots` (`scope`,
`sandboxId`, `limit`, `cursor`) and `delete_sandbox_snapshot` (`snapshotId`). No tool takes a snapshot or runs code
in a sandbox.

Four more tools manage [volumes](#volumes), an agent's memory (e6759df): `list_sandbox_volumes` (`poolId`, and
`scope`, `mine` by default or `all`; e68aba7),
`create_sandbox_volume` (`poolId`, `name`, `sizeGiB`, `idempotencyKey`; confirmation-aware like `create_sandbox`),
`resize_sandbox_volume` (`volumeId`, `sizeGiB`; grow only) and `delete_sandbox_volume` (`volumeId`; destructive, it
wipes the data). `create_sandbox_volume` is visible with `compute:pools:read` on its pool, and the other three are
open to any authenticated caller. No tool attaches a volume to a template.

`gateTools()` has a rule for each of them, so a gate built with `unlisted: 'deny'` shows them: `create_sandbox` is
visible with `compute:sandboxes:read` on its template (`compute-sandbox-template/{templateId}`), and the other seven
are open to any authenticated caller, since compute decides every call on the tenant, the sandbox, the snapshot or the
draft. See
[the gate](/docs/guides/clients/mcp#the-gate-checks-visibility-only).

## In the console [#in-the-console]

The console's **Sandboxes** page lists the tenant's templates with their warm pool sizes in a create form (template
and lifetime in minutes, capped at the template's maximum), and the person's own sandboxes, or everyone's, with a
countdown to `expiresAt&#x60; that turns to a warning in the last 5 minutes. Each row offers **+30 min**, **Snapshot** and
**Delete**, and shows the `better-compute sandbox exec sbx_…` line to copy, since a shell runs through the CLI. A
memory restore whose `identityRebound` is `false` carries a warning that it still answers to its source's SSH
identity, so exec waits until the source and sibling restores are gone or the host gives it its own (9958865). On a
deployment without a `SnapshotStore`, the first **Snapshot** answers `INVALID_CONFIG`; the page then says snapshots
are not configured and hides the button for the session. See [the console](/docs/guides/clients/console#screens).

The **Volumes** page (`/volumes`, 50add82) manages [volumes](#volumes): a form that creates one from a pool, a name
and a size in GiB (1 to 16,384), with a fresh `console-…` idempotency key; the person's own volumes, or with
**Everyone's volumes** the tenant's (e68aba7), each with its id, its `/dev/disk/by-id/virtio-<serial>` path, pool,
host, size and state; and, for a `ready` volume, **Grow** (asks for the new size, which must be larger) and **Delete**
(asks for confirmation, warning that the data is wiped). Volumes are attached to templates on the **Sandboxes** page
(dd5555b): its templates table lists each template's volumes with **Detach**, and **Attach a volume** picks a
template, a `ready` volume of its pool, a mount path and read-only, sending the whole list through
`sandboxTemplates.update`; **Warm +1** is off for a template with volumes. `sandbox-template update --volumes`, the
API and FleetConfig do the same.

## Where sandboxes run today [#where-sandboxes-run-today]

| Layer                                                                                     | State                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API, admission, TTL, warm pools, `env`, `network`, snapshots, exec trust, guard, teardown | Built and tested against PGlite with emulated host reports (`tests/vps-sandboxes.test.ts`, 33 tests; `tests/server-sandbox-exec.test.ts`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| End to end                                                                                | `tests/e2e/sandboxes.test.ts` runs the real server runtime and gateway with three FakeDaemon VPS hosts whose fake `cloud-hypervisor` pauses: a warm claim reaches `ready` without a placement (the paused instance resumes on the same node and allocation, booted once) and the pool refills to `warm.min`; 500 sandboxes (`E2E_SANDBOXES`) with a 60 s TTL expire on their own, each publishing `sandbox.expired` then `sandbox.deleted` once, with no leaks on hosts or in the database (domains, overlays, taps, IP leases, bookings, scalar claims) and the readiness and expiry series counted; a memory snapshot taken on one sandbox restores its guest memory into a new one (and the stored object goes with the sandbox); deleting a warm template drains its pool until the hosts are empty; `instances.stop`, `start` and `delete` on a sandbox's instance are refused `INVALID_STATE` naming the sandbox, and another tenant gets `NOT_FOUND` restoring from or reading the snapshot (bfb1d45). `tests/e2e/volumes.test.ts` (38405a0) runs the same shape over volumes: 40 sandboxes by default and 240 in the nightly run (`E2E_VOLUME_SANDBOXES`, df4b371) churned over 12 volumes on three hosts with 150 GiB volume stores, through two host reboots, with no leak, no second writer, no overbooked store, and the last writer's data found by the next sandbox; see [volumes at scale](/docs/operations/testing#volumes-at-scale)                                                                                                                                                                                                                                                                                                                                                                                               |
| Under `bcd helper`                                                                        | `tests/e2e/helper-vps.test.ts` (1c9684a) runs 12 sandboxes (`E2E_HELPER_SANDBOXES`) on two FakeDaemon VPS hosts playing a bcd under `bcd helper` (`helperVpsHost()`: `cloud-hypervisor`, with `br0` as the one bridge of `[helper] instance_bridges`), through the real server runtime and gateway: the hosts carry the system label `bc.dev/driver.cloud-hypervisor` from their `Hello`, every sandbox reaches `ready` with its VMM running, their live usage (`bcd.instance.*`) is exported to the collector `nodeTelemetryModule` names and a guest's work shows up in its host's next export, and deleting them leaves no leak and no usage in the next export. On the Go side `TestCloudHypervisorUnderTheHelper` starts a VMM through root's `ch.*` operations and reads its usage as the node, with no `ch.*` call                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Volumes on the FakeDaemon                                                                 | `FakeHypervisor` keeps host volumes as bcd does (38405a0): created or grown only with room in the store, never shrunk, one writer, kept across power loss, and reported in `StatusReport.volume_storage` when given a `volumeStore`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Upgrade                                                                                   | `tests/sim/upgrade.test.ts` runs a sandbox on a VPS host through the v1.0 upgrade in place: it stays running with the same boot count and keeps its expiry                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| FakeDaemon `FakeHypervisor`                                                               | Boots then pauses on a desired `PAUSED`, resumes without a reboot, runs `instance.snapshot` and carries guest memory to a restored domain, records the network policy without enforcing it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| bcd                                                                                       | The `instances` converger, `instance.snapshot` and `restore_from`, instance `ssh` tunnels, and the `network` policy as host nftables rules (Go unit tests). A host build with `virsh` runs `libvirt-kvm` (362317c): paused warm instances (`virsh suspend`), snapshots as a tar of the flattened disk, the seed and `virsh save`'s memory state, restores that refuse another runtime or CPU model; tested against a fake `virsh` and `qemu-img` only. A host build with the `cloud-hypervisor` binary runs `cloud-hypervisor` (2598f0a): paused warm instances (`vm.pause`), snapshots as a tar of the flattened disk, the seed and cloud-hypervisor's own snapshot of the paused VM, restores with the snapshot's configuration rebuilt from an allowlist around the new instance's disks, taps and console, refusing another runtime, CPU model, or vCPU and memory size; each VMM in a confined `bcd-ch-<id>.service` unit with its taps bridged before it starts (security review 12, fixed in 933c51b); tested against a fake VMM API only. Both runtimes refuse a tenant image or restored snapshot that names a backing file or an external data file. A template's `runtime` is its pool's, so a sandbox runs on `cloud-hypervisor` only in a pool whose runtime is `cloud-hypervisor`. Since 0fca413 it also lands only on a host whose bcd advertised `cloud-hypervisor` in `Hello` (the node label `bc.dev/driver.cloud-hypervisor`), built-in runtimes included: a host without the binary, or under `bcd helper` before 7fe74a1, does not advertise it, and a sandbox with no such host stays `starting`, its instance `Unschedulable` with a message naming the label (`tests/vps-sandboxes.test.ts`). fakehw builds run a file-backed runtime, and a bcx plugin can serve one (none is written; plugin runtimes take no snapshots) |
| Provider pools                                                                            | Not allowed for sandboxes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |

On a host whose bcd runs no instance runtime (no `virsh`, no `cloud-hypervisor`, no runtime plugin), a sandbox's instance is placed but
never reported, so it stays `provisioning` and the sandbox `starting`. Expiry and `delete` still set it to
`deleting`, but a hosted instance finishes deleting only when a complete host report no longer lists it or the host
is declared `lost`.

Not built yet: a run of `libvirt-kvm` or `cloud-hypervisor` on a real KVM host, and MCP tools that
take a snapshot or run code. Exec is the CLI's `sandbox exec` over your own OpenSSH client. The SDK helpers for
commands and files over the SSH session (`sandbox.exec(cmd)`, `files.put`, `files.get`) are deferred: PLAN.md
section 3.7 leaves them for later because they need an SSH client in the SDK, a new dependency decision. See the
[roadmap](/docs/reference/roadmap#sandboxes).

## Lower level: instances directly [#lower-level-instances-directly]

Anything a sandbox does not cover, such as provider pools or a VM without a lifetime, is an instance:

```ts
const vm = await client.instances.create({
  tenantId: 'acme',
  poolId: sandboxPool,
  flavor: 'sbx.small',
  imageId: 'ubuntu-24.04',
  sshKeys: ['ssh-ed25519 AAAA… ci@runner'],
  idempotencyKey: `vm-${jobId}`,
  wait: { timeoutMs: 20_000 },
});
// … and delete it yourself: instances have no time-to-live.
await client.instances.delete({ tenantId: 'acme', instanceId: vm.id });
```

It is decided as `compute:instances:create` and runs in a hosted or a provider pool. See
[instances](/docs/guides/vps/instances).

## Sandboxes and GPU reservations [#sandboxes-and-gpu-reservations]

Both come out of the same pools and the same database, so neither can take what the other holds:

|               | GPU reservation                                                   | Sandbox                                                         |
| ------------- | ----------------------------------------------------------------- | --------------------------------------------------------------- |
| What you get  | GPU atoms on a node, as CDI devices for your own workload manager | A VM with its own kernel, vCPUs, memory, disk and optional GPUs |
| Booked by     | `reservations.create`                                             | `sandboxes.create`                                              |
| Time          | A window (`duration`), extendable                                 | A TTL from the template, extendable up to `maxTtlSeconds`       |
| Who runs code | Your scheduler on the node (Kubernetes, Slurm, containerd)        | The guest, reached with `sandboxes.exec` over SSH               |
| Isolation     | The node's, per workload                                          | A VM per sandbox; nothing of the platform inside                |

  - [Instances](/docs/guides/vps/instances): The lifecycle, paused instances, the sandbox guard, runtimes and workload identity.

  - [Flavors](/docs/guides/vps/flavors): Shapes, scalar claims, overcommit, quota and metering.

  - [Tunnels](/docs/guides/access/tunnels): The single-use tickets and WebSocket that `sandboxes.exec` rides on.

  - [CLI](/docs/guides/clients/cli#sandboxes): The `sandbox` commands, `sandbox exec` included.

  - [Policies](/docs/guides/iam/policies): Bound sandbox shapes and lifetimes with conditions on the draft.


# billing (/docs/reference/api/billing)

> The platform's billing setup for compute: the meters and prices root keeps on the IAM root tenant, so root can bill its sub-tenants for what they use. Every method is decided as a `compute:billing:*` action on `compute-fleet`, which resolves only in the root tenant, so a tenant administrator is turned away before any IAM decision, however broad the tenant's own grants. A root member needs `compute:billing:read` or `compute:billing:manage` on `compute-fleet`.



The platform's billing setup for compute: the meters and prices root keeps on the IAM root tenant, so root can bill its sub-tenants for what they use. Every method is decided as a `compute:billing:*` action on `compute-fleet`, which resolves only in the root tenant, so a tenant administrator is turned away before any IAM decision, however broad the tenant's own grants. A root member needs `compute:billing:read` or `compute:billing:manage` on `compute-fleet`.

These methods call the IAM root tenant's billing API as a root service account. The IAM port must be configured with one: `bridge.embedded(iam, { rootTenantId, serviceCredential })` with `iam:billing:manage` and `iam:billing:read` on the root tenant (a remote bridge takes a `serviceToken`). Without it every method answers `UNSUPPORTED` and names that fix. Amounts are micros of the deployment's currency (1,000,000 micros = 1 unit).

The catalog these methods work from is the same one `better-compute iam setup --meters` and the IAM bridge sync: hosted and node GPU, CPU, memory and volume hours, provider VM meters, reservation-grant and managed-VM meters, the ancillary network, address, disk and snapshot meters, and the `flavor.*` meters root sets on flavors. `vps({ billing: { gpuModels, gpuFamilies } })` names the GPU models whose VM meters exist; name the same models as the bridge so the two agree.

All 7 methods are served by @better-compute/server 0.3.0.

| Method                            | Kind     | Authorization                              | Phase | Status     |
| --------------------------------- | -------- | ------------------------------------------ | ----- | ---------- |
| [`meters`](#meters)               | read     | `compute:billing:read` @ `compute-fleet`   | v1.0  | registered |
| [`syncMeters`](#syncmeters)       | mutation | `compute:billing:manage` @ `compute-fleet` | v1.0  | registered |
| [`prices`](#prices)               | read     | `compute:billing:read` @ `compute-fleet`   | v1.0  | registered |
| [`setPrices`](#setprices)         | mutation | `compute:billing:manage` @ `compute-fleet` | v1.0  | registered |
| [`proposePrices`](#proposeprices) | read     | `compute:billing:read` @ `compute-fleet`   | v1.0  | registered |
| [`margins`](#margins)             | read     | `compute:billing:read` @ `compute-fleet`   | v1.0  | registered |
| [`leaks`](#leaks)                 | read     | `compute:billing:read` @ `compute-fleet`   | v1.0  | registered |

## meters [#meters]

`POST /api/compute/billing/meters`

Returns the catalog compute defines, `catalog`, and `missing`: the keys of the catalog the IAM root tenant has no meter for yet. Usage recorded on a missing meter is refused by IAM; `syncMeters` creates the missing ones.

An archived meter still exists on the root tenant, so it is not listed as missing. `UNSUPPORTED` when the IAM port cannot list meters.

* **Kind:** read
* **Authorization:** `compute:billing:read` @ `compute-fleet`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

Takes an empty object: `{}`.

## syncMeters [#syncmeters]

`POST /api/compute/billing/syncMeters`

Creates the catalog's meters that the root tenant lacks, and returns `{ created, existing }` (meter keys). A meter that exists is left as it is, whatever its name, unit or pricing mode; calling it again creates nothing.

Decided as `compute:billing:manage` on `compute-fleet`. The IAM call happens outside any compute transaction.

* **Kind:** mutation
* **Authorization:** `compute:billing:manage` @ `compute-fleet`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

Takes an empty object: `{}`.

## prices [#prices]

`POST /api/compute/billing/prices`

Returns the rate-card price entries of one meter, or of every rate-card meter, newest first, as `{ meter, targetTenantId, effectiveFrom, spec }`. The list price is the entry whose `targetTenantId` is the root tenant; a negotiated price names the tenant (and its subtree) it applies to. `targetTenantId` filters.

`meter` and `targetTenantId` are optional. `reported` meters carry their cost on each usage event and have no prices.

* **Kind:** read
* **Authorization:** `compute:billing:read` @ `compute-fleet`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                |
| ---------------- | ------ | -------- | ---------------------------------------------------------- |
| `meter`          | string |          | at most 64 characters; matches `^[a-z][a-z0-9_.-]{0,63}$`  |
| `targetTenantId` | string |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## setPrices [#setprices]

`POST /api/compute/billing/setPrices`

Sets up to 100 prices on the root tenant's rate card and returns `{ applied, failed }`.

* **Each price** is `{ meter, targetTenantId?, effectiveFrom, spec }`. Without `targetTenantId` it is the list price; with one it is a negotiated price for that tenant's subtree. `effectiveFrom` is a billing period (`YYYY-MM`), and `spec` is a better-iam price (`per-unit`, `graduated`, `volume` or `package`) in micros.
* **Meters.** Every `meter` must be in the catalog (`meters`) or be a flavor meter; one that is not refuses the whole request with `INVALID_INPUT` before any price is applied.
* **Tenants.** `targetTenantId` is not checked here: better-iam validates it, and a tenant it refuses lands in `failed`.
* **Partial failure.** A price better-iam refuses does not stop the others: `applied` lists the prices that took effect, in request order, and `failed` lists `{ index, meter, targetTenantId?, code, message }` for the rest. Setting a price is idempotent, so a retry of the same request is safe.

Decided as `compute:billing:manage` on `compute-fleet`.

* **Kind:** mutation
* **Authorization:** `compute:billing:manage` @ `compute-fleet`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                                    | Type                                                | Required | Constraints                                                |
| ---------------------------------------- | --------------------------------------------------- | -------- | ---------------------------------------------------------- |
| `prices`                                 | list of object                                      | yes      | 1-100 items                                                |
| `prices[].meter`                         | string                                              | yes      | at most 64 characters; matches `^[a-z][a-z0-9_.-]{0,63}$`  |
| `prices[].targetTenantId`                | string                                              |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `prices[].effectiveFrom`                 | string                                              | yes      | matches `^\d{4}-(0[1-9]\|1[0-2])$`                         |
| `prices[].spec`                          | object                                              | yes      |                                                            |
| `prices[].spec.model`                    | one of `per-unit`, `graduated`, `volume`, `package` | yes      |                                                            |
| `prices[].spec.unitAmountMicros`         | integer                                             |          | at least 0                                                 |
| `prices[].spec.tiers`                    | list of object                                      |          | 0-50 items                                                 |
| `prices[].spec.tiers[].upTo`             | integer,null                                        | yes      | at least 1                                                 |
| `prices[].spec.tiers[].unitAmountMicros` | integer                                             | yes      | at least 0                                                 |
| `prices[].spec.tiers[].flatAmountMicros` | integer                                             |          | at least 0                                                 |
| `prices[].spec.packageSize`              | integer                                             |          | at least 1                                                 |
| `prices[].spec.packageAmountMicros`      | integer                                             |          | at least 0                                                 |
| `prices[].spec.includedQuantity`         | number                                              |          | at least 0                                                 |
| `prices[].spec.minimumAmountMicros`      | integer                                             |          | at least 0                                                 |
| `prices[].spec.maximumAmountMicros`      | integer                                             |          | at least 0                                                 |

## proposePrices [#proposeprices]

`POST /api/compute/billing/proposePrices`

Reads the list prices of one of the platform's provider accounts, adds a margin and returns prices root can publish. It sets nothing: `prices` is the same before and after, and root publishes the proposals it wants through `setPrices`. Decided as `compute:billing:read` on `compute-fleet`. It needs no billing member on the IAM port, only the provider.

* **Input.** `accountId` is a provider account of the root tenant whose provider lists prices (Compute Engine reads the Cloud Billing Catalog): another tenant's account, or one that does not exist, is `NOT_FOUND`, and a provider without prices is `UNSUPPORTED`. `marginPct` (0 to 1000) is the percentage added. `region` limits the prices read to one region; without it the cheapest basis across the regions the provider lists is used. `effectiveFrom` (`YYYY-MM`) is the period root means to publish the proposals from; it is checked and left to `setPrices`.

* **Result.** `{ proposals, unpriced, currency }`; `currency` is the deployment currency every proposal is in. A proposal is `{ meter, spec, basis }`: `spec` is `per-unit` at `round(unitPriceMicros x (1 + marginPct / 100))` micros, and `basis` lists the provider prices (`sku`, `unitPriceMicros`) it came from. A meter the provider's prices cover is proposed; the provider meters that remain are `unpriced`.

* **`vm.vcpu.hours` and `vm.memory.gib.hours`** take the cheapest general-purpose machine family (E2, N1, N2, N2D, N4, T2D, C3, C3D, C4, by one vCPU and four GiB of on-demand prices) in the region. Per-family accuracy is what flavor meters are for.

* **GPU meters** (`vm.gpu.<model>.hours` and the `.spot` meters) take the cheapest GPU SKU of that model, on demand or Spot. **Ancillary meters** (`ip.public.hours`, `disk.<type>.gib.hours`) take the matching SKU; monthly storage prices become GiB-hours at 730 hours a month.

* **Flavor meters** (`flavor.<name>.hours`) are composed for the flavors of the account's pools that carry them: the flavor's vCPUs, memory and GPUs at the prices of its offer's machine family in the offer's region (an offer id spelled `zone/machine-type`). Archived flavors are ignored. A flavor's GPU model is its pool's `gpuModel` as the deployment's GPU models name it (`vps({ billing: { gpuModels } })`, as metering does), so a model the deployment does not name is `other` and has no price. Flavors that share a meter must share a shape, else the meter is `unpriced`.

* **Left `unpriced`:** instance, reserved, reservation-grant and managed-VM meters, network and snapshot meters, GPU models the provider has no price for, and any meter whose price is in another currency than the deployment's (`vps({ billing: { currency } })`, default `USD`). Hosted, node and sandbox meters are the platform's own and are not listed.

* **Kind:** read

* **Authorization:** `compute:billing:read` @ `compute-fleet`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                        |
| --------------- | ------ | -------- | ------------------------------------------------------------------ |
| `accountId`     | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `marginPct`     | number | yes      | 0-1000                                                             |
| `region`        | string |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`       |
| `effectiveFrom` | string | yes      | matches `^\d{4}-(0[1-9]\|1[0-2])$`                                 |

## margins [#margins]

`POST /api/compute/billing/margins`

Returns the platform's margin over a range: revenue (what better-iam billed) minus provider cost (what the providers billed the platform, D12), grouped by `tenant`, `pool`, `account` (provider account) or `meter`, as `{ rows, unattributedCostMicros, currency, from, to }`.

* **Input.** `from` and `to` are epoch milliseconds (`to` after `from`, at most 400 days apart); `groupBy` is required.
* **Revenue** comes from the IAM port's spend report on the root tenant (`IamPort.spend`, by tenant, by meter, or by the `pool` tag). better-iam sums whole billing periods, so the answer covers every period the range touches, and costs are summed over exactly that range: `from` and `to` in the answer are the range both cover. Without `spend`, `revenueMicros`, `marginMicros` and `marginPct` are `null` and costs cover the query's range. better-iam has no provider-account dimension, so `groupBy: 'account'` always has `null` revenue.
* **Revenue is before contract terms.** It is better-iam's spend: usage rated at list or negotiated prices (`setPrices`), before the contract discounts of `setTerms`, which better-iam applies when it closes the period into a statement. Margins of a discounted tenant are overstated by its discount.
* **Root's own usage is not revenue.** better-iam never invoices the platform. By tenant, the root tenant's row keeps its provider cost and has `revenueMicros: 0`, so its margin is minus that cost. By pool or meter, the spend report cannot be scoped to the root tenant without the tenants below it, so pool and meter revenue include root's internal usage at its rated price.
* **Cost** is the sum of cost plus credits of the `vps.costs` worker's rows (`provider_costs`) of the platform's own provider accounts (the root tenant's; a tenant's own account is that tenant's cost, not the platform's) whose usage started in the range, in the deployment currency (`vps({ billing: { currency } })`, default `USD`; rows in another currency are excluded). Rows attributed to no tenant are not in `rows` but in `unattributedCostMicros`. By pool, a row counts toward its instance's pool; by meter, toward the reported meter it was passed through on. Attributed cost with no such pool or meter is grouped under the key `(none)`, whose revenue is `null`. Provider compute cost has no meter dimension (a provider bills a VM's cores and memory, not compute's meters), so with `groupBy: 'meter'` all compute cost is under `(none)` while its revenue is under the `vm.*` meters: read the meter view for pass-through margins, and the tenant or pool view for compute. Pass-through usage carries the instance's `pool` tag, so revenue by pool includes it.
* **Rows.** One per key that has revenue or cost, sorted by key with `(none)` last: `{ key, revenueMicros, costMicros, marginMicros, marginPct }`. `marginPct` is the margin over revenue in percent (2 decimals), `null` when revenue is not positive.

Decided as `compute:billing:read` on `compute-fleet`, so a tenant cannot read it. The IAM call happens outside any compute transaction.

**Leak detection** (reported as events, listed by `leaks`) — at the end of every `vps.costs` pass over a platform provider account, each instance's provider-billed compute is compared with what compute metered for it, per UTC day. The provider hours of an instance-day are the **distinct hour slots** the provider billed compute for it (regular rows of the provider's compute SKUs attributed to the instance, quantity above 0), not the sum of the rows' quantities: a provider bills a VM's cores and memory as separate rows in vCPU-hours and GiB-hours, so summed quantities are no count of hours. The metered hours are how many of those slots a `running` usage interval of the instance overlaps. When the provider hours exceed the metered ones by more than `vps({ costs: { leakToleranceHours } })` (default 1), `billing.leak` is published, at most once per instance and day. Instances with `billing: none` and tenants' own accounts are not checked.

A provider whose cost rows are period aggregates (`billing.periodRows`: a DigitalOcean invoice item is a Droplet's month so far) is compared differently: the row's billed hours against the instance's metered hours over the row's span, with a gap over `max(leakToleranceHours, 5%)` of the billed hours, raised once per instance and period; a provider that bills stopped machines (`billing.billsStopped`) has the instance's stopped hours counted as metered.

* **Kind:** read
* **Authorization:** `compute:billing:read` @ `compute-fleet`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field     | Type                                        | Required | Constraints |
| --------- | ------------------------------------------- | -------- | ----------- |
| `from`    | integer                                     | yes      | at least 0  |
| `to`      | integer                                     | yes      | at least 1  |
| `groupBy` | one of `tenant`, `pool`, `account`, `meter` | yes      |             |

## leaks [#leaks]

`POST /api/compute/billing/leaks`

Lists the leaks raised on the platform's (root's) provider accounts whose day is in `[from, to)`, newest day first, at most 500: `{ items }`.

* **Input.** `from` and `to` are epoch milliseconds; a leak's day is its UTC day (for a period row, the period's first day).

* **Items.** `{ accountId, accountName, provider, instanceId, tenantId, day, providerHours, meteredHours, raisedAt }`: `tenantId` is the instance's tenant, `null` once the instance is gone; `providerHours` is what the provider billed and `meteredHours` what compute metered. A tenant's own accounts are never checked, so they have no leaks; without a root tenant the list is empty.

* **IAM.** Decided as `compute:billing:read` on `compute-fleet`, so a tenant cannot read it. A read of the database: no provider is called.

* **Kind:** read

* **Authorization:** `compute:billing:read` @ `compute-fleet`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field  | Type    | Required | Constraints |
| ------ | ------- | -------- | ----------- |
| `from` | integer | yes      | at least 0  |
| `to`   | integer | yes      | at least 1  |


# capacity (/docs/reference/api/capacity)

> The capacity methods report what the pools a tenant can see hold, book and have free, without booking anything. `summary` and `byPool` read the per-pool rollup, which the booking transactions update and a periodic rebuild corrects. `preview` asks the placer whether a request fits, and `timeline` counts booked GPUs over time. All four are decided as `compute:capacity:read`. See [Capacity](/docs/guides/capacity).



The capacity methods report what the pools a tenant can see hold, book and have free, without booking anything. `summary` and `byPool` read the per-pool rollup, which the booking transactions update and a periodic rebuild corrects. `preview` asks the placer whether a request fits, and `timeline` counts booked GPUs over time. All four are decided as `compute:capacity:read`. See [Capacity](/docs/guides/capacity).

All 4 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind | Authorization                              | Phase | Status     |
| ----------------------- | ---- | ------------------------------------------ | ----- | ---------- |
| [`summary`](#summary)   | read | `compute:capacity:read` @ `compute-tenant` | v0.1  | registered |
| [`byPool`](#bypool)     | read | `compute:capacity:read` @ `compute-pool`   | v0.2  | registered |
| [`preview`](#preview)   | read | `compute:capacity:read` @ `compute-pool`   | v0.2  | registered |
| [`timeline`](#timeline) | read | `compute:capacity:read` @ `compute-pool`   | v0.2  | registered |

## summary [#summary]

`POST /api/compute/capacity/summary`

Returns every pool the tenant owns or is offered, each with its rollup rows, plus whole-GPU totals per GPU model.

Archived pools appear only for their owner. `byGpuModel` sums only each pool's `gpu` row (`total`, `allocatable`, `allocated`, `reserved`, `available`), and a pool without a GPU model is counted under `other`. The `gpu.mig`, `gpu.slot` and scalar rows are in `pools[].capacity`. The decision is made on the tenant.

* **Kind:** read
* **Authorization:** `compute:capacity:read` @ `compute-tenant`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## byPool [#bypool]

`POST /api/compute/capacity/byPool`

Returns one pool's rollup rows, one per resource, ordered by resource name.

Each row carries `total`, `allocatable`, `allocated`, `reserved`, `available`, the `provisioning`, `unhealthy`, `maintenance` and `unreachable` buckets, and `updatedAt`. `available` is conservative: an atom booked now and again later counts in both `allocated` and `reserved`. The call is `NOT_FOUND` when the pool is not owned by or offered to the tenant, or is archived and the tenant is not its owner. `pools.capacity` returns the same rows.

* **Kind:** read
* **Authorization:** `compute:capacity:read` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## preview [#preview]

`POST /api/compute/capacity/preview`

Runs the placer on a reservation request without booking anything and returns whether it fits now, the candidate nodes with the atoms each would take, and a reason when it does not fit.

* The input is the `reservations.create` input, validated the same way, `idempotencyKey` included. Nothing is written and no quota is checked.

* Up to 10 ranked candidates are returned, each with `nodeId`, `atoms` and `score`. With `gang`, the candidates are the members of the best gang. With `binding: 'node'`, they are the whole nodes a lease would take.

* When a regular (not `spot`) request does not fit, `preemptible` says how many spot reservations would get a notice and when the request would start. It gives a count only, never the other reservations' ids.

* `NOT_FOUND` when the pool is not visible to the tenant.

* **Kind:** read

* **Authorization:** `compute:capacity:read` @ `compute-pool`

* **Module:** `capacity`

* **Phase:** v0.2 (registered)

**Input**

| Field                    | Type                                                          | Required | Constraints                                                       |
| ------------------------ | ------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`               | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`                 | string                                                        | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`              | string                                                        |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `devices`                | list of object                                                | yes      | 1-8 items                                                         |
| `devices[].class`        | one of `gpu`, `gpu.mig`, `gpu.slot`, `nic.vf`, `disk`, `node` | yes      |                                                                   |
| `devices[].count`        | integer                                                       | yes      | 1-64                                                              |
| `devices[].selector`     | object                                                        |          |                                                                   |
| `devices[].topology`     | one of `any`, `nvlink-all`                                    |          |                                                                   |
| `devices[].sharing`      | one of `exclusive`, `mig`, `timeslice`, `mps`                 |          |                                                                   |
| `topology`               | one of `any`, `nvlink-all`                                    |          |                                                                   |
| `start`                  | integer                                                       |          | 0-140737488355328                                                 |
| `duration`               | string,integer                                                | yes      |                                                                   |
| `binding`                | string                                                        |          | default `"hold"`                                                  |
| `resources`              | object                                                        |          |                                                                   |
| `resources.vcpu`         | integer                                                       |          | 0-4096                                                            |
| `resources.memoryGiB`    | integer                                                       |          | 0-65536                                                           |
| `resources.hugepages1Gi` | integer                                                       |          | 0-4096                                                            |
| `node`                   | object                                                        |          |                                                                   |
| `node.profileId`         | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:/-]{1,128}$`       |
| `node.profileVersion`    | integer                                                       |          | at least 1                                                        |
| `leaseSeconds`           | integer                                                       |          | 30-86400                                                          |
| `spot`                   | boolean                                                       |          |                                                                   |
| `idempotencyKey`         | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `wait`                   | object                                                        |          |                                                                   |
| `wait.timeoutMs`         | integer                                                       | yes      | 0-30000                                                           |
| `gang`                   | object                                                        |          |                                                                   |
| `gang.size`              | integer                                                       | yes      | 2-64                                                              |
| `gang.topologyKey`       | string                                                        |          | at most 128 characters                                            |

## timeline [#timeline]

`POST /api/compute/capacity/timeline`

Returns, for one pool, the number of whole GPUs booked in each time bucket and how many of the pool's allocatable GPUs remain free.

`from` defaults to the start of the current hour, `to` to 24 hours after `from`, and `stepMinutes` to 60. A timeline spans at most 7 days and 336 buckets, and `to` must be after `from`; otherwise the call is `INVALID_INPUT`. `allocatable` comes from the pool's current `gpu` rollup row, and each bucket's `free` is `allocatable` minus `booked`, never below 0. MIG and slot atoms are not counted. `NOT_FOUND` when the pool is not visible to the tenant.

* **Kind:** read
* **Authorization:** `compute:capacity:read` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field         | Type    | Required | Constraints                                                       |
| ------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`    | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`      | string  | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `from`        | integer |          | 0-140737488355328                                                 |
| `to`          | integer |          | 0-140737488355328                                                 |
| `stepMinutes` | integer |          | 5-1440                                                            |


# certificates (/docs/reference/api/certificates)

> TLS certificates on a provider account (DigitalOcean first), for the tenant's [load balancers](/docs/reference/api/load-balancers). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.certificates`. Reading takes `compute:certificates:read`, everything else `compute:certificates:manage`, on `provider-certificate/{id}`. Certificates book no quota (at most 100 per tenant) and do not bill.



TLS certificates on a provider account (DigitalOcean first), for the tenant's [load balancers](/docs/reference/api/load-balancers). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.certificates`. Reading takes `compute:certificates:read`, everything else `compute:certificates:manage`, on `provider-certificate/{id}`. Certificates book no quota (at most 100 per tenant) and do not bill.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                                  | Phase | Status     |
| ------------------- | -------- | -------------------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:certificates:manage` @ `compute-provider-certificate` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:certificates:read` @ `compute-tenant` + items         | v1.0  | registered |
| [`get`](#get)       | read     | `compute:certificates:read` @ `compute-provider-certificate`   | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:certificates:manage` @ `compute-provider-certificate` | v1.0  | registered |

## create [#create]

`POST /api/compute/certificates/create`

Makes a certificate and returns it `pending`. `kind: 'managed'` has the provider issue and renew it for `domains`, each of which must be in one of the tenant's own [DNS zones](/docs/reference/api/dns-zones) on the same provider account (`INVALID_INPUT`, reason `domainNotOwned`): the provider issues certificates for any domain the account hosts, which may be another tenant's. `kind: 'custom'` uploads the tenant's own PEMs: `certificate`, `privateKey` and optionally `chain` are references to the tenant's own secrets, `secret://<tenantId>/<id>`, and any other scheme or tenant is `INVALID_INPUT`. The row keeps only the references; they are resolved in the call that uploads the PEMs, by a resolver that refuses every reference the certificate does not name, and never appear in a view, event or log. The name is unique in the tenant (`CONFLICT`); the same `idempotencyKey` returns the same certificate. Publishes `providerCertificate.created`, then `providerCertificate.ready` or `providerCertificate.error`.

* **Kind:** mutation
* **Authorization:** `compute:certificates:manage` @ `compute-provider-certificate`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                       | Required | Constraints                                                                                    |
| ---------------- | -------------------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `tenantId`       | string                     | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                     |
| `projectId`      | string                     |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`                               |
| `poolId`         | string                     | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`                              |
| `name`           | string                     | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$`                             |
| `kind`           | one of `managed`, `custom` | yes      |                                                                                                |
| `domains`        | list of string             |          | 1-100 items                                                                                    |
| `certificate`    | string                     |          | at most 270 characters; matches `^secret:\/\/([A-Za-z0-9._:-]{1,128})\/[A-Za-z0-9._-]{1,128}$` |
| `privateKey`     | string                     |          | at most 270 characters; matches `^secret:\/\/([A-Za-z0-9._:-]{1,128})\/[A-Za-z0-9._-]{1,128}$` |
| `chain`          | string                     |          | at most 270 characters; matches `^secret:\/\/([A-Za-z0-9._:-]{1,128})\/[A-Za-z0-9._-]{1,128}$` |
| `idempotencyKey` | string                     | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                     |

## list [#list]

`POST /api/compute/certificates/list`

Returns the tenant's certificates in the order they were made (at most 1000), optionally of one pool or project. Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:certificates:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/certificates/get`

Returns one certificate: its kind, names, state and, once issued or uploaded, `notAfter` and `fingerprint`.

* **Kind:** read
* **Authorization:** `compute:certificates:read` @ `compute-provider-certificate`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                        |
| --------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `certificateId` | string | yes      | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/certificates/delete`

Marks the certificate for deletion and returns it `deleting`; the reconciler deletes it at the provider and publishes `providerCertificate.deleted`. A certificate a load balancer of the tenant still names is `INVALID_STATE` (reason `inUse`): change or delete that load balancer first. A certificate is immutable; to change one, make another and point the load balancer at it.

* **Kind:** mutation
* **Authorization:** `compute:certificates:manage` @ `compute-provider-certificate`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                        |
| --------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `certificateId` | string | yes      | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |


# cloudResources (/docs/reference/api/cloud-resources)

> Cloud resources are a provider's managed data and platform services (managed databases and what lives in them, App Platform apps, Functions, Marketplace add-ons, partner attachments), served by the `@better-compute/cloud-resources` module through the provider-neutral `ProviderResources` contract. One table, one reconciler and these calls serve every kind a provider lists: DigitalOcean today (`do.database.pg`, `do.app`, …), AWS and GCP kinds later under their own prefixes. A resource belongs to one provider pool, so to that pool's account. Two hosting models use the same calls: a platform account's pool offered to the tenant (the resource lives on the platform's account, which other tenants share), or a pool on the tenant's own account. The calls record what the tenant wants and return at once; the `cloud.resources` reconciler then creates the resource at the provider, applies changes in place and deletes what the tenant deleted. `state` shows where it is: `pending`, `provisioning`, `ready`, `updating`, `deleting`, `deleted`, `released` or `error` (with a `message`). Resources still converging are looked at every 30 seconds; ready and failed ones are refreshed every five minutes, which retries an error and picks up what changed at the provider.



Cloud resources are a provider's managed data and platform services (managed databases and what lives in them, App Platform apps, Functions, Marketplace add-ons, partner attachments), served by the `@better-compute/cloud-resources` module through the provider-neutral `ProviderResources` contract. One table, one reconciler and these calls serve every kind a provider lists: DigitalOcean today (`do.database.pg`, `do.app`, …), AWS and GCP kinds later under their own prefixes. A resource belongs to one provider pool, so to that pool's account. Two hosting models use the same calls: a platform account's pool offered to the tenant (the resource lives on the platform's account, which other tenants share), or a pool on the tenant's own account. The calls record what the tenant wants and return at once; the `cloud.resources` reconciler then creates the resource at the provider, applies changes in place and deletes what the tenant deleted. `state` shows where it is: `pending`, `provisioning`, `ready`, `updating`, `deleting`, `deleted`, `released` or `error` (with a `message`). Resources still converging are looked at every 30 seconds; ready and failed ones are refreshed every five minutes, which retries an error and picks up what changed at the provider.

A resource's provider-side name is `bc-r-` and 24 characters of its id, and it carries the ownership labels `better-compute-managed`, `better-compute-resource` and `better-compute-tenant` (as tags where the provider has no labels). Secret outputs (a database password, a connection URI that embeds it, a namespace key) are sealed into the secret store in the reconciler run that received them; a resource returns only their `SecretRef`s in `secretOutputs`, and the values never reach a row, an event or a log. A resource bills `cloud.resource.hours` (one per hour, to the tenant) from the moment the provider first accepted it (an adopted one from its first `ready`) until it is deleted or released, on pools whose billing mode is `metered` or `platform-fee` (a resource living in another, such as a database user, pool, firewall or replica, bills no fee of its own); a kind's own meters bill only when the operator names them in `cloudResources({ kindMeters })`.

Every call is decided on `compute-cloud-resource/{id}` (or the tenant, for listings) with the attributes `kind`, `family` (`data`, `platform`, `ai`, `storage`, `network`, `edge`, `observability`), `provider`, `providerAccountId`, `poolId`, `region` and `projectId`. One action set serves every family; a policy narrows it by attribute: `StringEquals: { 'resource.family': 'data' }` allows a team managed databases without allowing App Platform.

All 11 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                               | Phase | Status     |
| ----------------------- | -------- | ----------------------------------------------------------- | ----- | ---------- |
| [`kinds`](#kinds)       | read     | `compute:cloud-resources:read` @ `compute-tenant`           | v1.0  | registered |
| [`catalog`](#catalog)   | read     | `compute:cloud-resources:read` @ `compute-tenant`           | v1.0  | registered |
| [`signals`](#signals)   | read     | `compute:cloud-resources:read` @ `compute-tenant`           | v1.0  | registered |
| [`create`](#create)     | mutation | `compute:cloud-resources:create` @ `compute-cloud-resource` | v1.0  | registered |
| [`list`](#list)         | read     | `compute:cloud-resources:read` @ `compute-tenant` + items   | v1.0  | registered |
| [`get`](#get)           | read     | `compute:cloud-resources:read` @ `compute-cloud-resource`   | v1.0  | registered |
| [`update`](#update)     | mutation | `compute:cloud-resources:update` @ `compute-cloud-resource` | v1.0  | registered |
| [`delete`](#delete)     | mutation | `compute:cloud-resources:delete` @ `compute-cloud-resource` | v1.0  | registered |
| [`discover`](#discover) | read     | `compute:cloud-resources:read` @ `compute-tenant`           | v1.0  | registered |
| [`adopt`](#adopt)       | mutation | `compute:cloud-resources:create` @ `compute-cloud-resource` | v1.0  | registered |
| [`release`](#release)   | mutation | `compute:cloud-resources:delete` @ `compute-cloud-resource` | v1.0  | registered |

## kinds [#kinds]

`POST /api/compute/cloudResources/kinds`

Returns the kinds the pool's provider serves: for each its `family`, the JSON Schema of its `spec`, the plain `outputs` and `secretOutputs` it reports, whether it is `regional`, which spec fields are `updatable` in place, its `meters`, its `parent` kind (a database user lives in a cluster), whether it is a provider `preview` and `available` on this account (previews need `previews: true` on the account), and the tenant's `quota` of it. A pool that is not a provider pool, or whose provider serves no cloud resources, is `UNSUPPORTED` (`NOT_FOUND` for a pool the tenant cannot see). Decided as `compute:cloud-resources:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-tenant`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## catalog [#catalog]

`POST /api/compute/cloudResources/catalog`

Returns the installable applications the pool's provider lists (DigitalOcean's 1-Click apps), at most 1000, optionally only those with `installsOn` `instance` (the slug is the image of an instance create) or `kubernetes` (an add-on of a managed cluster). Read-only: they are installed through those surfaces, never through `create`. The provider is asked with no transaction open. A provider without a catalog is `UNSUPPORTED`. Decided as `compute:cloud-resources:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-tenant`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type                            | Required | Constraints                                                       |
| ------------ | ------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`     | string                          | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `installsOn` | one of `instance`, `kubernetes` |          |                                                                   |

## signals [#signals]

`POST /api/compute/cloudResources/signals`

Returns the read-only signals the pool's provider raises about its resources (DigitalOcean's Managed Agents Signals: a run that failed, an agent asking for a human), newest first, at most 500, for an attention list. Each has the provider's `id`, a `severity` (`critical`, `warning` or `info`), a one-line `title`, an optional `detail` (one line, at most 500 characters), `at` (epoch ms), the `kind` and `providerId` of the resource it is about, and the `resourceId` of the cloud resource that records it, if any; `resourceId` in the input narrows it to that resource. A tenant of a shared pool sees only the signals about its own resources; the pool's owner (the root tenant for the platform's pool, a tenant on its own account) sees every signal, recorded or not. The provider is asked with no transaction open and sends no mutation; malformed signals are dropped. One answer of the provider account is reused for 30 seconds, whichever tenant asked. A provider without signals is `UNSUPPORTED`, as is a provider that holds them behind previews on an account without `previews: true` (`details.reason` `preview`). Decided as `compute:cloud-resources:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-tenant`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`     | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `resourceId` | string |          | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## create [#create]

`POST /api/compute/cloudResources/create`

Records a resource and returns it `pending`.

* The pool must be visible to the tenant and a `provider` pool whose provider serves the `kind`; otherwise `UNSUPPORTED` (`NOT_FOUND` for a pool the tenant cannot see). A preview kind on an account without previews, or a kind with secret outputs on a deployment without a secret store, is `UNSUPPORTED`.
* `spec` (at most 64 KiB of JSON) is validated by the kind's manifest (`INVALID_INPUT`, with the field under `spec.`). A regional kind's `spec.region` must be a region the account configures. A secret in a spec is a `SecretRef` to the tenant's own secret (`secret://<tenant>/…`), resolved only just before the provider call.
* On a shared (hosted) account, a spec that reaches account-wide objects (a VPC, a container registry, linked repositories, a database cluster by name, Droplet tags or ids as trusted sources) is `INVALID_INPUT` with `details.reason` `accountWide` and the fields; on the tenant's own account it may. A spec may instead name the tenant's instances (a database's trusted `instances`): each must be a live instance of the tenant on the pool's account, otherwise `INVALID_INPUT` with `details.reason` `instance` (the same for one that does not exist). A spec may also name the tenant's networks (a database cluster's `network`, which joins that network's VPC in the cluster's region): each must be a live network of the tenant on the pool's account, otherwise `INVALID_INPUT` with `details.reason` `network` and the `networks`; the reconciler checks them again before every provider call.
* A kind with a parent names it in `parentId`: a resource of the parent kind on the same account, not being deleted. A kind that declares an identity within its parent (a database user's name; a cluster's single firewall) allows one live resource of it per parent: a second is `CONFLICT` with `details.reason` `identity` and the `resourceId` that has it.
* `name` is 1 to 63 characters, `[a-z]([-a-z0-9]*[a-z0-9])?`, unique per kind among the tenant's resources that are not deleted or released (`CONFLICT` otherwise).
* A tenant that already has its quota of the kind (the module's `maxPerKind`, default 20, or a per-kind `quotas` entry; one being deleted still counts) is `QUOTA_EXCEEDED`, with the cap in `details.limit`.
* A kind that books capacity quota (a DigitalOcean Dedicated Inference endpoint books its GPUs: GPUs per replica times the larger of `replicas` and `maxReplicas`) books it against the tenant's quota on the pool in the same transaction that records the resource. Over the quota is `QUOTA_EXCEEDED` and nothing is recorded.
* Decided as `compute:cloud-resources:create` on the resource it would create, with its kind and family as attributes. A refusal leaves nothing behind. After the decision the tenant's offer is checked (a suspended offer or an excluded region refuses), and the tenant's budgets unless the pool's billing mode is `none`.

The resource id derives from the tenant, the caller and `idempotencyKey`. A retry with the same key returns the same resource; the same key with any other field changed is `CONFLICT`. The caller owns the resource. Publishes `cloud.resource.created`, then `cloud.resource.ready` the first time it is ready (with the names of its outputs and secret outputs) or `cloud.resource.failed`.

* **Kind:** mutation
* **Authorization:** `compute:cloud-resources:create` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`      | string |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`         | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `kind`           | string | yes      | at most 64 characters; matches `^[a-z0-9]+(?:[.-][a-z0-9]+)*$`     |
| `name`           | string | yes      | at most 63 characters; matches `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` |
| `description`    | string |          | at most 256 characters                                             |
| `parentId`       | string |          | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `spec`           | object | yes      |                                                                    |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/cloudResources/list`

Returns the tenant's resources, oldest first; `poolId`, `projectId`, `kind` and `family` narrow it, and `includeDeleted: true` adds deleted and released ones. At most 500 are returned. Decided as `compute:cloud-resources:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-tenant` + items
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                                                                           | Required | Constraints                                                       |
| ---------------- | ------------------------------------------------------------------------------ | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string                                                                         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string                                                                         |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string                                                                         |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `kind`           | string                                                                         |          | at most 64 characters; matches `^[a-z0-9]+(?:[.-][a-z0-9]+)*$`    |
| `family`         | one of `storage`, `network`, `edge`, `data`, `platform`, `ai`, `observability` |          |                                                                   |
| `includeDeleted` | boolean                                                                        |          | default `false`                                                   |

## get [#get]

`POST /api/compute/cloudResources/get`

Returns one resource with its spec, `providerId` once the provider accepted it, `outputs`, `secretOutputs` (SecretRefs only) and state. A resource of another tenant is `NOT_FOUND` here, and is refused by the decision (`ACCESS_DENIED`) before that. Decided as `compute:cloud-resources:read` on the resource.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                      |
| ------------ | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `resourceId` | string | yes      | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/cloudResources/update`

Merges `spec` onto the stored spec and validates the result with the kind's manifest; `description` is better-compute's alone. A change to the kind's `updatable` fields is applied in place: a `ready` or failed resource goes `updating` and back to `ready` (`cloud.resource.updating`, then `cloud.resource.updated`). A change to any other field is `INVALID_STATE` with `details.reason` `replaceRequired` and the fields: create a new resource instead. The account-wide rule of `create` applies to the merged spec, and an instance or network the change newly names is checked as on `create`. For a kind that books quota, the booking follows the spec: growth is charged in the same transaction (`QUOTA_EXCEEDED` refuses the update and changes nothing); a shrink is returned once the provider reports the new spec applied (the resource is `ready` again), since until then the old amount still runs. An adopted resource's spec cannot change before its takeover completed (`INVALID_STATE`, `adopting`); its description can. A resource being deleted or released is `INVALID_STATE`. Every update bumps `version`. Decided as `compute:cloud-resources:update` on the resource.

* **Kind:** mutation
* **Authorization:** `compute:cloud-resources:update` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field         | Type        | Required | Constraints                                                      |
| ------------- | ----------- | -------- | ---------------------------------------------------------------- |
| `tenantId`    | string      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `resourceId`  | string      | yes      | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `spec`        | object      |          |                                                                  |
| `description` | string,null |          | at most 256 characters                                           |

## delete [#delete]

`POST /api/compute/cloudResources/delete`

Marks the resource and every resource living in it for deletion and returns it `deleting`. The reconciler deletes the children first (a child created while the parent was being deleted is deleted too), then the resource at the provider; it is then `deleted`, its authorization row is removed, its sealed secrets are destroyed, the quota its spec booked is returned (once) and `cloud.resource.deleted` is published. A resource the provider never accepted is deleted without a provider call. A resource this deployment does not manage at the provider is never deleted there: it stays in `error`. An adopted resource whose takeover never completed is let go instead (`released`, with no provider call). Deleting a resource that is being deleted returns it unchanged. Deleting one that is being released overrides the release: it is deleted instead (should the provider already have let it go, the delete is refused there as not managed and nothing is destroyed). Publishes `cloud.resource.deleting`. Decided as `compute:cloud-resources:delete` on the resource.

* **Kind:** mutation
* **Authorization:** `compute:cloud-resources:delete` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                      |
| ------------ | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `resourceId` | string | yes      | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$` |

## discover [#discover]

`POST /api/compute/cloudResources/discover`

Lists the kind's resources on the tenant's own account (at most 500): each with its `providerId`, `name`, `region`, `spec` as the provider reports it, `managedBy` (`this` deployment, an `other` one, or absent: none) and the `resourceId` of the cloud resource that records it, if any. A pool on an account of another tenant (a platform pool offered to the tenant) is `ACCESS_DENIED` with `details.reason` `notAccountOwner`. The provider is asked with no transaction open. Decided as `compute:cloud-resources:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:cloud-resources:read` @ `compute-tenant`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `kind`     | string | yes      | at most 64 characters; matches `^[a-z0-9]+(?:[.-][a-z0-9]+)*$`    |

## adopt [#adopt]

`POST /api/compute/cloudResources/adopt`

Takes over an unmanaged resource of the tenant's own account and returns it `pending`, `adopted: true`, with an empty spec. Before anything is recorded the provider is asked for that one resource: one that does not exist is `NOT_FOUND`, one another deployment manages is `INVALID_STATE` with `details.reason` `managedByAnother`. A kind that lives in a parent cannot be adopted (adopt the parent and create its children); a kind that books quota (a Dedicated Inference endpoint) is `UNSUPPORTED` with `details.reason` `quotaBooked`, since nothing was charged for what it runs on (create it through better-compute instead); and a resource already recorded, or a name taken, is `CONFLICT`. The reconciler writes the ownership labels in its first run (the provider refuses there too a resource another deployment took meanwhile) and records the spec the provider reports; from then on it is observed until the tenant changes its spec, then managed like any other. A platform account is `ACCESS_DENIED` as for `discover`. Publishes `cloud.resource.adopted`. Decided as `compute:cloud-resources:create`, as for `create`.

* **Kind:** mutation
* **Authorization:** `compute:cloud-resources:create` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`      | string |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`         | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `kind`           | string | yes      | at most 64 characters; matches `^[a-z0-9]+(?:[.-][a-z0-9]+)*$`     |
| `providerId`     | string | yes      | at most 200 characters                                             |
| `name`           | string | yes      | at most 63 characters; matches `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` |
| `description`    | string |          | at most 256 characters                                             |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## release [#release]

`POST /api/compute/cloudResources/release`

Lets a resource go: the reconciler removes its ownership labels at the provider and leaves it running, and the resource is `released` (its authorization row removed, its sealed secrets destroyed, its quota booking returned, `cloud.resource.released` published). A resource the provider has not accepted yet is `INVALID_STATE` (delete it instead), as is one that lives in a parent (release that) or one that still has live children. A kind the provider cannot release (one owned by its name rather than by labels, such as a DigitalOcean Functions namespace or model access key) is `UNSUPPORTED` with `details.reason` `release`, and nothing is recorded: keep it or delete it. Should a provider refuse only when the reconciler asks, the release of a resource better-compute created is withdrawn (it stays managed, in `error` with the reason until the next refresh), and an adopted one is let go without the provider call. Publishes `cloud.resource.releasing`. Decided as `compute:cloud-resources:delete` on the resource.

* **Kind:** mutation
* **Authorization:** `compute:cloud-resources:delete` @ `compute-cloud-resource`
* **Module:** `cloud-resources`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                      |
| ------------ | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `resourceId` | string | yes      | at most 56 characters; matches `^crs_[0-9a-hjkmnp-tv-z]{26,52}$` |


# commandBatches (/docs/reference/api/command-batches)

> Destructive-command batches (v1.0 quorum). A batch fixes a set of (subject, kind) leaves, optionally bound to exact payloads, under one Merkle root when it is created (modules create them through `commands.createBatch`). When the deployment publishes approver keys (`pki.approvers`), the batch's commands wait until enough approvers, holding keys off the control plane, have signed its root for the trust domain. The gateway then sends each command with the approval and its leaf's proof, and the daemon verifies both. Without approver keys a batch is recorded for audit and its commands go out at once. See [Security](/docs/operations/security).



Destructive-command batches (v1.0 quorum). A batch fixes a set of (subject, kind) leaves, optionally bound to exact payloads, under one Merkle root when it is created (modules create them through `commands.createBatch`). When the deployment publishes approver keys (`pki.approvers`), the batch's commands wait until enough approvers, holding keys off the control plane, have signed its root for the trust domain. The gateway then sends each command with the approval and its leaf's proof, and the daemon verifies both. Without approver keys a batch is recorded for audit and its commands go out at once. See [Security](/docs/operations/security).

All 7 methods are served by @better-compute/server 0.3.0.

| Method                | Kind     | Authorization                                | Phase | Status     |
| --------------------- | -------- | -------------------------------------------- | ----- | ---------- |
| [`create`](#create)   | mutation | `compute:batches:create` @ `compute-tenant`  | v1.0  | registered |
| [`get`](#get)         | read     | `compute:batches:read` @ `compute-tenant`    | v1.0  | registered |
| [`list`](#list)       | read     | `compute:batches:read` @ `compute-tenant`    | v1.0  | registered |
| [`scope`](#scope)     | read     | `compute:batches:read` @ `compute-tenant`    | v1.0  | registered |
| [`members`](#members) | read     | `compute:batches:read` @ `compute-tenant`    | v1.0  | registered |
| [`approve`](#approve) | mutation | `compute:batches:approve` @ `compute-tenant` | v1.0  | registered |
| [`cancel`](#cancel)   | mutation | `compute:batches:approve` @ `compute-tenant` | v1.0  | registered |

## create [#create]

`POST /api/compute/commandBatches/create`

Creates a batch from its (subject, kind) leaves, each optionally bound to a payload's SHA-256, a reason and a lifetime; returns it `pending`, with the approvals its size needs.

* The tree is fixed now: its root, size and every leaf's proof are stored, so what approvers review is what the daemons check. Subjects are computed as the executing agent does (see pki `batchSubject`): the node id, `bmc:<site agent>/<address>` for BMC kinds, `node:<target>` for `site.httpProxy`. A leaf without `payloadSha256` admits any payload of its kind, and `scope` flags it to approvers.
* **Idempotency.** The batch id derives from the tenant, the caller and `idempotencyKey`. The same request again (the same leaves in any order, reason and lifetime) returns the stored batch without a new event; a different request under the key is `CONFLICT`.
* The lifetime is `notAfterSeconds` (default 24 hours): after it the batch admits no commands and can no longer be approved. At most 50,000 leaves.

Publishes `command.batch.created` with the scope size and the approvals required.

* **Kind:** mutation
* **Authorization:** `compute:batches:create` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field                     | Type           | Required | Constraints |
| ------------------------- | -------------- | -------- | ----------- |
| `tenantId`                | string         | yes      |             |
| `reason`                  | string         | yes      |             |
| `scope`                   | list of object | yes      |             |
| `scope[].subject`         | string         | yes      |             |
| `scope[].kind`            | string         | yes      |             |
| `scope[].payloadSha256`   | string         |          |             |
| `scope[].payload`         | object         |          |             |
| `scope[].payload.typeUrl` | string         | yes      |             |
| `scope[].payload.value`   | string         | yes      |             |
| `notAfterSeconds`         | integer        |          |             |
| `idempotencyKey`          | string         | yes      |             |

## get [#get]

`POST /api/compute/commandBatches/get`

Returns one batch of the tenant: its reason, state, scope root and size, the approvals it needs and has, and how many of its commands are in each state.

The state is `expired` as soon as `notAfter` passes, even before the minute sweep writes it. A batch of another tenant is `NOT_FOUND`.

`createdBy` is the identity that created the batch. When an agent proposed it on that identity's behalf (MCP `propose_command_batch`), `createdByAgent` and `delegationId` say so, and the `reason` is the agent's text, not the person's.

* **Kind:** read
* **Authorization:** `compute:batches:read` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints |
| ---------- | ------ | -------- | ----------- |
| `tenantId` | string | yes      |             |
| `batchId`  | string | yes      |             |

## list [#list]

`POST /api/compute/commandBatches/list`

Returns the tenant's batches, newest first, filtered by `state`.

`limit` is 1 to 200 (default 50); `next`, when present, is the `cursor` for the following page.

* **Kind:** read
* **Authorization:** `compute:batches:read` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type                                                 | Required | Constraints |
| ---------- | ---------------------------------------------------- | -------- | ----------- |
| `tenantId` | string                                               | yes      |             |
| `state`    | one of `pending`, `approved`, `expired`, `cancelled` |          |             |
| `limit`    | integer                                              |          |             |
| `cursor`   | string                                               |          |             |

## scope [#scope]

`POST /api/compute/commandBatches/scope`

Returns a batch's leaves in leaf order, for approvers to review before signing: each leaf's index, subject, kind and `payloadSha256`, plus the leaf's `payload` (`{ typeUrl, value }` base64) when its creator stored it. Decode it by the kind's payload type, after checking it against `payloadSha256`.

An empty `payloadSha256` marks a leaf that admits any payload of its kind, which approvers should look at closely. Pages hold 1 to 5,000 leaves (1,000 by default); `next` is the leaf index to pass as `cursor`.

* **Kind:** read
* **Authorization:** `compute:batches:read` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type    | Required | Constraints |
| ---------- | ------- | -------- | ----------- |
| `tenantId` | string  | yes      |             |
| `batchId`  | string  | yes      |             |
| `cursor`   | integer |          |             |
| `limit`    | integer |          |             |

## members [#members]

`POST /api/compute/commandBatches/members`

Returns the commands enqueued into a batch so far, in command-id order: each one's node, (subject, kind), state and reason, and `awaitingApproval` while it is held for the batch's approval.

`state` keeps one command state. Pages hold 1 to 500 commands (100 by default); `next` is the command id to pass as `cursor`. A batch of another tenant is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:batches:read` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type                                                                                                          | Required | Constraints |
| ---------- | ------------------------------------------------------------------------------------------------------------- | -------- | ----------- |
| `tenantId` | string                                                                                                        | yes      |             |
| `batchId`  | string                                                                                                        | yes      |             |
| `state`    | one of `pending`, `dispatched`, `acked`, `running`, `succeeded`, `failed`, `rejected`, `expired`, `cancelled` |          |             |
| `limit`    | integer                                                                                                       |          |             |
| `cursor`   | string                                                                                                        |          |             |

## approve [#approve]

`POST /api/compute/commandBatches/approve`

Adds one approver's signature over the batch's root; once the signatures meet the threshold for its size, the batch becomes `approved` and its held commands are released at once.

* `signature` is `{ kid, sig }`: the approver key id and a base64url P-256 signature over the batch (id, root, size, end, trust domain), made off the control plane. A signature by a key outside the published set, or over anything else, is `ACCESS_DENIED`. The same approver twice counts once.
* Decided as `compute:batches:approve`, on top of the signature, never instead of it.
* `INVALID_STATE` for a batch that is not `pending`, or one that needs no approvals (no approver keys, or a size the tiers exempt). `INVALID_CONFIG` when the deployment has no trust domain (no `pki.join`).

Publishes `command.batch.approved` with the approver key ids when the batch completes.

* **Kind:** mutation
* **Authorization:** `compute:batches:approve` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type   | Required | Constraints |
| --------------- | ------ | -------- | ----------- |
| `tenantId`      | string | yes      |             |
| `batchId`       | string | yes      |             |
| `signature`     | object | yes      |             |
| `signature.kid` | string | yes      |             |
| `signature.sig` | string | yes      |             |

## cancel [#cancel]

`POST /api/compute/commandBatches/cancel`

Ends a pending batch: it becomes `cancelled` and its held commands are cancelled with it.

A batch that is not `pending` is `INVALID_STATE`. Decided as `compute:batches:approve`.

* **Kind:** mutation
* **Authorization:** `compute:batches:approve` @ `compute-tenant`
* **Module:** `quorum`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints |
| ---------- | ------ | -------- | ----------- |
| `tenantId` | string | yes      |             |
| `batchId`  | string | yes      |             |


# commands (/docs/reference/api/commands)

> One-shot actions on one node, queued as durable rows in the node's command queue and delivered in order over its session. Every method is decided on `compute-node/{nodeId}`: `run` and `cancel` with `compute:nodes:exec`, `get` and `list` with `compute:nodes:read`. See [Commands](/docs/guides/nodes/commands).



One-shot actions on one node, queued as durable rows in the node's command queue and delivered in order over its session. Every method is decided on `compute-node/{nodeId}`: `run` and `cancel` with `compute:nodes:exec`, `get` and `list` with `compute:nodes:read`. See [Commands](/docs/guides/nodes/commands).

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                         | Phase | Status     |
| ------------------- | -------- | ------------------------------------- | ----- | ---------- |
| [`run`](#run)       | mutation | `compute:nodes:exec` @ `compute-node` | v0.1  | registered |
| [`get`](#get)       | read     | `compute:nodes:read` @ `compute-node` | v0.1  | registered |
| [`list`](#list)     | read     | `compute:nodes:read` @ `compute-node` | v0.1  | registered |
| [`cancel`](#cancel) | mutation | `compute:nodes:exec` @ `compute-node` | v0.2  | registered |

## run [#run]

`POST /api/compute/commands/run`

Queues a command of an allowlisted kind (`diag.nvml`, `inventory.refresh`, `logs.collect`, `node.reboot`) on a node the tenant can see; returns the command, `pending` when first queued.

* Idempotent per tenant, caller identity and `idempotencyKey`: a retry with the same key returns the command already queued. The same key with a different `kind` is `CONFLICT`.

* `notAfterSeconds` (10 to 3600, default 600) sets how long the command may wait for the node before it expires.

* Any other kind is `UNSUPPORTED`; a payload that does not match the kind's schema is `INVALID_INPUT`; a node outside the tenant's view is `NOT_FOUND`.

* For `logs.collect`, when the server has `supportBundles` with a presigning blob store, the daemon receives a presigned upload URL and the response carries the bundle's `bundleKey`.

* Emits `node.command.queued` when the command is first queued, not on a replay.

* **Kind:** mutation

* **Authorization:** `compute:nodes:exec` @ `compute-node`

* **Module:** `server`

* **Phase:** v0.1 (registered)

**Input**

| Field             | Type    | Required | Constraints                                                              |
| ----------------- | ------- | -------- | ------------------------------------------------------------------------ |
| `tenantId`        | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`               |
| `nodeId`          | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$`        |
| `kind`            | string  | yes      | at most 64 characters; matches `^[a-z][a-z0-9-]*(?:\.[a-z][a-z0-9-]*)+$` |
| `payload`         | object  |          |                                                                          |
| `idempotencyKey`  | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`           |
| `notAfterSeconds` | integer |          | 10-3600; default `600`                                                   |
| `batchId`         | string  |          | at most 64 characters; matches `^batch_[0-9a-z]{26,52}$`                 |

## get [#get]

`POST /api/compute/commands/get`

Returns one command of a node by id, with its state, deadlines and safety flags.

`NOT_FOUND` when the command is not on that node or the node is outside the tenant's view.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                       |
| ----------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`    | string | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `commandId` | string | yes      | at most 64 characters; matches `^cmd_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## list [#list]

`POST /api/compute/commands/list`

Lists a node's commands newest first (by queue sequence), optionally filtered by `state`; returns a page with a cursor.

`limit` is at most 200 (100 by default). The cursor is the sequence of the last command returned.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type                                                                                                          | Required | Constraints                                                       |
| ---------- | ------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string                                                                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string                                                                                                        | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `state`    | one of `pending`, `dispatched`, `acked`, `running`, `succeeded`, `failed`, `rejected`, `expired`, `cancelled` |          |                                                                   |
| `limit`    | integer                                                                                                       |          | 1-200; default `100`                                              |
| `cursor`   | string                                                                                                        |          | at most 19 characters; matches `^\d{1,19}$`                       |

## cancel [#cancel]

`POST /api/compute/commands/cancel`

Cancels a command: a `pending` one becomes `cancelled` and never runs, an in-flight one (`dispatched`, `acked`, `running`) is asked to stop through its gateway; returns the command as it stands after the request.

* A command already in a terminal state (`succeeded`, `failed`, `rejected`, `expired`, `cancelled`) is returned unchanged and no event is emitted.

* An in-flight command keeps its state until the node reports; the request is recorded on the row.

* Emits `node.command.cancel-requested`. `NOT_FOUND` when the command is not on that node or the node is outside the tenant's view.

* **Kind:** mutation

* **Authorization:** `compute:nodes:exec` @ `compute-node`

* **Module:** `server`

* **Phase:** v0.2 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                       |
| ----------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`    | string | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `commandId` | string | yes      | at most 64 characters; matches `^cmd_[0-9a-hjkmnp-tv-z]{26,52}$`  |


# dnsRecords (/docs/reference/api/dns-records)

> The records of a tenant's [DNS zone](/docs/reference/api/dns-zones). Every method names the zone and is decided on it: reading takes `compute:dns:read`, everything else `compute:dns:manage`, on `provider-dns-zone/{zoneId}`. Records book no quota (at most 500 per zone) and do not bill. The provider's ownership marker (`_better-compute`) and its apex NS and SOA records are never touched.



The records of a tenant's [DNS zone](/docs/reference/api/dns-zones). Every method names the zone and is decided on it: reading takes `compute:dns:read`, everything else `compute:dns:manage`, on `provider-dns-zone/{zoneId}`. Records book no quota (at most 500 per zone) and do not bill. The provider's ownership marker (`_better-compute`) and its apex NS and SOA records are never touched.

All 5 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                      | Phase | Status     |
| ------------------- | -------- | -------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:dns:manage` @ `compute-provider-dns-zone` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:dns:read` @ `compute-provider-dns-zone`   | v1.0  | registered |
| [`get`](#get)       | read     | `compute:dns:read` @ `compute-provider-dns-zone`   | v1.0  | registered |
| [`update`](#update) | mutation | `compute:dns:manage` @ `compute-provider-dns-zone` | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:dns:manage` @ `compute-provider-dns-zone` | v1.0  | registered |

## create [#create]

`POST /api/compute/dnsRecords/create`

Makes a record (`name` relative to the zone, `@` for the apex; `type` A, AAAA, CNAME, TXT, MX, SRV, CAA or NS, default A) and returns it `pending` until its zone is ready and the provider has it. `data` is checked against the type before anything is written: an A record holds an IPv4 address, an AAAA record an IPv6 one, CNAME, MX, SRV and NS an absolute host name with a final dot. An apex NS or CNAME and the reserved `_better-compute` name are `INVALID_INPUT`; the same name, type and data twice is `CONFLICT`. An A or AAAA record may follow an instance of the tenant instead (`instanceId`, no `data`): its data is a reserved address assigned to the instance, else the instance's first public address; the record waits until there is one and is deleted with the instance. Another tenant's instance is `ACCESS_DENIED`; the instance is also decided as `compute:instances:update`, with the caller's own credential. Publishes `providerDnsRecord.created`, then `providerDnsRecord.ready` or `providerDnsRecord.error`.

* **Kind:** mutation
* **Authorization:** `compute:dns:manage` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                                                         | Required | Constraints                                                       |
| ---------------- | ------------------------------------------------------------ | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string                                                       | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `zoneId`         | string                                                       | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`           | string                                                       | yes      | at most 200 characters                                            |
| `type`           | one of `A`, `AAAA`, `CNAME`, `TXT`, `MX`, `SRV`, `CAA`, `NS` |          | default `"A"`                                                     |
| `instanceId`     | string                                                       |          | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `data`           | string                                                       |          | at most 512 characters                                            |
| `ttlSeconds`     | integer                                                      |          | 30-86400                                                          |
| `priority`       | integer                                                      |          | 0-65535                                                           |
| `port`           | integer                                                      |          | 1-65535                                                           |
| `weight`         | integer                                                      |          | 0-65535                                                           |
| `flags`          | integer                                                      |          | 0-255                                                             |
| `tag`            | one of `issue`, `issuewild`, `iodef`                         |          |                                                                   |
| `idempotencyKey` | string                                                       | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |

## list [#list]

`POST /api/compute/dnsRecords/list`

Returns the zone's records in the order they were made (at most 1000). Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:dns:read` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                      |
| ---------------- | ------- | -------- | ---------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`         | string  | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `includeDeleted` | boolean |          | default `false`                                                  |

## get [#get]

`POST /api/compute/dnsRecords/get`

Returns one record: its name, `fqdn`, type, data, TTL, the type's own fields and its state.

* **Kind:** read
* **Authorization:** `compute:dns:read` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`   | string | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `recordId` | string | yes      | at most 56 characters; matches `^pdr_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/dnsRecords/update`

Changes `data`, `ttlSeconds` and the type's own fields (`priority`, `port`, `weight`, `flags`, `tag`) in place and returns it `updating`; the name and type never change. A record that follows an instance takes no `data`.

* **Kind:** mutation
* **Authorization:** `compute:dns:manage` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type                                 | Required | Constraints                                                      |
| ------------ | ------------------------------------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string                               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`     | string                               | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `recordId`   | string                               | yes      | at most 56 characters; matches `^pdr_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `data`       | string                               |          | at most 512 characters                                           |
| `ttlSeconds` | integer                              |          | 30-86400                                                         |
| `priority`   | integer                              |          | 0-65535                                                          |
| `port`       | integer                              |          | 1-65535                                                          |
| `weight`     | integer                              |          | 0-65535                                                          |
| `flags`      | integer                              |          | 0-255                                                            |
| `tag`        | one of `issue`, `issuewild`, `iodef` |          |                                                                  |

## delete [#delete]

`POST /api/compute/dnsRecords/delete`

Marks the record for deletion and returns it `deleting`; the reconciler deletes it at the provider and publishes `providerDnsRecord.deleted`. A record whose id the provider has since given to another record is not deleted: the row is finished with a warning in the log.

* **Kind:** mutation
* **Authorization:** `compute:dns:manage` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`   | string | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `recordId` | string | yes      | at most 56 characters; matches `^pdr_[0-9a-hjkmnp-tv-z]{26,52}$` |


# dnsZones (/docs/reference/api/dns-zones)

> DNS zones on a provider account (DigitalOcean Domains first), with their [records](/docs/reference/api/dns-records). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.dns`. Reading takes `compute:dns:read`, everything else `compute:dns:manage`, on `provider-dns-zone/{id}`. Zones book no quota (at most 50 per tenant) and do not bill.



DNS zones on a provider account (DigitalOcean Domains first), with their [records](/docs/reference/api/dns-records). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.dns`. Reading takes `compute:dns:read`, everything else `compute:dns:manage`, on `provider-dns-zone/{id}`. Zones book no quota (at most 50 per tenant) and do not bill.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                      | Phase | Status     |
| ------------------- | -------- | -------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:dns:manage` @ `compute-provider-dns-zone` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:dns:read` @ `compute-tenant` + items      | v1.0  | registered |
| [`get`](#get)       | read     | `compute:dns:read` @ `compute-provider-dns-zone`   | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:dns:manage` @ `compute-provider-dns-zone` | v1.0  | registered |

## create [#create]

`POST /api/compute/dnsZones/create`

Makes the zone for a domain (`zone`, lower case, without a final dot) and returns it `pending`. A domain has one live zone per provider account, so two tenants never share one: a second is `CONFLICT`, without saying whose it is, and so is a zone inside or above another tenant's (`shop.example.com` next to `example.com`), which the provider's shared name servers could answer for. A zone the account already holds and vps did not make goes to `error` unless the account's owner tenant adopts it with `adopt: true`; any other tenant asking to adopt is `ACCESS_DENIED` (reason `adoptNotOwner`). Publishes `providerDnsZone.created`, then `providerDnsZone.ready` or `providerDnsZone.error`.

* **Kind:** mutation
* **Authorization:** `compute:dns:manage` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `poolId`         | string  | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `zone`           | string  | yes      | at most 253 characters                                            |
| `adopt`          | boolean |          | default `false`                                                   |
| `idempotencyKey` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |

## list [#list]

`POST /api/compute/dnsZones/list`

Returns the tenant's zones in the order they were made (at most 1000), optionally of one pool or project. Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:dns:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/dnsZones/get`

Returns one zone: its domain, whether it was adopted, its state and, once the provider reported them, the `nameServers` the domain's registrar must delegate to.

* **Kind:** read
* **Authorization:** `compute:dns:read` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`   | string | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/dnsZones/delete`

Marks the zone for deletion and returns it `deleting`; the reconciler deletes it at the provider and publishes `providerDnsZone.deleted`. A zone that still holds records is `INVALID_STATE` (reason `zoneNotEmpty`). An adopted zone is only forgotten: it was the account's before and stays at the provider.

* **Kind:** mutation
* **Authorization:** `compute:dns:manage` @ `compute-provider-dns-zone`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `zoneId`   | string | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$` |


# drivers (/docs/reference/api/drivers)

> The drivers this deployment registered through its options, modules and plugins, so `metal.enroll` callers and the console can offer valid choices. The group returns declarations only, never a driver's configuration or credentials. See [Drivers](/docs/extending/drivers).



The drivers this deployment registered through its options, modules and plugins, so `metal.enroll` callers and the console can offer valid choices. The group returns declarations only, never a driver's configuration or credentials. See [Drivers](/docs/extending/drivers).

All 1 methods are served by @better-compute/server 0.3.0.

| Method          | Kind | Authorization                             | Phase | Status     |
| --------------- | ---- | ----------------------------------------- | ----- | ---------- |
| [`list`](#list) | read | `compute:drivers:read` @ `compute-tenant` | v0.2  | registered |

## list [#list]

`POST /api/compute/drivers/list`

Lists the registered drivers with their id, version, `runsOn`, implemented kinds, capabilities and the operations safe to retry per kind; `kind` keeps only drivers that implement it.

Decided with `compute:drivers:read` on `compute-tenant/{tenantId}`. Items are sorted by id, then version, and the result is not paged.

* **Kind:** read
* **Authorization:** `compute:drivers:read` @ `compute-tenant`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `kind`     | string |          | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`    |


# events (/docs/reference/api/events)

> A tenant's CloudEvents-style events, read from the transactional outbox that every change writes to in its own transaction. Pages follow the outbox's `txid:seq` position, so no event ever appears behind a cursor already handed out. See [Events](/docs/guides/events).



A tenant's CloudEvents-style events, read from the transactional outbox that every change writes to in its own transaction. Pages follow the outbox's `txid:seq` position, so no event ever appears behind a cursor already handed out. See [Events](/docs/guides/events).

All 1 methods are served by @better-compute/server 0.3.0.

| Method          | Kind | Authorization                            | Phase | Status     |
| --------------- | ---- | ---------------------------------------- | ----- | ---------- |
| [`list`](#list) | read | `compute:events:read` @ `compute-tenant` | v0.1  | registered |

## list [#list]

`POST /api/compute/events/list`

Returns a page of the tenant's committed events after `cursor`, oldest first, optionally only those whose type starts with `type` (for example `node.` or `reservation.`).

* Decided with `compute:events:read` on `compute-tenant/{tenantId}`.

* The cursor is an event's `sequence` in the form `txid:seq`; without one the page starts at the oldest event still in the outbox.

* Only events whose transaction is older than every transaction still open are returned, so a later page never gains an event that sorts before its cursor.

* `limit` is at most 500 (100 by default).

* **Kind:** read

* **Authorization:** `compute:events:read` @ `compute-tenant`

* **Module:** `server`

* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                |
| ---------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `type`     | string  |          | at most 64 characters; matches `^[a-z][a-z0-9.-]{0,63}$`   |
| `limit`    | integer |          | 1-500; default `100`                                       |
| `cursor`   | string  |          | at most 41 characters; matches `^\d{1,20}:\d{1,20}$`       |


# filesystems (/docs/reference/api/filesystems)

> Filesystems are shared filesystems on a tenant [network](/docs/reference/api/networks) that the network's instances mount at boot: `nfs`, an elastic NFS filesystem across every zone of the network (Amazon EFS on AWS), or `lustre`, a Lustre filesystem in one zone for training scratch and checkpoints (Amazon FSx for Lustre). Like [firewall rules](/docs/reference/api/firewall-rules) they belong to a network and are decided on it: reading them takes `compute:networks:read`, changing them `compute:networks:update`. The provider must support filesystems (AWS does; another provider is `UNSUPPORTED`). The calls record what the tenant wants and return at once. The `vps.filesystems` reconciler then makes the filesystem at the provider once its network has been ready, behind its own security group that admits only the network's machines, and always encrypted at rest. `state` shows where it is: `pending`, `ready`, `deleting`, `deleted` or `error` (with a `message`); an `error` is retried every five minutes. Once ready, `mount` says how a machine mounts it (`fstype`, `source`, `options`): DNS names and mount names, never a secret.



Filesystems are shared filesystems on a tenant [network](/docs/reference/api/networks) that the network's instances mount at boot: `nfs`, an elastic NFS filesystem across every zone of the network (Amazon EFS on AWS), or `lustre`, a Lustre filesystem in one zone for training scratch and checkpoints (Amazon FSx for Lustre). Like [firewall rules](/docs/reference/api/firewall-rules) they belong to a network and are decided on it: reading them takes `compute:networks:read`, changing them `compute:networks:update`. The provider must support filesystems (AWS does; another provider is `UNSUPPORTED`). The calls record what the tenant wants and return at once. The `vps.filesystems` reconciler then makes the filesystem at the provider once its network has been ready, behind its own security group that admits only the network's machines, and always encrypted at rest. `state` shows where it is: `pending`, `ready`, `deleting`, `deleted` or `error` (with a `message`); an `error` is retried every five minutes. Once ready, `mount` says how a machine mounts it (`fstype`, `source`, `options`): DNS names and mount names, never a secret.

Instances mount filesystems with `instances.create({ networkId, filesystems: [{ filesystem, path }] })`: ready filesystems of the instance's own network, at most 16, each at an absolute path of plain segments outside the system directories (`/etc`, `/usr`, `/var` and the like). The provider renders the mounts into the machine's user data, before the tenant's own part, with `nofail`, so a missing filesystem never blocks a boot.

The provider-side name of a filesystem is `bc-` and its id, lower-cased. A filesystem the provider holds under that name that better-compute did not make is never adopted or deleted (`error`). Events are published on the network: `network.filesystem.created`, `ready`, `error`, `deleting` and `deleted`, with `networkId` and `filesystemId`.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                 | Phase | Status     |
| ------------------- | -------- | --------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:networks:update` @ `compute-network` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:networks:read` @ `compute-network`   | v1.0  | registered |
| [`get`](#get)       | read     | `compute:networks:read` @ `compute-network`   | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:networks:update` @ `compute-network` | v1.0  | registered |

## create [#create]

`POST /api/compute/filesystems/create`

Records a filesystem `pending` on the network and returns it.

* `kind` is `nfs` or `lustre`. An `nfs` filesystem is elastic and spans every zone of the network (a mount target in each zone that has a subnet): it takes no `sizeGiB`, `deployment` or `zone` (`INVALID_INPUT`).

* A `lustre` filesystem takes `sizeGiB` (AWS: 1200, or a multiple of 2400; default 1200), `deployment` (`scratch`, the default, or `persistent`) and `zone` (one of the account's zones with a subnet of the network; default the account's first). A size or zone the provider can never make is `INVALID_INPUT` and nothing is recorded.

* The name is unique in the network (`CONFLICT`); the name of a deleted filesystem is free again. A network has at most 8 filesystems that are not deleted (`QUOTA_EXCEEDED`, with the cap in `details.limit`).

* A network being deleted is `INVALID_STATE`; an adopted network (`networks.adopt`) is `INVALID_INPUT`: mount targets need subnets better-compute made.

* Idempotent: the same request under the same `idempotencyKey` returns the filesystem; another request under the key is `CONFLICT`.

* **Kind:** mutation

* **Authorization:** `compute:networks:update` @ `compute-network`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                           | Required | Constraints                                                        |
| ---------------- | ------------------------------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string                         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `networkId`      | string                         | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `name`           | string                         | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `kind`           | one of `nfs`, `lustre`         | yes      |                                                                    |
| `sizeGiB`        | integer                        |          | 1-1000000                                                          |
| `deployment`     | one of `scratch`, `persistent` |          |                                                                    |
| `zone`           | string                         |          | at most 64 characters                                              |
| `idempotencyKey` | string                         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/filesystems/list`

Returns the network's filesystems that are not deleted, sorted by name.

* **Kind:** read
* **Authorization:** `compute:networks:read` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |

## get [#get]

`POST /api/compute/filesystems/get`

Returns one filesystem of the network.

* **Kind:** read
* **Authorization:** `compute:networks:read` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field          | Type   | Required | Constraints                                                      |
| -------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`     | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId`    | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `filesystemId` | string | yes      | at most 55 characters; matches `^fs_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## delete [#delete]

`POST /api/compute/filesystems/delete`

Marks the filesystem for deletion and returns it `deleting` with `version` bumped; the reconciler removes the mount targets or file servers, the filesystem and its security group, without a final backup. A filesystem that an instance (not deleted) mounts is `INVALID_STATE` with `details.instanceIds`. Deleting one that is being deleted returns it unchanged. A network with filesystems that are not deleted cannot be deleted (`networks.delete` is `INVALID_STATE` with `details.filesystemIds`).

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field          | Type   | Required | Constraints                                                      |
| -------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`     | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId`    | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `filesystemId` | string | yes      | at most 55 characters; matches `^fs_[0-9a-hjkmnp-tv-z]{26,52}$`  |


# firewallRules (/docs/reference/api/firewall-rules)

> Firewall rules belong to a [network](/docs/reference/api/networks) and are decided on it: reading them takes `compute:networks:read`, changing them `compute:networks:update`. A rule says which traffic is allowed or denied, from where and to which instances, and the `vps.networks` reconciler applies it at the provider as a firewall named `bc-` and the rule's id. A rule that names instances applies to the machines carrying those instances' network tags (`bc-` and 16 hex digits of the sha256 of the instance id), which the reconciler sets on each instance that owns its machine (applying rules first: a tag set later still matches); a rule that names none applies to every instance on the network. A network has at most 100 rules that are not deleted.



Firewall rules belong to a [network](/docs/reference/api/networks) and are decided on it: reading them takes `compute:networks:read`, changing them `compute:networks:update`. A rule says which traffic is allowed or denied, from where and to which instances, and the `vps.networks` reconciler applies it at the provider as a firewall named `bc-` and the rule's id. A rule that names instances applies to the machines carrying those instances' network tags (`bc-` and 16 hex digits of the sha256 of the instance id), which the reconciler sets on each instance that owns its machine (applying rules first: a tag set later still matches); a rule that names none applies to every instance on the network. A network has at most 100 rules that are not deleted.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                 | Phase | Status     |
| ------------------- | -------- | --------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:networks:update` @ `compute-network` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:networks:read` @ `compute-network`   | v1.0  | registered |
| [`update`](#update) | mutation | `compute:networks:update` @ `compute-network` | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:networks:update` @ `compute-network` | v1.0  | registered |

## create [#create]

`POST /api/compute/firewallRules/create`

Adds a rule to the network and returns it `pending`.

* `direction` is `ingress` or `egress`. `action` defaults to `allow`, `priority` to 1000 (0 to 65535, lower wins) and `enabled` to `true`; a disabled rule exists at the provider but matches nothing.
* `rules` lists 1 to 20 entries `{ protocol, ports? }` with `protocol` one of `tcp`, `udp`, `icmp` and `all`. `ports` (tcp and udp only) are ports (`"22"`) or ranges (`"8000-8100"`); without them every port is covered.
* An ingress rule must name its `sources`, IPv4 CIDRs, and takes no `destinations`; the whole internet is `0.0.0.0/0`, written out, never implied. An egress rule takes `destinations` (without them, everywhere) and no `sources`. Violations are `INVALID_INPUT`.
* `targets.instanceIds` names instances of the tenant that are on this network and not deleted; any other is `INVALID_INPUT`. Empty targets cover the whole network, except on an adopted network (`networks.adopt`), where a rule must name its instances (`INVALID_INPUT` otherwise, on `update` too): it never covers machines of the VPC that are not the tenant's.
* The name is unique in the network (`CONFLICT`); the name of a deleted rule is free again. A hundred live rules already is `QUOTA_EXCEEDED`. A network being deleted is `INVALID_STATE`.

Publishes `network.rule.ready` once the provider has the rule, or `network.rule.error` with the reason. Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                 | Type                               | Required | Constraints                                                        |
| --------------------- | ---------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`            | string                             | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `networkId`           | string                             | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `name`                | string                             | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `direction`           | one of `ingress`, `egress`         | yes      |                                                                    |
| `action`              | one of `allow`, `deny`             |          | default `"allow"`                                                  |
| `priority`            | integer                            |          | 0-65535; default `1000`                                            |
| `rules`               | list of object                     | yes      | 1-20 items                                                         |
| `rules[].protocol`    | one of `tcp`, `udp`, `icmp`, `all` | yes      |                                                                    |
| `rules[].ports`       | list of string                     |          | 0-50 items                                                         |
| `sources`             | list of string                     |          | 0-256 items                                                        |
| `destinations`        | list of string                     |          | 0-256 items                                                        |
| `targets`             | object                             |          |                                                                    |
| `targets.instanceIds` | list of string                     |          | 0-64 items                                                         |
| `enabled`             | boolean                            |          | default `true`                                                     |

## list [#list]

`POST /api/compute/firewallRules/list`

Returns the network's rules that are not deleted, sorted by name, with their state (`pending`, `ready`, `deleting`, `error`).

* **Kind:** read
* **Authorization:** `compute:networks:read` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/firewallRules/update`

Changes the given fields of a rule (`direction`, `action`, `priority`, `rules`, `sources`, `destinations`, `targets`, `enabled`; the name never changes), checks the result as `create` does, bumps `version`, returns the rule `pending` and wakes the reconciler. An empty `sources` or `destinations` list clears it. A rule that changes direction names its other range list again.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                 | Type                               | Required | Constraints                                                      |
| --------------------- | ---------------------------------- | -------- | ---------------------------------------------------------------- |
| `tenantId`            | string                             | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId`           | string                             | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `ruleId`              | string                             | yes      | at most 56 characters; matches `^fwr_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `direction`           | one of `ingress`, `egress`         |          |                                                                  |
| `action`              | one of `allow`, `deny`             |          | default `"allow"`                                                |
| `priority`            | integer                            |          | 0-65535; default `1000`                                          |
| `rules`               | list of object                     |          | 1-20 items                                                       |
| `rules[].protocol`    | one of `tcp`, `udp`, `icmp`, `all` | yes      |                                                                  |
| `rules[].ports`       | list of string                     |          | 0-50 items                                                       |
| `sources`             | list of string                     |          | 0-256 items                                                      |
| `destinations`        | list of string                     |          | 0-256 items                                                      |
| `targets`             | object                             |          |                                                                  |
| `targets.instanceIds` | list of string                     |          | 0-64 items                                                       |
| `enabled`             | boolean                            |          | default `true`                                                   |

## delete [#delete]

`POST /api/compute/firewallRules/delete`

Marks the rule for deletion and returns it `deleting` with `version` bumped; the reconciler removes the firewall at the provider. Deleting a rule that is being deleted returns it unchanged.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `ruleId`    | string | yes      | at most 56 characters; matches `^fwr_[0-9a-hjkmnp-tv-z]{26,52}$` |


# flavors (/docs/reference/api/flavors)

> Flavors are the named instance shapes of a VPS pool: vCPU, memory, disk and GPU count, plus the provider offer a provider pool creates. The pool's owner manages them, and tenants the pool is offered to can list them. An instance keeps the shape it was admitted with, so changing or deleting a flavor never changes a live instance. See [Flavors](/docs/guides/vps/flavors).



Flavors are the named instance shapes of a VPS pool: vCPU, memory, disk and GPU count, plus the provider offer a provider pool creates. The pool's owner manages them, and tenants the pool is offered to can list them. An instance keeps the shape it was admitted with, so changing or deleting a flavor never changes a live instance. See [Flavors](/docs/guides/vps/flavors).

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                             | Phase | Status     |
| ------------------- | -------- | ----------------------------------------- | ----- | ---------- |
| [`list`](#list)     | read     | `compute:flavors:read` @ `compute-pool`   | v0.3  | registered |
| [`create`](#create) | mutation | `compute:flavors:manage` @ `compute-pool` | v0.3  | registered |
| [`update`](#update) | mutation | `compute:flavors:manage` @ `compute-pool` | v0.3  | registered |
| [`delete`](#delete) | mutation | `compute:flavors:manage` @ `compute-pool` | v0.3  | registered |

## list [#list]

`POST /api/compute/flavors/list`

Returns the pool's flavors that are not archived, sorted by name.

The pool must be owned by or offered to the tenant; otherwise the call is `NOT_FOUND`.

**Estimates.** Each item of a provider pool carries `estimate` when the IAM port can quote (`IamPort.quote`): `{ perHourMicros, currency, lines }`, what one hour of the flavor costs the tenant named in the call.

* **How.** The lines are the meters an instance of the flavor would record per hour (the pool's billing mode, the flavor's meter and the shape's vCPUs, memory and GPUs, as `meterLinesFor` computes them), each priced by IAM with the tenant's effective price, so a negotiated price shows. `lines` are `{ meter, quantity, amountMicros }`; a line IAM has no price for has `amountMicros: null`, and then `perHourMicros` is `null` rather than a partial sum. A pool whose billing mode is `none` has no lines and costs `0`.

* **Absent** when the IAM port has no `quote` (see `billing`), when the pool is not a provider pool (hosted pools record usage differently), or when IAM cannot answer: an estimate never fails the list.

* **Currency** is the deployment currency (`vps({ billing: { currency } })`, default `USD`).

* **Before a create:** `instances.estimate` answers the same estimate for one flavor.

* **Freshness.** Estimates are kept five minutes per tenant and pool, so a changed price shows within five minutes.

* **Assumptions.** The estimate is for an on-demand instance of the flavor: Spot, reservation grants and the instance's disks and addresses are not in it. A flavor whose offer is held by a reservation is priced on the `vm.reserved.*` meters.

* **Kind:** read

* **Authorization:** `compute:flavors:read` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v0.3 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## create [#create]

`POST /api/compute/flavors/create`

Adds a flavor to a `vps-host` or `provider` pool the tenant owns and returns it at version 1.

* `gpus` defaults to 0. A flavor of a provider pool needs `offerId`, the provider offer it creates; without it the call is `INVALID_INPUT`.

* `placement` (hosted pools only; `INVALID_INPUT` on a provider pool) says where the guest lives on its host, and sandbox templates on the flavor inherit it:
  * `hugepages: '2Mi' | '1Gi'` backs guest memory with hugepages. These are booked from the host's hugepage pools instead of its memory, still charged to quota as `memoryGiB`, and never ballooned.
  * `numaAffine` keeps the vCPUs and memory on one NUMA node: the GPUs' node, or otherwise the least-loaded node with room for the memory.
  * `pinned` gives each vCPU a host CPU that no other pinned guest on the host has.
  * A host that cannot honour `numaAffine` or `pinned` still runs the guest, floating, and the instance carries a `PlacementDegraded` condition with the reason (`NoNumaTopology`, `NoNumaNodeFits`, `GpuNumaUnknown`, `NumaMemoryTooSmall` or `NotEnoughFreeCpus`).
  * Resizing to a flavor with a different `placement` is `UNSUPPORTED`.

* In a pool that overcommits memory (`policy.overcommit.memoryGiB` > 1), every guest gets a virtio balloon so the host can reclaim its free memory. The exceptions are hugepage-backed guests and guests with GPUs, whose memory is pinned. This needs no flavor field.

* `billing.meter` (root only) bills instances of the flavor as one line, `flavor.<name>.hours` at one unit per hour, instead of the composed `vm.vcpu`, `vm.memory`, `vm.gpu` and `vm.instance` lines. It must match `^flavor.[a-z0-9.-]{1,56}.hours$` and be at most 64 characters (a meter key's limit), else `INVALID_INPUT`; any other caller who adds, changes or clears one gets `ACCESS_DENIED` before the decision (resending the stored value is allowed), because the meter is defined on the platform's billing tenant. When the IAM port can define meters (`ensureMeters`), the meter is created on the root tenant once IAM has allowed the call, and `billing.syncMeters` and `billing.meters` include every flavor's meter; a port that cannot accepts the flavor and `billing.meters` reports the meter as `missing`. An instance copies the meter into its own flavor snapshot when it is created, so changing the flavor's meter later never changes how a running or stopped instance is billed; a granted or platform-fee instance ignores it.

* `NOT_FOUND` when the tenant does not own the pool; `INVALID_STATE` when the pool is of another class.

* `CONFLICT` when the pool already has a flavor with that name, archived ones included.

* **Kind:** mutation

* **Authorization:** `compute:flavors:manage` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v0.3 (registered)

**Input**

| Field                  | Type                | Required | Constraints                                                        |
| ---------------------- | ------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`             | string              | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `poolId`               | string              | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`                 | string              | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`       |
| `vcpu`                 | integer             | yes      | 1-1024                                                             |
| `memoryGiB`            | integer             | yes      | 1-16384                                                            |
| `diskGiB`              | integer             | yes      | 1-65536                                                            |
| `gpus`                 | integer             |          | 0-16; default `0`                                                  |
| `offerId`              | string              |          | 1-128 characters                                                   |
| `placement`            | object              |          |                                                                    |
| `placement.numaAffine` | boolean             |          |                                                                    |
| `placement.pinned`     | boolean             |          |                                                                    |
| `placement.hugepages`  | one of `2Mi`, `1Gi` |          |                                                                    |
| `billing`              | object              |          |                                                                    |
| `billing.meter`        | string              |          | at most 64 characters; matches `^flavor\.[a-z0-9.-]{1,56}\.hours$` |

## update [#update]

`POST /api/compute/flavors/update`

Replaces the shape of the named flavor and increments its version; an archived flavor becomes active again.

The input is the full shape, as for `create`. An omitted `offerId` keeps the stored one, and so does an omitted `billing`; `billing: {}` clears the meter. Setting, changing or clearing a meter is root only (`ACCESS_DENIED` otherwise, before the decision, and nothing is written); a tenant may resend the stored `billing` unchanged. Instances already admitted keep their old shape; only new instances and resizes use the new one. The call is `NOT_FOUND` when the tenant does not own the pool or the flavor does not exist.

* **Kind:** mutation
* **Authorization:** `compute:flavors:manage` @ `compute-pool`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field                  | Type                | Required | Constraints                                                        |
| ---------------------- | ------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`             | string              | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `poolId`               | string              | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`                 | string              | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`       |
| `vcpu`                 | integer             | yes      | 1-1024                                                             |
| `memoryGiB`            | integer             | yes      | 1-16384                                                            |
| `diskGiB`              | integer             | yes      | 1-65536                                                            |
| `gpus`                 | integer             |          | 0-16; default `0`                                                  |
| `offerId`              | string              |          | 1-128 characters                                                   |
| `placement`            | object              |          |                                                                    |
| `placement.numaAffine` | boolean             |          |                                                                    |
| `placement.pinned`     | boolean             |          |                                                                    |
| `placement.hugepages`  | one of `2Mi`, `1Gi` |          |                                                                    |
| `billing`              | object              |          |                                                                    |
| `billing.meter`        | string              |          | at most 64 characters; matches `^flavor\.[a-z0-9.-]{1,56}\.hours$` |

## delete [#delete]

`POST /api/compute/flavors/delete`

Deletes the named flavor, or archives it when any instance, deleted ones included, still references it; returns `deleted` and `archived` to say which.

An archived flavor disappears from `list` and can no longer be used to create or resize instances. `update` makes it active again. The call is `NOT_FOUND` when the tenant does not own the pool or the flavor does not exist.

* **Kind:** mutation
* **Authorization:** `compute:flavors:manage` @ `compute-pool`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `name`     | string | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`      |


# fleet (/docs/reference/api/fleet)

> The fleet group manages a tenant's platform objects as one FleetConfig document: sites, provider accounts, pools, owner quotas, offers and flavors. `plan` diffs a document against what exists, `apply` runs the plan through the owning modules' own methods, and the applied document is kept so that drift from it is reported. Every method decides `compute:fleet:admin` on the fleet scope. See [Fleet configuration](/docs/guides/fleet-config).



The fleet group manages a tenant's platform objects as one FleetConfig document: sites, provider accounts, pools, owner quotas, offers and flavors. `plan` diffs a document against what exists, `apply` runs the plan through the owning modules' own methods, and the applied document is kept so that drift from it is reported. Every method decides `compute:fleet:admin` on the fleet scope. See [Fleet configuration](/docs/guides/fleet-config).

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                           | Phase | Status     |
| ------------------- | -------- | --------------------------------------- | ----- | ---------- |
| [`export`](#export) | read     | `compute:fleet:admin` @ `compute-fleet` | v0.2  | registered |
| [`plan`](#plan)     | read     | `compute:fleet:admin` @ `compute-fleet` | v0.2  | registered |
| [`apply`](#apply)   | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.2  | registered |
| [`status`](#status) | read     | `compute:fleet:admin` @ `compute-fleet` | v0.2  | registered |

## export [#export]

`POST /api/compute/fleet/export`

Returns the tenant's current sites, provider accounts, pools, owner quotas, offers and flavors as a FleetConfig document, with `missing` listing the tables of modules this deployment does not run.

Objects that fleet never named appear under their ids. Archived pools and archived flavors are left out.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `fleet`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## plan [#plan]

`POST /api/compute/fleet/plan`

Diffs a FleetConfig document against the tenant's current state; returns the changes for people, the exact API calls `apply` would make, a summary, `blocked`, and a `planHash`.

* The call changes nothing. With `prune: true` the plan also deletes what the document no longer lists: flavors and offers are removed, pools archived, and unused provider accounts deleted. Sites are never changed or pruned.

* A `policy.substrate` the document no longer lists on a pool it keeps is a delete too (the pool's cluster is torn down): with `prune` it is a `delete` change on the pool with `fields: ['policy.substrate']`; without it the substrate stays in the policy that is sent and the pool's change carries a note.

* A change that cannot be applied (an immutable field such as a pool's class or an account's provider, an unknown reference, an offer to the pool's owner) is marked `blocked` with the reason.

* The ids of planned creates derive from the caller's identity, so the call needs an authenticated caller (`UNAUTHENTICATED` otherwise).

* **Kind:** read

* **Authorization:** `compute:fleet:admin` @ `compute-fleet`

* **Module:** `fleet`

* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                |
| ---------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `config`   | object  | yes      |                                                            |
| `prune`    | boolean |          | default `false`                                            |

## apply [#apply]

`POST /api/compute/fleet/apply`

Plans the document again and runs every planned call as the caller, all in one transaction; returns the plan and the number of calls applied.

* **Decisions.** Each call is authorized first, as the caller and with no transaction open, by the method it invokes; a refusal stops the apply before anything changes.

* **Consistency.** Pass the `planHash` from `plan` to apply exactly that plan: if the plan has changed, the call is `CONFIG_DRIFT`. The plan is made once more inside the transaction, and a change there is also `CONFIG_DRIFT`, with nothing applied.

* **Idempotency.** Creates carry keys derived from the tenant, the kind and the name, so repeating an apply creates nothing twice. A plan with no calls still records the document.

* **Errors.** `INVALID_STATE` with `details.blocked` when any change is blocked. `UNSUPPORTED` when the host cannot run composite methods. Errors of the invoked methods pass through.

* The document becomes the one that `status` and the `fleet.drift` reconciler compare against.

* **Kind:** mutation

* **Authorization:** `compute:fleet:admin` @ `compute-fleet`

* **Module:** `fleet`

* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                |
| ---------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `config`   | object  | yes      |                                                            |
| `prune`    | boolean |          | default `false`                                            |
| `planHash` | string  |          | at most 64 characters; matches `^[0-9a-f]{64}$`            |

## status [#status]

`POST /api/compute/fleet/status`

Returns the tenant's last applied document's plan hash, who applied it and when, and, while the fleet differs from it, the drift found at the last check.

`applied` is absent when the tenant never applied a document. The `fleet.drift` reconciler re-plans each applied document every 15 minutes by default and publishes `fleet.drifted` or `fleet.converged` when the answer changes.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `fleet`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |


# images (/docs/reference/api/images)

> The images group is the catalog of OS images and ramdisks that deploys write to disk. An image is registered by its digest and source URL, never uploaded, and a signature from a signer on the tenant's or the platform's allowlist marks it verified. Unsigned images stay private to their tenant; only verified images can be made public. See [Profiles and images](/docs/guides/bare-metal/profiles-and-images).



The images group is the catalog of OS images and ramdisks that deploys write to disk. An image is registered by its digest and source URL, never uploaded, and a signature from a signer on the tenant's or the platform's allowlist marks it verified. Unsigned images stay private to their tenant; only verified images can be made public. See [Profiles and images](/docs/guides/bare-metal/profiles-and-images).

All 5 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                                                             | Phase | Status     |
| ----------------------- | -------- | ----------------------------------------------------------------------------------------- | ----- | ---------- |
| [`register`](#register) | mutation | `compute:images:publish` @ `compute-tenant` or `compute:images:publish` @ `compute-fleet` | v0.2  | registered |
| [`get`](#get)           | read     | `compute:images:read` @ `compute-image`                                                   | v0.2  | registered |
| [`list`](#list)         | read     | `compute:images:read` @ `compute-tenant` + items                                          | v0.2  | registered |
| [`promote`](#promote)   | mutation | `compute:images:publish` @ `compute-image`                                                | v0.2  | registered |
| [`resolve`](#resolve)   | read     | `compute:flavors:read` @ `compute-pool`                                                   | v1.0  | registered |

## register [#register]

`POST /api/compute/images/register`

Records an image by `ref` (`name@version`), format, digest and `https://` or `oci://` source in the tenant's catalog; with a valid signature it is marked verified.

* Decided as `compute:images:publish` on the tenant.
* `signature` and `signerId` go together. The signature must verify against that signer on the tenant's allowlist or the platform's, else `INVALID_INPUT`.
* A `ramdisk` must be signed by a platform signer (`INVALID_INPUT` otherwise).
* `CONFLICT` when the tenant already has that `ref`, so a repeated call does not create a second image.

The new image is private. Site agents whose pools pre-warm this ref (`policy.prewarmImages`) get their desired state refreshed.

* **Kind:** mutation
* **Authorization:** `compute:images:publish` @ `compute-tenant` or `compute:images:publish` @ `compute-fleet`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field         | Type                          | Required | Constraints                                                |
| ------------- | ----------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId`    | string                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `ref`         | string                        | yes      | 1-200 characters                                           |
| `kind`        | one of `os`, `ramdisk`        |          | default `"os"`                                             |
| `format`      | one of `raw`, `qcow2`, `iso`  | yes      |                                                            |
| `compression` | one of `none`, `zstd`, `gzip` |          | default `"none"`                                           |
| `sha256`      | string                        | yes      | 1-64 characters; matches `^[0-9A-Fa-f]{64}$`               |
| `sizeBytes`   | integer                       |          | at least 0                                                 |
| `source`      | string                        | yes      | 1-2048 characters                                          |
| `signature`   | string                        |          | 1-1024 characters                                          |
| `signerId`    | string                        |          | 1-128 characters                                           |

## get [#get]

`POST /api/compute/images/get`

Returns one image the tenant owns, or a public one, by its `img_` id.

`NOT_FOUND` when the image does not exist, is deleted, or belongs to another tenant and is not public.

* **Kind:** read
* **Authorization:** `compute:images:read` @ `compute-image`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `imageId`  | string | yes      | at most 56 characters; matches `^img_[0-9a-hjkmnp-tv-z]{26,52}$` |

## list [#list]

`POST /api/compute/images/list`

Returns a page of the tenant's own images and public ones, ordered by `ref`, optionally filtered by `kind` (`os` or `ramdisk`).

`limit` defaults to 50 and is at most 200; pass the returned `cursor` to read the next page.

* **Kind:** read
* **Authorization:** `compute:images:read` @ `compute-tenant` + items
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type                   | Required | Constraints                                                |
| ---------- | ---------------------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `kind`     | one of `os`, `ramdisk` |          |                                                            |
| `limit`    | integer                |          | 1-200; default `50`                                        |
| `cursor`   | string                 |          | at most 256 characters                                     |

## promote [#promote]

`POST /api/compute/images/promote`

Makes a verified image the tenant owns public, so every tenant can read and deploy it.

Decided as `compute:images:publish` on the image. An unsigned image is `INVALID_STATE`; an image the tenant does not own is `NOT_FOUND`. Promoting an image that is already public returns it unchanged. Site agents whose pools pre-warm this ref get their desired state refreshed.

* **Kind:** mutation
* **Authorization:** `compute:images:publish` @ `compute-image`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `imageId`  | string | yes      | at most 56 characters; matches `^img_[0-9a-hjkmnp-tv-z]{26,52}$` |

## resolve [#resolve]

`POST /api/compute/images/resolve`

Resolves image families on a `provider` pool for a launch picker: each family's newest image that is not deprecated, as the pool's provider account sees it. Returns `poolId`, `provider`, `items` (one per family that resolved, in the order asked) and `missing` (each with `family` and `reason`: `notFound` when the family has no image, `deprecated` when its newest image is deprecated, obsolete or deleted). vps serves this method; the rest of the group is the bare-metal catalog.

* **Input.** `families` lists 1 to 50 families; duplicates count once. On Compute Engine a family is a bare name, which resolves in the account's `imageProject` (`ubuntu-2404-lts-amd64`), `<project>/<family>` (`debian-cloud/debian-12`), or `projects/<project>/global/images/family/<family>`.

* **Output.** Each item has `family` as asked; `image`, the reference `instances.create` takes as `imageId` to boot exactly that image (`projects/<project>/global/images/<name>`); `name` and `project`; `architecture` (`x86_64` or `arm64`); `minimumDiskGiB`, the smallest boot disk the image fits on; `guestOsFeatures` (`UEFI_COMPATIBLE`, `GVNIC`, `SEV_CAPABLE`, …, sorted); `loginUser`, the user the instance's `sshKeys` log in as; `deprecation` when a deprecation is scheduled (`replacement` and the `deprecated`, `obsolete` and `deleted` times); `createdAt` and `description`.

* **Caching.** The provider keeps each answer a day per account (a family with no image an hour) and never returns an image whose deprecation time has passed, also from its cache. A failed read is not kept.

* **Allowlist.** A family in a project the account does not allow (the Compute Engine account's `imageProjects`) refuses the whole call with `INVALID_INPUT` (`details.reason` `imageProjectNotAllowed`, with `details.allowed`), and nothing is read. `instances.create` applies the same allowlist to `imageId`.

* **IAM.** Decided as `compute:flavors:read` on the pool: its owner and the tenants it is offered to, who see its flavors. Any other tenant gets `NOT_FOUND`. The pool's provider account is not shown.

* **Errors.** `INVALID_INPUT` for a pool that is not a `provider` pool, `INVALID_STATE` for a provider pool without an account, `UNSUPPORTED` (`details.provider`) when its provider does not resolve image families. Provider failures (`ACCESS_DENIED`, `UNAVAILABLE`, `RATE_LIMITED`) are passed on.

* **Kind:** read

* **Authorization:** `compute:flavors:read` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type           | Required | Constraints                                                       |
| ---------- | -------------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string         | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `families` | list of string | yes      | 1-50 items                                                        |


# API (/docs/reference/api)

> Every method of the compute API, by group: 291 registered, 289 in PLAN.md §9.2.



Every method is `POST /api/compute/{group}/{method}` with a JSON object body, `content-type: application/json` and `X-Better-Compute: 1` (else `CSRF_REJECTED`), at most 64 KiB. Answers are `{ "data": … }` or `{ "error": { "code", "message", "details"? } }`; see [errors](/docs/reference/errors). Plugins are served under `/api/compute/plugins/{id}/{group}/{method}`.

A call is authenticated (except public methods), refused before any database access when the caller is outside the input tenant and the root tenant, and decided once on `action @ resource&#x60;. *+ items* marks list methods, which also filter items one by one.

This reference is generated from @better-compute/server 0.3.0 with every module registered. *Served* counts the methods that release routes; the others are planned (PLAN.md §9.2).

| Group                                                               | Module            | Methods | Served |
| ------------------------------------------------------------------- | ----------------- | ------- | ------ |
| [`billing`](/docs/reference/api/billing)                            | `vps`             | 7       | 7      |
| [`capacity`](/docs/reference/api/capacity)                          | `capacity`        | 4       | 4      |
| [`certificates`](/docs/reference/api/certificates)                  | `vps`             | 4       | 4      |
| [`cloudResources`](/docs/reference/api/cloud-resources)             | `cloud-resources` | 11      | 11     |
| [`commandBatches`](/docs/reference/api/command-batches)             | `quorum`          | 7       | 7      |
| [`commands`](/docs/reference/api/commands)                          | `server`          | 4       | 4      |
| [`dnsRecords`](/docs/reference/api/dns-records)                     | `vps`             | 5       | 5      |
| [`dnsZones`](/docs/reference/api/dns-zones)                         | `vps`             | 4       | 4      |
| [`drivers`](/docs/reference/api/drivers)                            | `server`          | 1       | 1      |
| [`events`](/docs/reference/api/events)                              | `server`          | 1       | 1      |
| [`filesystems`](/docs/reference/api/filesystems)                    | `vps`             | 4       | 4      |
| [`firewallRules`](/docs/reference/api/firewall-rules)               | `vps`             | 4       | 4      |
| [`flavors`](/docs/reference/api/flavors)                            | `vps`             | 4       | 4      |
| [`fleet`](/docs/reference/api/fleet)                                | `fleet`           | 4       | 4      |
| [`images`](/docs/reference/api/images)                              | `provision`       | 5       | 5      |
| [`instances`](/docs/reference/api/instances)                        | `vps`             | 13      | 13     |
| [`inventory`](/docs/reference/api/inventory)                        | `inventory`       | 3       | 3      |
| [`joinRequests`](/docs/reference/api/join-requests)                 | `server`          | 3       | 3      |
| [`joinTokens`](/docs/reference/api/join-tokens)                     | `server`          | 3       | 3      |
| [`kubernetesClusters`](/docs/reference/api/kubernetes-clusters)     | `kubernetes`      | 7       | 7      |
| [`kubernetesNodePools`](/docs/reference/api/kubernetes-node-pools)  | `kubernetes`      | 4       | 4      |
| [`kubernetesSubstrates`](/docs/reference/api/kubernetes-substrates) | `kubernetes`      | 1       | 1      |
| [`loadBalancers`](/docs/reference/api/load-balancers)               | `vps`             | 5       | 5      |
| [`metal`](/docs/reference/api/metal)                                | `metal`           | 15      | 15     |
| [`networks`](/docs/reference/api/networks)                          | `vps`             | 11      | 11     |
| [`nodes`](/docs/reference/api/nodes)                                | `server`          | 14      | 14     |
| [`pools`](/docs/reference/api/pools)                                | `capacity`        | 13      | 13     |
| [`profiles`](/docs/reference/api/profiles)                          | `provision`       | 7       | 7      |
| [`projects`](/docs/reference/api/projects)                          | `server`          | 4       | 4      |
| [`providerAccounts`](/docs/reference/api/provider-accounts)         | `vps`             | 23      | 23     |
| [`providerAddresses`](/docs/reference/api/provider-addresses)       | `vps`             | 6       | 6      |
| [`providerSnapshots`](/docs/reference/api/provider-snapshots)       | `vps`             | 4       | 4      |
| [`providerVolumes`](/docs/reference/api/provider-volumes)           | `vps`             | 8       | 8      |
| [`quotas`](/docs/reference/api/quotas)                              | `capacity`        | 2       | 2      |
| [`releases`](/docs/reference/api/releases)                          | `rollout`         | 2       | 2      |
| [`reservationGrants`](/docs/reference/api/reservation-grants)       | `vps`             | 4       | 4      |
| [`reservations`](/docs/reference/api/reservations)                  | `capacity`        | 7       | 7      |
| [`rollouts`](/docs/reference/api/rollouts)                          | `rollout`         | 6       | 6      |
| [`sandboxSnapshots`](/docs/reference/api/sandbox-snapshots)         | `vps`             | 3       | 3      |
| [`sandboxTemplates`](/docs/reference/api/sandbox-templates)         | `vps`             | 5       | 5      |
| [`sandboxVolumes`](/docs/reference/api/sandbox-volumes)             | `vps`             | 5       | 5      |
| [`sandboxes`](/docs/reference/api/sandboxes)                        | `vps`             | 8       | 8      |
| [`secrets`](/docs/reference/api/secrets)                            | `server`          | 2       | 2      |
| [`siteTokens`](/docs/reference/api/site-tokens)                     | `server`          | 2       | 2      |
| [`sites`](/docs/reference/api/sites)                                | `server`          | 5       | 5      |
| [`system`](/docs/reference/api/system)                              | `server`          | 4       | 4      |
| [`tunnels`](/docs/reference/api/tunnels)                            | `tunnels`         | 1       | 1      |
| [`usage`](/docs/reference/api/usage)                                | `server`          | 6       | 6      |
| [`webhooks`](/docs/reference/api/webhooks)                          | `events`          | 10      | 10     |
| [`workflows`](/docs/reference/api/workflows)                        | `provision`       | 6       | 6      |


# instances (/docs/reference/api/instances)

> Instances are VMs in a VPS pool: hosted pools place them on a host through capacity, and provider pools create them through the pool's provider account. After `create`, every call only changes `desiredState`; the `vps.instances` reconciler converges the instance and publishes `instance.*` events as its `state` moves. Instances are also the sandbox primitive today. See [Instances](/docs/guides/vps/instances).



Instances are VMs in a VPS pool: hosted pools place them on a host through capacity, and provider pools create them through the pool's provider account. After `create`, every call only changes `desiredState`; the `vps.instances` reconciler converges the instance and publishes `instance.*` events as its `state` moves. Instances are also the sandbox primitive today. See [Instances](/docs/guides/vps/instances).

All 13 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                       | Phase | Status     |
| ----------------------- | -------- | --------------------------------------------------- | ----- | ---------- |
| [`create`](#create)     | mutation | `compute:instances:create` @ `compute-instance`     | v0.3  | registered |
| [`list`](#list)         | read     | `compute:instances:read` @ `compute-tenant` + items | v0.3  | registered |
| [`get`](#get)           | read     | `compute:instances:read` @ `compute-instance`       | v0.3  | registered |
| [`start`](#start)       | mutation | `compute:instances:update` @ `compute-instance`     | v0.3  | registered |
| [`stop`](#stop)         | mutation | `compute:instances:update` @ `compute-instance`     | v0.3  | registered |
| [`resize`](#resize)     | mutation | `compute:instances:update` @ `compute-instance`     | v0.3  | registered |
| [`delete`](#delete)     | mutation | `compute:instances:delete` @ `compute-instance`     | v0.3  | registered |
| [`attach`](#attach)     | mutation | `compute:instances:attach` @ `compute-instance`     | v1.0  | registered |
| [`detach`](#detach)     | mutation | `compute:instances:delete` @ `compute-instance`     | v1.0  | registered |
| [`reboot`](#reboot)     | mutation | `compute:instances:update` @ `compute-instance`     | v1.0  | registered |
| [`update`](#update)     | mutation | `compute:instances:update` @ `compute-instance`     | v1.0  | registered |
| [`issued`](#issued)     | read     | `compute:instances:read` @ `compute-fleet`          | v1.0  | registered |
| [`estimate`](#estimate) | read     | `compute:flavors:read` @ `compute-pool`             | v1.0  | registered |

## create [#create]

`POST /api/compute/instances/create`

Admits a VM from a pool, flavor and image after one IAM decision on the draft; returns the instance in `scheduling`, or in `provisioning` with a `nodeId` when a hosted pool placed it at once.

* **Idempotency.** `idempotencyKey` is required. The instance id derives from the tenant, the caller and the key: a retry of a finished request returns the same instance without a second decision, and a different request under the same key is `CONFLICT` with `details.instanceId`.

* **IAM.** `compute:instances:create` is decided on `compute-instance/` plus the draft id, whose attributes (`vcpu`, `memoryGiB`, `gpus`, `imageId`, `imageVerified`, `projectId`, `poolId`, `flavor`, `region` (the pool's site), `provider` (the pool's account, provider pools) and one `labels.<key>` per label, in key order while the row has room: a row holds 64 attributes, and when labels are left out the last slot is `labelsTruncated: true`) policies can test, e.g. `resource.flavor` or `resource.labels.env`. A refusal deletes the draft; a delegated session can get `CONFIRMATION_REQUIRED`.

* **Admission.** For every caller, after the decision and in the same transaction as the charge: the flavor must be one the tenant's offer lists (`pools.offer({ flavors })`, `QUOTA_EXCEEDED` `flavor-not-offered`) and one every quota row it consumes allows (`quotas.set({ flavors })`, `flavor-not-allowed`); then the host's `vps({ admissionPolicy })` runs with the request (pool, flavor, shape, demand, region, provider, labels, owner, decision) and may refuse with any error; then quota is charged: the shape, `instances` (one, where a row exists) and the tenant-level `*` rows. A refusal changes nothing and keeps the draft.

* **Networks.** In a provider pool, `networkId` places the VM on a tenant network (`networks.create`) and `subnetId` in one of its subnets; the instance shows both. The network must be the tenant's and on the pool's provider account, must have been provisioned (it reached `ready` once; a network that went back to `pending` because a subnet or rule was added still takes instances) and must not be being deleted, and the subnet must be on it, in the offer's region and `ready`. A network not provisioned yet, being deleted, or a subnet not ready yet is `INVALID_STATE` with a retry hint. Without `subnetId` the network's only subnet in the offer's region is used; none, or several (the error names them), is `INVALID_INPUT`, as is an unknown network or subnet, one of another tenant or provider account, or a `subnetId` without a `networkId`. The VM is created in the subnet with a per-instance network tag, which the network's firewall rules target. The network and subnet stay until the instance is deleted: `networks.delete` and `networks.removeSubnet` are `INVALID_STATE` while it is on them.

* **Labels.** `labels` (at most 62, so a VM stays within Compute Engine's 64 with better-compute's own two) are kept on the instance in any pool, where `list`'s `labelSelector` matches them. In a provider pool they are also passed to the provider's create, which writes them on the VM and on its boot disk. Keys are a lower-case letter followed by lower-case letters, digits, `_` or `-`, at most 63 characters; values take the same characters, at most 63, and may be empty. Keys starting with `better-compute-` or `bc-` are better-compute's own and `INVALID_INPUT`; the provider also drops any such key and writes its ownership labels last, so they always win. Labels are part of the request: the same key with other labels is `CONFLICT`. `update` changes them later.

* **Workload identity (AWS).** In a hosted pool, `workloadIdentity.awsRoleArn` names the IAM role the instance's workloads assume with no static keys: bcd exchanges the instance's workload token (audience `sts.amazonaws.com`, subject `t:<tenantId>:instance:<id>`) with STS `AssumeRoleWithWebIdentity` and serves the credentials on the guest's AWS-compatible metadata paths. The pool must turn workload identity on (`policy.workloadIdentity: true`) and list the role in `policy.workloadRoles.aws`; otherwise, and in a provider pool, the create is `INVALID_INPUT` (`details.reason` `workloadIdentityOff`, `awsRoleNotAllowed` or `workloadIdentityHostedOnly`). The instance shows the role. The host is sent the role only while the pool still allows it. The role's own trust policy (`better-compute aws workload-role`) limits it to the tenant's instances and to tokens tagged for that role (session tag `bc-role`), which the workload broker adds only for the role the pool allows.

* **Placement.** In a provider pool, `placement` (`{ group, strategy: 'cluster' | 'spread' | 'partition', partition? }`) puts the machine in a provider placement group (an EC2 placement group on AWS): every instance of one tenant's `group` in the pool shares one, under a name vps derives from the tenant, pool and group, so two tenants never share a group. `cluster` keeps machines on one low-latency network segment, `spread` on distinct hardware, `partition` on disjoint racks (`partition` 1–7 picks one). The group is a lower-case letter followed by lower-case letters, digits or `-`, at most 63 characters. Another strategy for a group in use is `CONFLICT`; placement on a hosted pool, on a provider without placement groups or without that strategy (only AWS has them today; such a provider would ignore it, so it is refused rather than claimed), an unknown strategy or a partition without the partition strategy is `INVALID_INPUT`. Placement is part of the request: the same key with another placement is `CONFLICT`. The instance shows its `placement`.

* **Filesystems.** On a tenant network, `filesystems` (`[{ filesystem, path }]`, at most 16) mounts shared [filesystems](/docs/reference/api/filesystems) of that network at boot: each must be the tenant's, on the instance's own network (`INVALID_INPUT` otherwise, also without a `networkId`) and `ready` (`INVALID_STATE` otherwise). A path is absolute, made of plain segments, outside the system directories, and used once (`INVALID_INPUT`). The provider renders the mounts into the machine's user data. The instance shows its `filesystems`; a filesystem it mounts cannot be deleted until the instance is.

* **Name and metadata.** `name` is a display name (1 to 128 characters, no control characters, no surrounding whitespace), not unique. `metadata` is up to 64 string entries (keys a letter or digit, then letters, digits, `.`, `_`, `/` or `-`, at most 128; values at most 1024 characters; 16 KiB in all; `better-compute-` and `bc-` keys reserved). Both are kept by better-compute only, never sent to a provider or host, and both are part of the request. `update` changes them later.

* **Provenance.** `issuedBy` records who issued the instance (`legionedge:admin`: up to 128 letters, digits, `.`, `_`, `:`, `@`, `/`, `+` or `-`). Only the platform (a root caller) sets it: anyone else is `ACCESS_DENIED` with `details.reason: 'issued-by-platform-only'`, before any decision. The instance shows it to everyone who can read it, and `issued` filters on it.

* **Owner.** The instance is owned by the caller, or by `ownerIdentityId`: the platform (a root caller creating in another tenant) issues an instance to a person of that tenant this way, as `attach` does, so it shows in their `scope: 'mine'` list and owner-conditioned policies (`resource.ownerId`) apply to them. Anyone else naming an identity other than their own, and a root caller in its own tenant, is `ACCESS_DENIED` before any decision; a malformed one is `INVALID_INPUT`. When the IAM port can say (`IamPort.isMember`, the embedded better-iam bridge), the owner must also be an active identity of the tenant, else `INVALID_INPUT` and the draft is dropped; other ports check the id's shape only.

* **DNS.** `dns: { zoneId, name?, ipv6? }` publishes the instance in a DNS zone of the tenant ([dnsZones](/docs/reference/api/dns-zones)): an A record `<name>.<zone>` (and an AAAA record with `ipv6: true`) that follows the instance's public address (its assigned reserved address first) and is deleted with the instance, exactly as a [dnsRecords](/docs/reference/api/dns-records) record with `instanceId`. `name` is relative to the zone (`web`, `web.eu`; no apex or wildcard) and defaults to the instance id with `-` for `_`. The records are made in the same transaction as the instance: a zone that is not the tenant's is `NOT_FOUND`, one being deleted `INVALID_STATE`, and a name the zone already holds `CONFLICT`, each refusing the whole create. Besides the decision on the instance, it is decided as `compute:dns:manage` on the zone, on the caller's own credential.

* **Errors.** `NOT_FOUND` for a pool the tenant neither owns nor is offered, an archived pool, a missing or archived flavor, or an image a hosted pool cannot resolve. `INVALID_INPUT` for a pool that is not `vps-host` or `provider`. `QUOTA_EXCEEDED` when the shape does not fit the tenant's quota; `CAPACITY_UNAVAILABLE` when a hosted pool has no free IP address. `INVALID_STATE` for a provider pool without an account, `INVALID_CONFIG` for a hosted pool when `vps()` has no `images`.

* **Events.** `instance.created`, and `instance.scheduled` with the `nodeId` when a hosted instance is placed. When no host fits, the instance stays in `scheduling` with an `Unschedulable` condition and is retried every 30 seconds. In a provider pool, a create the provider refuses for capacity or quota (at once, or after accepting it, as GCE reports a stockout), or refuses as invalid (`INVALID_INPUT`; on Compute Engine, an `imageId` from a project the account does not allow), moves the instance to `error` with a `ProvisionFailed` condition (reason `CAPACITY_UNAVAILABLE`, `QUOTA_EXCEEDED` or `INVALID_INPUT`) and publishes `instance.failed`; it is not retried, and deleting it refunds the quota.

* **Kind:** mutation

* **Authorization:** `compute:instances:create` @ `compute-instance`

* **Module:** `vps`

* **Phase:** v0.3 (registered)

**Input**

| Field                         | Type                                    | Required | Constraints                                                                                                       |
| ----------------------------- | --------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------- |
| `tenantId`                    | string                                  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                                        |
| `poolId`                      | string                                  | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`                                                 |
| `projectId`                   | string                                  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`                                                  |
| `flavor`                      | string                                  | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`                                                      |
| `imageId`                     | string                                  | yes      | at most 200 characters; matches `^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,199}$`                                           |
| `sshKeys`                     | list of string                          |          | 0-32 items; default `[]`                                                                                          |
| `userData`                    | string                                  |          | at most 32768 characters                                                                                          |
| `desiredState`                | one of `running`, `stopped`             |          | default `"running"`                                                                                               |
| `networkId`                   | string                                  |          | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`                                                  |
| `subnetId`                    | string                                  |          | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$`                                                 |
| `labels`                      | map of string                           |          | at most 62 entries; keys match `^[a-z][a-z0-9_-]{0,62}$`                                                          |
| `placement`                   | object                                  |          |                                                                                                                   |
| `placement.group`             | string                                  | yes      | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`                                                           |
| `placement.strategy`          | one of `cluster`, `spread`, `partition` | yes      |                                                                                                                   |
| `placement.partition`         | integer                                 |          | 1-7                                                                                                               |
| `filesystems`                 | list of object                          |          | 0-16 items                                                                                                        |
| `filesystems[].filesystem`    | string                                  | yes      | at most 55 characters; matches `^fs_[0-9a-hjkmnp-tv-z]{26,52}$`                                                   |
| `filesystems[].path`          | string                                  | yes      | at most 200 characters; matches `^(/[A-Za-z0-9._-]+)+$`                                                           |
| `name`                        | string                                  |          | at most 128 characters                                                                                            |
| `metadata`                    | map of string                           |          | at most 64 entries; keys match `^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$`                                              |
| `issuedBy`                    | string                                  |          | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:@/+-]{0,127}$`                                          |
| `ownerIdentityId`             | string                                  |          | at most 128 characters; matches `^[A-Za-z0-9._:@-]{1,128}$`                                                       |
| `workloadIdentity`            | object                                  |          |                                                                                                                   |
| `workloadIdentity.awsRoleArn` | string                                  | yes      | at most 2048 characters; matches `^arn:(aws\|aws-cn\|aws-us-gov):iam::[0-9]{12}:role/[A-Za-z0-9_+=,.@/-]{1,512}$` |
| `dns`                         | object                                  |          |                                                                                                                   |
| `dns.zoneId`                  | string                                  | yes      | at most 56 characters; matches `^pdz_[0-9a-hjkmnp-tv-z]{26,52}$`                                                  |
| `dns.name`                    | string                                  |          | at most 200 characters                                                                                            |
| `dns.ipv6`                    | boolean                                 |          | default `false`                                                                                                   |
| `idempotencyKey`              | string                                  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                                        |
| `wait`                        | object                                  |          |                                                                                                                   |
| `wait.timeoutMs`              | integer                                 | yes      | 0-30000                                                                                                           |

## list [#list]

`POST /api/compute/instances/list`

Returns a page of the tenant's instances, newest first, with a `cursor` when more remain.

`scope` defaults to `mine`, the caller's own instances; `all` returns every instance of the tenant that the caller may read, each one checked for `compute:instances:read` after the page is read, so a page can hold fewer than `limit` items. `limit` is 1 to 200 (default 50); `cursor` is the last instance id of the previous page. Drafts and warm sandbox instances are never listed.

* **Deleted instances.** Left out unless `state: 'deleted'` asks for them, in SQL, before the per-item checks: a deleted instance's access is gone, so it could never pass one. `state: 'deleted'` therefore needs `scope: 'mine'` (the owner's own, with no per-item check); with `scope: 'all'` it is `INVALID_INPUT`.
* **Filters.** `state`, `poolId`, `projectId` and `labelSelector` all filter in SQL. A label selector is comma-separated requirements that must all hold: `key=value` (or `==`), `key!=value` (also true without the key), `key` (has it), `!key` (lacks it), `key in (a,b)` and `key notin (a,b)` (also true without the key); at most 20 requirements and 1024 characters, keys and values as the label rules allow, else `INVALID_INPUT`.

Each instance shows its `name`, `metadata`, `labels`, `issuedBy` and `createdAt` (epoch ms), and, for a provider machine, the `disks` the provider last reported (boot disk first: `type`, `sizeGiB`, `boot`), which is what `disk.<type>.gib.hours` meters.

* **Kind:** read
* **Authorization:** `compute:instances:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field           | Type                                                                                                                                                        | Required | Constraints                                                       |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`      | string                                                                                                                                                      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `scope`         | one of `mine`, `all`                                                                                                                                        |          | default `"mine"`                                                  |
| `state`         | one of `pending`, `scheduling`, `provisioning`, `starting`, `running`, `stopped`, `rebooting`, `resizing`, `deleting`, `deleted`, `error`, `lost`, `paused` |          |                                                                   |
| `poolId`        | string                                                                                                                                                      |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`     | string                                                                                                                                                      |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `labelSelector` | string                                                                                                                                                      |          | at most 1024 characters                                           |
| `limit`         | integer                                                                                                                                                     |          | 1-200                                                             |
| `cursor`        | string                                                                                                                                                      |          | at most 57 characters; matches `^inst_[0-9a-z]{26,52}$`           |

## get [#get]

`POST /api/compute/instances/get`

Returns one instance of the tenant, deleted instances included.

Drafts and instances of other tenants are `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:instances:read` @ `compute-instance`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## start [#start]

`POST /api/compute/instances/start`

Sets `desiredState` to `running` and wakes the reconciler; returns the instance before anything has changed on the host or provider.

An instance in `error` that has a host or a provider machine moves to `starting` and is tried again. An instance whose provider create failed (a `ProvisionFailed` condition) is `INVALID_STATE` with `details: { condition: 'ProvisionFailed', reason }` and is left unchanged: delete it and create again. An instance being deleted is `INVALID_STATE`. Publishes `instance.starting`.

* **Kind:** mutation
* **Authorization:** `compute:instances:update` @ `compute-instance`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## stop [#stop]

`POST /api/compute/instances/stop`

Sets `desiredState` to `stopped` and wakes the reconciler; returns the instance before anything has changed on the host or provider.

An instance being deleted is `INVALID_STATE`. Publishes `instance.stopping`.

* **Kind:** mutation
* **Authorization:** `compute:instances:update` @ `compute-instance`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## resize [#resize]

`POST /api/compute/instances/resize`

Moves a stopped hosted instance to another flavor of its pool, in place on the same host; returns the instance `stopped` with the new flavor.

The new flavor must pass the flavor allowlists of the offer and of every quota row the instance consumes (`QUOTA_EXCEEDED` `flavor-not-offered` or `flavor-not-allowed`), then the host's `admissionPolicy` sees the growth. Growth is charged to quota first and shrinkage refunded after; if the host lacks room the charge is undone and the call is `CAPACITY_UNAVAILABLE` with `details.resource`. Other errors: `UNSUPPORTED` for a provider instance or a flavor with a different GPU count, `INVALID_STATE` when the instance is not stopped (both `state` and `desiredState`) or holds no allocation, `NOT_FOUND` for a missing or archived flavor, `QUOTA_EXCEEDED` when the growth does not fit. Publishes `instance.resized`.

* **Kind:** mutation
* **Authorization:** `compute:instances:update` @ `compute-instance`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `flavor`     | string | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`      |

## delete [#delete]

`POST /api/compute/instances/delete`

Sets `desiredState` to `deleted` and wakes the reconciler, which releases the allocation, IP lease and quota; returns the instance at once.

The call is idempotent: an instance already `deleted` is returned unchanged, and stays readable through `get`. Publishes `instance.deleting`; `instance.deleted` follows when everything is released.

* **Kind:** mutation
* **Authorization:** `compute:instances:delete` @ `compute-instance`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## attach [#attach]

`POST /api/compute/instances/attach`

Makes a provider machine that already runs an instance of a tenant, without recreating it; returns the instance in `scheduling` with `origin: 'attached'`. The `vps.instances` reconciler then writes the provider's ownership label (`claim`) and follows the machine like any provider instance.

* **Input.** `poolId` is a `provider` pool owned by or offered to `tenantId` and backed by the machine's provider account. `externalId` is the machine as `providerAccounts.machines` lists it, which discovery must have seen and which must be free. `ownerIdentityId` defaults to the caller, so an operator can hand the instance to a person of the tenant (it then shows in their `scope: 'mine'` list). As for `create`, only a root caller attaching into another tenant names someone else (`ACCESS_DENIED` otherwise, before any decision), and the owner must be an active identity of the tenant when the IAM port can say (`INVALID_INPUT`). `onDelete` is `detach` (the default: deleting releases the machine, which keeps running) or `delete` (deleting deletes the machine).

* **Networks.** An attached machine keeps the network it is on. If its NIC is on one of the tenant's networks of the same provider account that is not deleted, whatever state it is in (`pending` included), the instance is linked to it (`networkId`, and `subnetId` when the NIC names one of its subnets), so its firewall rules apply and the network cannot be deleted from under it. `networkId` asserts that: the machine must already be on that network, otherwise `INVALID_STATE` with `details.hint: 'attached VMs keep their existing network'` (GCE cannot move a NIC to another VPC without a stop). A network that does not exist, is not the tenant's or is on another account is `INVALID_INPUT`; one being deleted is `INVALID_STATE` (`details.networkId`), also when the machine's NIC is on it and no `networkId` was given: attach neither links nor ignores a network being torn down.

* **Identity.** `name`, `metadata` and `labels` follow the rules of `create` and are part of the request; the labels are written on the machine (`ProviderImpl.setLabels`) once it is claimed, next to the labels it already carries. `issuedBy` is the platform's only, as for `create` (`ACCESS_DENIED` otherwise, before any decision).

* **Billing.** `billing: 'none'` writes off the machine: it opens no usage interval, so nothing of it is metered, stopped or running. Only the platform (a root caller) may pass it; any other caller gets `ACCESS_DENIED` before any decision and nothing is written. `billing: 'metered'`, or omitting it, leaves the decision to the pool's billing mode (`policy.billing.mode`, else metered, or a platform fee for a tenant's own cloud account). It is part of the request, so a retry with the same key but another `billing` is `CONFLICT`.

* **Budgets.** A billed attach (the resulting mode is `metered` or `platform-fee`, so not `billing: 'none'` and not a pool whose mode is `none`) is checked against the tenant's budgets after the IAM decision, as `create` is: when the pool asks for it (`policy.admission.budgetCheck: 'meter'`) or the tenant's offer is `stop-new`, `drain` or `suspend`, a spent budget is `SPEND_LIMIT_REACHED` and nothing is written. A root caller attaching into a tenant is checked against that tenant's budgets, not root's. A suspended offer, or an offer whose `allowedRegions` leave out the pool's region, refuses every attach (`SPEND_LIMIT_REACHED`, `ACCESS_DENIED`), billed or not.

* **Shape.** The flavor is the machine's own: its machine type, vCPU, memory and GPU count, with no disk. It is charged to the tenant's quota on the pool, as a create on a provider pool is, `instances` included.

* **Admission.** The machine is judged by the pool's flavor whose offer is its type (`<zone>/<type>` in its zone, else `<type>`, else any zone's): the offer's and the quota rows' flavor allowlists apply to that flavor as on `create`, and a machine whose type no flavor offers is refused by any allowlist (`details.flavor: null`). The host's `admissionPolicy` runs too, also for label-driven auto-attach (without a decision).

* **Idempotency.** `idempotencyKey` is required. The instance id derives from the tenant, the caller and the key: a retry returns the same instance, and a different request under the same key is `CONFLICT` with `details.instanceId`.

* **IAM.** `compute:instances:attach` is decided on `compute-instance/` plus the instance id, whose attributes (`vcpu`, `memoryGiB`, `gpus`, `poolId`, `flavor` (the mapped flavor, when one is), `region`, `provider`) policies can test. Only the platform (root callers) attaches machines of an account its own tenant does not own; a tenant attaches from its own accounts, and a tenant the pool is offered to attaches, into itself, a machine bound to the pool (`pools.bindMachines`). Anything else is `ACCESS_DENIED`, whether or not the machine exists.

* **Pools.** A machine bound to a pool is attached into that pool only: another pool is `CONFLICT` with `details.poolId`. Detaching or deleting the instance leaves the binding, so the machine returns to its pool. For Compute Engine accounts (`gcp`), a machine whose region (or its zone's) is not the region of the pool's site is `INVALID_INPUT` with `details.machineRegion` and `details.poolRegion`; other providers, and machines without a region or zone, are not compared.

* **Errors.** `NOT_FOUND` for a pool the tenant neither owns nor is offered, or a machine discovery has not seen or marked gone (run `providerAccounts.discover`). `CONFLICT` with `details.instanceId` when another instance holds the machine, with `details.claimedBy` when discovery saw its ownership label name another instance, and when the pool is backed by a different provider account than the machine's. `INVALID_INPUT` for a pool that is not a provider pool, `QUOTA_EXCEEDED` when the machine does not fit the quota; nothing is written then.

* **Events.** `instance.attached` with `instanceId`, `accountId`, `externalId`, `poolId` and `onDelete`.

* **Ownership.** Once claimed, every read of the machine must show this instance's label. A machine whose label is gone or names another instance is never adopted, recovered or acted on again: the instance gets an `OwnershipLost` condition (an `AttachConflict` one when the first claim was refused), `start`, `stop` and `reboot` are `INVALID_STATE`, and deleting it finishes as a detach without any provider call.

* **Kind:** mutation

* **Authorization:** `compute:instances:attach` @ `compute-instance`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field             | Type                      | Required | Constraints                                                              |
| ----------------- | ------------------------- | -------- | ------------------------------------------------------------------------ |
| `tenantId`        | string                    | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`               |
| `poolId`          | string                    | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`        |
| `externalId`      | string                    | yes      | at most 256 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,255}$`  |
| `projectId`       | string                    |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`         |
| `ownerIdentityId` | string                    |          | at most 128 characters; matches `^[A-Za-z0-9._:@-]{1,128}$`              |
| `onDelete`        | one of `detach`, `delete` |          | default `"detach"`                                                       |
| `name`            | string                    |          | at most 128 characters                                                   |
| `metadata`        | map of string             |          | at most 64 entries; keys match `^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$`     |
| `labels`          | map of string             |          | at most 62 entries; keys match `^[a-z][a-z0-9_-]{0,62}$`                 |
| `issuedBy`        | string                    |          | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:@/+-]{0,127}$` |
| `networkId`       | string                    |          | at most 57 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`         |
| `billing`         | one of `metered`, `none`  |          |                                                                          |
| `idempotencyKey`  | string                    | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`               |

## detach [#detach]

`POST /api/compute/instances/detach`

Deletes an attached instance without deleting its machine; returns the instance in `deleting`. The reconciler removes the provider's ownership label (`release`), refunds the quota and drops the instance's access, and the machine keeps running in the provider account, free to be attached again.

* **Semantics.** `onDelete` becomes `detach`, whatever the attach chose. `delete` on an attached instance does the same when its `onDelete` is `detach` (the default for attach); with `onDelete: 'delete'` the reconciler first marks the machine as managed by better-compute and then deletes it. A machine whose label names another instance is never deleted: the instance fails with an `AttachConflict` condition.

* **Errors.** `INVALID_STATE` for an instance better-compute created (`origin: 'created'`): detaching it would leave a machine running that no one is billed for, so delete it instead. Instances of other tenants, or none, are `NOT_FOUND`. A `deleted` instance is returned unchanged.

* **IAM.** Decided as `compute:instances:delete` on the instance.

* **Events.** `instance.deleting`; `instance.detached` (not `instance.deleted`) follows when the label is released.

* **Kind:** mutation

* **Authorization:** `compute:instances:delete` @ `compute-instance`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## reboot [#reboot]

`POST /api/compute/instances/reboot`

Asks for a restart of a running provider instance's machine; returns the instance, still `running`. The `vps.instances` reconciler restarts the machine once, gracefully (`graceful-restart`) or, with `hard: true`, by a reset, and publishes `instance.rebooted` with `hard`.

* **Semantics.** A second reboot asked for while the first is pending is one restart; one asked for after it restarts the machine again. A provider without power control drops the request and records a `ProviderError` condition.

* **Errors.** `INVALID_STATE` when the instance is not running, or is being stopped or deleted. `UNSUPPORTED` for instances of hosted pools, for now. Instances of other tenants, or none, are `NOT_FOUND`.

* **IAM.** Decided as `compute:instances:update` on the instance.

* **Kind:** mutation

* **Authorization:** `compute:instances:update` @ `compute-instance`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field        | Type    | Required | Constraints                                                       |
| ------------ | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `instanceId` | string  | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `hard`       | boolean |          | default `false`                                                   |

## update [#update]

`POST /api/compute/instances/update`

Renames an instance and replaces its metadata and labels; the platform records or corrects its provenance. Returns the instance.

* **Input.** At least one of `name`, `metadata`, `labels` and `issuedBy`, with the rules of `create`; `null` clears a field, and `{}` clears metadata or labels. Metadata and labels are replaced as a whole, not merged. `expectedVersion` makes the update conditional: an instance at another `version` is `CONFLICT` with `details.version`.

* **IAM.** `compute:instances:update` on the instance. `issuedBy` is the platform's: from anyone but a root caller it is `ACCESS_DENIED` (`details.reason: 'issued-by-platform-only'`) before any decision.

* **Labels on the machine.** In a provider pool, new labels are written on the VM and its boot disk by the `vps.instances` reconciler (`ProviderImpl.setLabels`), which the update wakes: every wanted key is set, every key the provider last got that is no longer wanted is removed, labels someone else put on the machine stay, and better-compute's own (`better-compute-*`) are never touched. An attached machine gets them once claimed. A write the provider refuses leaves a `LabelsNotApplied` condition (the error's code as `reason`) and is tried again at the next poll; the instance keeps running. Hosted instances have nothing to write.

* **Semantics.** An update that changes nothing returns the instance as it is, with no new version and no event. Otherwise the version moves on and `instance.updated` is published with `changed` (the fields that changed). Nothing else about the instance changes.

* **Errors.** `NOT_FOUND` for an instance of another tenant, or none; `INVALID_STATE` for an instance being deleted or deleted, a sandbox's instance (use `sandboxes.update`) or a warm one; `INVALID_INPUT` for an empty update, an unknown field or a value the rules refuse.

* **Kind:** mutation

* **Authorization:** `compute:instances:update` @ `compute-instance`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field             | Type        | Required | Constraints                                                              |
| ----------------- | ----------- | -------- | ------------------------------------------------------------------------ |
| `tenantId`        | string      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`               |
| `instanceId`      | string      | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$`        |
| `name`            | string,null |          | at most 128 characters                                                   |
| `metadata`        | object,null |          | at most 64 entries; keys match `^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$`     |
| `labels`          | object,null |          | at most 62 entries; keys match `^[a-z][a-z0-9_-]{0,62}$`                 |
| `issuedBy`        | string,null |          | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:@/+-]{0,127}$` |
| `expectedVersion` | integer     |          | 1-9007199254740991                                                       |

## issued [#issued]

`POST /api/compute/instances/issued`

The platform's view of the instances it issued: a page of instances in tenants other than the root tenant, on pools the root tenant owns (created there, or attached into them), newest first, with a `cursor` when more remain.

* **IAM.** `compute:instances:read` on `compute-fleet/global`, which resolves only in the root tenant: any other caller is `ACCESS_DENIED`. `INVALID_CONFIG` when the IAM port names no root tenant.

* **Filters.** `tenantId` keeps one tenant's instances (here it is a filter, not the scope), `poolId` one pool's, `state` one state (without it, deleted instances are left out), `issuedBy` the instances whose recorded provenance is that value (`*`: any recorded provenance), and `labelSelector` as `list` takes it. `limit` is 1 to 500 (default 100); `cursor` is the last instance id of the previous page.

* **Answer.** Instances as `get` shows them, with their `tenantId` and `ownerIdentityId`. It only reads.

* **Kind:** read

* **Authorization:** `compute:instances:read` @ `compute-fleet`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field           | Type                                                                                                                                                        | Required | Constraints                                                       |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`      | string                                                                                                                                                      |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`        | string                                                                                                                                                      |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `state`         | one of `pending`, `scheduling`, `provisioning`, `starting`, `running`, `stopped`, `rebooting`, `resizing`, `deleting`, `deleted`, `error`, `lost`, `paused` |          |                                                                   |
| `issuedBy`      | string                                                                                                                                                      |          | at most 128 characters                                            |
| `labelSelector` | string                                                                                                                                                      |          | at most 1024 characters                                           |
| `limit`         | integer                                                                                                                                                     |          | 1-500                                                             |
| `cursor`        | string                                                                                                                                                      |          | at most 57 characters; matches `^inst_[0-9a-z]{26,52}$`           |

## estimate [#estimate]

`POST /api/compute/instances/estimate`

Returns what one hour of a flavor would cost the tenant, before it creates anything: `{ estimate }` with `{ perHourMicros, currency, lines }` as `flavors.list` shows it, or `{}` when the IAM port cannot quote, the pool is not a provider pool or IAM cannot answer.

* **Input.** `tenantId`, `poolId` and `flavor`: the fields of a create that decide its price. `NOT_FOUND` for a pool the tenant neither owns nor is offered, or a missing or archived flavor; `INVALID_INPUT` for a pool that is not a VPS pool.

* **Decided as a read of the pool's flavors** (`compute:flavors:read` on the pool), not as a create: whoever can list the pool's flavors can ask, and holding only that action is not enough to create (`create` is still `compute:instances:create`). It is a read: no instance or draft, no quota, no authz projection row, no provider call and no audit row.

* **Same figures as `flavors.list`**: the lines are the meters an instance of the flavor would record per hour, priced for the tenant with its negotiated prices, and cached five minutes per tenant and pool. It does not check quota, budgets, the image or the network.

* **Kind:** read

* **Authorization:** `compute:flavors:read` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `flavor`   | string | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`      |


# inventory (/docs/reference/api/inventory)

> A node's hardware and software inventory as the daemon reports it, kept as an append-only history of snapshots: a new snapshot is stored each time the node's reported section set changes. Every method decides `compute:nodes:read` on `compute-node/{nodeId}` and only answers for nodes the calling tenant owns. See [Inventory and health](/docs/guides/nodes/inventory-and-health).



A node's hardware and software inventory as the daemon reports it, kept as an append-only history of snapshots: a new snapshot is stored each time the node's reported section set changes. Every method decides `compute:nodes:read` on `compute-node/{nodeId}` and only answers for nodes the calling tenant owns. See [Inventory and health](/docs/guides/nodes/inventory-and-health).

All 3 methods are served by @better-compute/server 0.3.0.

| Method                | Kind | Authorization                         | Phase | Status     |
| --------------------- | ---- | ------------------------------------- | ----- | ---------- |
| [`get`](#get)         | read | `compute:nodes:read` @ `compute-node` | v0.1  | registered |
| [`history`](#history) | read | `compute:nodes:read` @ `compute-node` | v0.2  | registered |
| [`diff`](#diff)       | read | `compute:nodes:read` @ `compute-node` | v0.2  | registered |

## get [#get]

`POST /api/compute/inventory/get`

Returns the node's latest inventory snapshot: its hash, its sections and when it was collected.

`NOT_FOUND` when the node has no inventory yet or is not owned by the tenant.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `inventory`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                |
| ---------- | ------ | -------- | ------------------------------------------ |
| `tenantId` | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `nodeId`   | string | yes      | matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## history [#history]

`POST /api/compute/inventory/history`

Lists the node's inventory snapshots newest first; returns a page with a cursor.

`limit` is at most 100 (20 by default). The cursor is the id of the last snapshot returned. `NOT_FOUND` when the node is not owned by the tenant.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `inventory`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                |
| ---------- | ------- | -------- | ------------------------------------------ |
| `tenantId` | string  | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `nodeId`   | string  | yes      | matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `limit`    | integer |          | 1-100                                      |
| `cursor`   | string  |          | matches `^inv_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## diff [#diff]

`POST /api/compute/inventory/diff`

Compares two of the node's snapshots, named by snapshot id or hash, and returns the changes between them with their stable device ids (GPU UUID, disk WWN or MAC).

* Without `to`, the latest snapshot is used; without `from`, the snapshot before `to`. A node with a single snapshot diffs against nothing: `from` is empty and there are no changes.

* Change types are `gpu.missing`, `gpu.added`, `disk.replaced`, `driver.changed` and `mig.layout.changed`, each with `before` and `after` values where they apply.

* `NOT_FOUND` when the node has no inventory, a named snapshot does not exist for the node, or the node is not owned by the tenant.

* **Kind:** read

* **Authorization:** `compute:nodes:read` @ `compute-node`

* **Module:** `inventory`

* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                               |
| ---------- | ------ | -------- | --------------------------------------------------------- |
| `tenantId` | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`                        |
| `nodeId`   | string | yes      | matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$`                |
| `from`     | string |          | matches `^(inv_[0-9a-hjkmnp-tv-z]{26,52}\|[0-9a-f]{64})$` |
| `to`       | string |          | matches `^(inv_[0-9a-hjkmnp-tv-z]{26,52}\|[0-9a-f]{64})$` |


# joinRequests (/docs/reference/api/join-requests)

> Join requests are the approval queue for joins an attestor deferred: a machine with placeholder hardware identifiers, a machine the `discovery` attestor did not know, or a TPM whose endorsement key is on no allow-list. An approval admits exactly one join, which the machine's retry with the same CSR consumes. The group is decided on `compute-pool/{poolId}` with the join-token actions. See [Joining nodes](/docs/guides/nodes/joining).



Join requests are the approval queue for joins an attestor deferred: a machine with placeholder hardware identifiers, a machine the `discovery` attestor did not know, or a TPM whose endorsement key is on no allow-list. An approval admits exactly one join, which the machine's retry with the same CSR consumes. The group is decided on `compute-pool/{poolId}` with the join-token actions. See [Joining nodes](/docs/guides/nodes/joining).

All 3 methods are served by @better-compute/server 0.3.0.

| Method                | Kind     | Authorization                                 | Phase | Status     |
| --------------------- | -------- | --------------------------------------------- | ----- | ---------- |
| [`list`](#list)       | read     | `compute:join-tokens:read` @ `compute-pool`   | v0.2  | registered |
| [`approve`](#approve) | mutation | `compute:join-tokens:create` @ `compute-pool` | v0.2  | registered |
| [`reject`](#reject)   | mutation | `compute:join-tokens:create` @ `compute-pool` | v0.2  | registered |

## list [#list]

`POST /api/compute/joinRequests/list`

Lists the join requests a pool can decide: its own, and its site's requests that no pool has claimed yet.

Decided as `compute:join-tokens:read`. `state` defaults to `pending`; `all` lists every state. A pending or approved request past its expiry reads as `expired`, whether or not retention has marked it yet. Pages hold up to 200 requests (100 by default). A pool the tenant does not own is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:join-tokens:read` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type                                                                   | Required | Constraints                                                           |
| ---------- | ---------------------------------------------------------------------- | -------- | --------------------------------------------------------------------- |
| `tenantId` | string                                                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`   | string                                                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `state`    | one of `pending`, `approved`, `rejected`, `expired`, `consumed`, `all` |          | default `"pending"`                                                   |
| `limit`    | integer                                                                |          | 1-200; default `100`                                                  |
| `cursor`   | string                                                                 |          | at most 64 characters; matches `^jreq_[0-9a-hjkmnp-tv-z]{26,52}$`     |

## approve [#approve]

`POST /api/compute/joinRequests/approve`

Approves a pending join request, assigns it to the pool and returns it with a new expiry of `ttlSeconds` (60 to 86,400, one hour by default).

Decided as `compute:join-tokens:create`. Approving a request that is already approved for this pool returns the approval unchanged. Any other decided state (`rejected`, `expired`, `consumed`) is `INVALID_STATE`, and so is an archived pool. A pool the tenant does not own, or a request the pool cannot see, is `NOT_FOUND`.

Publishes `security.join-request.approved` with the new `expiresAt`.

* **Kind:** mutation
* **Authorization:** `compute:join-tokens:create` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type    | Required | Constraints                                                           |
| ------------ | ------- | -------- | --------------------------------------------------------------------- |
| `tenantId`   | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`     | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `requestId`  | string  | yes      | at most 64 characters; matches `^jreq_[0-9a-hjkmnp-tv-z]{26,52}$`     |
| `ttlSeconds` | integer |          | 60-86400; default `3600`                                              |
| `reason`     | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`           |

## reject [#reject]

`POST /api/compute/joinRequests/reject`

Rejects a pending join request, or withdraws an approval the machine has not used yet, and returns the request.

Decided as `compute:join-tokens:create`. Rejecting a request that is already rejected returns it unchanged. An `expired` or `consumed` request is `INVALID_STATE`. A later retry with the request's CSR fails.

Publishes `security.join-request.rejected`, with `reason` when one is given.

* **Kind:** mutation
* **Authorization:** `compute:join-tokens:create` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                           |
| ----------- | ------ | -------- | --------------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`    | string | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `requestId` | string | yes      | at most 64 characters; matches `^jreq_[0-9a-hjkmnp-tv-z]{26,52}$`     |
| `reason`    | string |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`           |


# joinTokens (/docs/reference/api/join-tokens)

> Join tokens are single-purpose credentials that let a limited number of daemons join one pool for a limited time. The bearer form `bcjt_<id>.<secret>` is returned once, when the token is created; the control plane stores only the SHA-256 of the secret. Every method is decided on `compute-pool/{poolId}`. See [Joining nodes](/docs/guides/nodes/joining).



Join tokens are single-purpose credentials that let a limited number of daemons join one pool for a limited time. The bearer form `bcjt_<id>.<secret>` is returned once, when the token is created; the control plane stores only the SHA-256 of the secret. Every method is decided on `compute-pool/{poolId}`. See [Joining nodes](/docs/guides/nodes/joining).

All 3 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                 | Phase | Status     |
| ------------------- | -------- | --------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:join-tokens:create` @ `compute-pool` | v0.1  | registered |
| [`list`](#list)     | read     | `compute:join-tokens:read` @ `compute-pool`   | v0.1  | registered |
| [`revoke`](#revoke) | mutation | `compute:join-tokens:revoke` @ `compute-pool` | v0.1  | registered |

## create [#create]

`POST /api/compute/joinTokens/create`

Mints a role-`node` join token for one pool and returns its record with the bearer `token`, which is never shown again.

Only the pool's owner tenant can mint tokens for it; another tenant gets `NOT_FOUND`. A pool whose machines run no daemon (its `agent` is not `bcd`) answers `INVALID_STATE`. `expect` pins the token to one machine and must name at least one identifier, otherwise the call is `INVALID_INPUT`. `ttlSeconds` runs from 60 to 86,400 and `maxUses` defaults to 1.

`hosted: 'kubernetes'` marks a token for `bcd` running as a DaemonSet on a Kubernetes cluster. That cluster's admins have root on its machines, so a node joining with it becomes `tenant-controlled`, with a `node-tenant` certificate that never receives platform secrets. The exception is a pool whose `policy.kubernetes.platformOperated` is true, and only when the platform owns that pool (it is the pool's site's owner, on a site that is not `custom`): anyone who can edit a pool's policy could otherwise vouch for their own cluster, so set it only on pools whose clusters the platform runs. The token decides this, never the machine's claims, and it only ever lowers `platform` trust: a tenant's own site stays `tenant-owned`.

The token id derives from the tenant, the calling identity and `idempotencyKey`. A retry with the same key returns the stored record without `token` and without a new IAM decision. The same key with a different pool, `expect`, `allowReclaim` or `hosted` is `CONFLICT`.

Publishes `security.join-token.created`, whose `pinned` field says whether the token has an `expect` binding; `hosted` is included when set.

* **Kind:** mutation
* **Authorization:** `compute:join-tokens:create` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field                | Type                | Required | Constraints                                                             |
| -------------------- | ------------------- | -------- | ----------------------------------------------------------------------- |
| `tenantId`           | string              | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `poolId`             | string              | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`   |
| `maxUses`            | integer             |          | 1-10000; default `1`                                                    |
| `ttlSeconds`         | integer             | yes      | 60-86400                                                                |
| `expect`             | object              |          |                                                                         |
| `expect.productUuid` | string              |          | at most 64 characters; matches `^[0-9A-Fa-f-]{8,64}$`                   |
| `expect.serial`      | string              |          | at most 128 characters; matches `^[A-Za-z0-9._:/+-]{1,128}$`            |
| `expect.mac`         | string              |          | at most 17 characters; matches `^[0-9A-Fa-f]{2}(?::[0-9A-Fa-f]{2}){5}$` |
| `expect.gpuUuid`     | string              |          | at most 68 characters; matches `^GPU-[0-9A-Fa-f-]{8,64}$`               |
| `allowReclaim`       | boolean             |          | default `false`                                                         |
| `hosted`             | one of `kubernetes` |          |                                                                         |
| `idempotencyKey`     | string              | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`          |

## list [#list]

`POST /api/compute/joinTokens/list`

Lists a pool's join tokens in id order, never with their secrets.

By default only live tokens appear: not revoked, with uses left and not expired. `includeInactive: true` adds used-up, expired and revoked tokens. Pages hold up to 200 tokens (100 by default); pass the returned `cursor` to read the next page.

* **Kind:** read
* **Authorization:** `compute:join-tokens:read` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field             | Type    | Required | Constraints                                                           |
| ----------------- | ------- | -------- | --------------------------------------------------------------------- |
| `tenantId`        | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`          | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `includeInactive` | boolean |          | default `false`                                                       |
| `limit`           | integer |          | 1-200; default `100`                                                  |
| `cursor`          | string  |          | at most 64 characters; matches `^jt_[0-9a-hjkmnp-tv-z]{26,52}$`       |

## revoke [#revoke]

`POST /api/compute/joinTokens/revoke`

Revokes a join token at once and returns its record with `revokedAt` set.

Revoking a token that is already revoked returns it unchanged and publishes nothing. A token that does not exist in the tenant and pool is `NOT_FOUND`. A first revocation publishes `security.join-token.revoked`.

* **Kind:** mutation
* **Authorization:** `compute:join-tokens:revoke` @ `compute-pool`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                           |
| ---------- | ------ | -------- | --------------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `tokenId`  | string | yes      | at most 64 characters; matches `^jt_[0-9a-hjkmnp-tv-z]{26,52}$`       |


# kubernetesClusters (/docs/reference/api/kubernetes-clusters)

> Kubernetes clusters are managed Kubernetes clusters (Google Kubernetes Engine, Standard and Autopilot) on a provider account, served by the `@better-compute/kubernetes` module. The provider runs the control plane; better-compute never runs one itself. A cluster belongs to one provider pool, so to that pool's account, and the account's provider must provide managed Kubernetes (a provider that does not is `UNSUPPORTED`). Two hosting models use the same calls: a platform account's pool offered to the tenant (the cluster lives in the platform's cloud project), or a pool on the tenant's own account (the cluster lives in the tenant's project). The calls record what the tenant wants and return at once; the `kubernetes.clusters` reconciler then creates the cluster at the provider, its node pools once the control plane is up, applies changes in place and deletes what the tenant deleted. `state` shows where it is: `pending`, `provisioning`, `ready` (the control plane and every node pool are ready), `updating`, `deleting`, `deleted` or `error` (with a `message`). The reconciler publishes `kubernetes.cluster.ready` the first time, `kubernetes.cluster.updated` when it is ready again after a change or the provider moved it to another version (a release-channel upgrade), `kubernetes.cluster.error` and `kubernetes.cluster.deleted`. Clusters still converging are looked at every 30 seconds; ready and failed ones are refreshed every five minutes, which retries an error and picks up what changed at the provider.



Kubernetes clusters are managed Kubernetes clusters (Google Kubernetes Engine, Standard and Autopilot) on a provider account, served by the `@better-compute/kubernetes` module. The provider runs the control plane; better-compute never runs one itself. A cluster belongs to one provider pool, so to that pool's account, and the account's provider must provide managed Kubernetes (a provider that does not is `UNSUPPORTED`). Two hosting models use the same calls: a platform account's pool offered to the tenant (the cluster lives in the platform's cloud project), or a pool on the tenant's own account (the cluster lives in the tenant's project). The calls record what the tenant wants and return at once; the `kubernetes.clusters` reconciler then creates the cluster at the provider, its node pools once the control plane is up, applies changes in place and deletes what the tenant deleted. `state` shows where it is: `pending`, `provisioning`, `ready` (the control plane and every node pool are ready), `updating`, `deleting`, `deleted` or `error` (with a `message`). The reconciler publishes `kubernetes.cluster.ready` the first time, `kubernetes.cluster.updated` when it is ready again after a change or the provider moved it to another version (a release-channel upgrade), `kubernetes.cluster.error` and `kubernetes.cluster.deleted`. Clusters still converging are looked at every 30 seconds; ready and failed ones are refreshed every five minutes, which retries an error and picks up what changed at the provider.

The provider-side name of a cluster is `bc-k-` and 24 characters of its id, and it carries the label `better-compute-cluster` with the cluster id. A cluster bills `kubernetes.cluster.hours` (one per hour, to the tenant) from the moment the provider first accepted it until it is deleted, on pools whose billing mode is `metered` or `platform-fee`. Its nodes are not metered here: the provider bills them to the account, and on a platform account they reach the tenant through the account's cost pass-through rules.

All 7 methods are served by @better-compute/server 0.3.0.

| Method                        | Kind     | Authorization                                               | Phase | Status     |
| ----------------------------- | -------- | ----------------------------------------------------------- | ----- | ---------- |
| [`create`](#create)           | mutation | `compute:kubernetes:create` @ `compute-kubernetes-cluster`  | v1.0  | registered |
| [`list`](#list)               | read     | `compute:kubernetes:read` @ `compute-tenant` + items        | v1.0  | registered |
| [`get`](#get)                 | read     | `compute:kubernetes:read` @ `compute-kubernetes-cluster`    | v1.0  | registered |
| [`update`](#update)           | mutation | `compute:kubernetes:update` @ `compute-kubernetes-cluster`  | v1.0  | registered |
| [`delete`](#delete)           | mutation | `compute:kubernetes:delete` @ `compute-kubernetes-cluster`  | v1.0  | registered |
| [`credentials`](#credentials) | mutation | `compute:kubernetes:connect` @ `compute-kubernetes-cluster` | v1.0  | registered |
| [`versions`](#versions)       | read     | `compute:kubernetes:read` @ `compute-tenant`                | v1.0  | registered |

## create [#create]

`POST /api/compute/kubernetesClusters/create`

Records a cluster and returns it `pending`, with its node pools.

* The pool must be visible to the tenant (its own, or offered to it), a `provider` pool, and its account's provider must provide managed Kubernetes; otherwise `UNSUPPORTED` (`NOT_FOUND` for a pool the tenant cannot see).
* `location` is a zone the account is configured with (a zonal cluster) or the region of one (a regional cluster); anything else is `INVALID_INPUT` with the allowed zones and regions in `details`.
* `mode`: `standard` takes 1 to 10 `nodePools` (created with the cluster); `autopilot` takes none (`INVALID_INPUT` otherwise), the provider runs its nodes. Node pool fields are as for [kubernetesNodePools.create](/docs/reference/api/kubernetes-node-pools); a node pool's `zones` must be zones of the cluster's region (and of the account).
* `releaseChannel` defaults to `regular`; `version` (a minor such as `1.33` or a full version) pins the control plane, otherwise the channel's default applies. `authorizedNetworks` lists at most 50 IPv4 CIDRs that may reach the control-plane endpoint (empty: any address; requests are still authenticated). `privateNodes: true` gives nodes no public addresses.
* `networkId` names a tenant network ([networks](/docs/reference/api/networks)) on the same account that has been provisioned and has a `ready` subnet in the cluster's region; the nodes use it and that subnet (the first by name when there are several). Another tenant's network, or one on another account, is `INVALID_INPUT`; one not provisioned yet is `INVALID_STATE`.
* `name` is 1 to 40 characters, `[a-z]([-a-z0-9]*[a-z0-9])?`, unique among the tenant's clusters that are not deleted (`CONFLICT` otherwise; a deleted cluster's name is free again).
* A tenant that already has 10 clusters that are not deleted (the module's `maxClustersPerTenant`; one being deleted still counts) is `QUOTA_EXCEEDED`, with the cap in `details.limit`.
* Decided as `compute:kubernetes:create` on the cluster it would create (`compute-kubernetes-cluster`, under the tenant), whose attributes are `providerAccountId`, `poolId`, `projectId`, `mode` and `location`. A refusal leaves nothing behind. After the decision the tenant's offer is checked as for `instances.create` (a suspended offer or an excluded region refuses), and the tenant's budgets unless the pool's billing mode is `none`.

The cluster id derives from the tenant, the caller and `idempotencyKey`. A retry with the same key returns the same cluster; the same key with any other field changed is `CONFLICT`. The caller owns the cluster. Publishes `kubernetes.cluster.created`.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:create` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field                              | Type                                                 | Required | Constraints                                                                                 |
| ---------------------------------- | ---------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------- |
| `tenantId`                         | string                                               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                  |
| `projectId`                        | string                                               |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`                            |
| `poolId`                           | string                                               | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`                           |
| `name`                             | string                                               | yes      | at most 40 characters; matches `^[a-z]([-a-z0-9]{0,38}[a-z0-9])?$`                          |
| `mode`                             | one of `standard`, `autopilot`                       | yes      |                                                                                             |
| `location`                         | string                                               | yes      | at most 40 characters; matches `^[a-z][a-z0-9-]{1,38}[a-z0-9]$`                             |
| `releaseChannel`                   | one of `rapid`, `regular`, `stable`, `extended`      |          | default `"regular"`                                                                         |
| `version`                          | string                                               |          | at most 64 characters; matches `^[0-9]{1,3}\.[0-9]{1,3}(\.[0-9]{1,3})?(-[a-z0-9.]{1,40})?$` |
| `networkId`                        | string                                               |          | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`                            |
| `authorizedNetworks`               | list of string                                       |          | 0-50 items                                                                                  |
| `privateNodes`                     | boolean                                              |          | default `false`                                                                             |
| `description`                      | string                                               |          | at most 256 characters                                                                      |
| `nodePools`                        | list of object                                       |          | 0-10 items                                                                                  |
| `nodePools[].name`                 | string                                               | yes      | at most 40 characters; matches `^[a-z]([-a-z0-9]{0,38}[a-z0-9])?$`                          |
| `nodePools[].machineType`          | string                                               | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9.-]{0,62}$`                                 |
| `nodePools[].nodeCount`            | integer                                              | yes      | 0-1000                                                                                      |
| `nodePools[].autoscaling`          | object                                               |          |                                                                                             |
| `nodePools[].autoscaling.minNodes` | integer                                              | yes      | 0-1000                                                                                      |
| `nodePools[].autoscaling.maxNodes` | integer                                              | yes      | 1-1000                                                                                      |
| `nodePools[].diskGiB`              | integer                                              |          | 10-65536                                                                                    |
| `nodePools[].diskType`             | string                                               |          | matches `^[a-z][a-z0-9-]{1,40}$`                                                            |
| `nodePools[].spot`                 | boolean                                              |          | default `false`                                                                             |
| `nodePools[].accelerators`         | object                                               |          |                                                                                             |
| `nodePools[].accelerators.type`    | string                                               | yes      | matches `^[a-z0-9][a-z0-9-]{0,62}$`                                                         |
| `nodePools[].accelerators.count`   | integer                                              | yes      | 1-16                                                                                        |
| `nodePools[].zones`                | list of string                                       |          | 0-10 items                                                                                  |
| `nodePools[].labels`               | map of string                                        |          |                                                                                             |
| `nodePools[].taints`               | list of object                                       |          | 0-32 items                                                                                  |
| `nodePools[].taints[].key`         | string                                               | yes      |                                                                                             |
| `nodePools[].taints[].value`       | string                                               |          | at most 63 characters                                                                       |
| `nodePools[].taints[].effect`      | one of `NoSchedule`, `PreferNoSchedule`, `NoExecute` | yes      |                                                                                             |
| `idempotencyKey`                   | string                                               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                  |

## list [#list]

`POST /api/compute/kubernetesClusters/list`

Returns the tenant's clusters with their node pools, oldest first; `poolId` and `projectId` narrow it, and `includeDeleted: true` adds deleted ones. At most 500 are returned. Decided as `compute:kubernetes:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:kubernetes:read` @ `compute-tenant` + items
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/kubernetesClusters/get`

Returns one cluster with its node pools, its `endpoint` and the `kubernetesVersion` and `nodeVersion` the provider last reported. A cluster of another tenant is `NOT_FOUND` here, and is refused by the decision (`ACCESS_DENIED`) before that. Decided as `compute:kubernetes:read` on the cluster.

* **Kind:** read
* **Authorization:** `compute:kubernetes:read` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId` | string | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/kubernetesClusters/update`

Changes the control-plane `version`, the `releaseChannel`, the `authorizedNetworks` or the `description`; name at least one (`INVALID_INPUT` otherwise). The first three change in place at the provider: a `ready` or failed cluster goes `updating` and back to `ready` once the provider has them (`kubernetes.cluster.updated`). The description is better-compute's alone and changes nothing at the provider. Every update bumps `version`. A cluster being deleted is `INVALID_STATE`. A version the provider refuses (a downgrade, one it does not offer) shows as `error` with its message. Decided as `compute:kubernetes:update` on the cluster.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:update` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field                | Type                                            | Required | Constraints                                                                                 |
| -------------------- | ----------------------------------------------- | -------- | ------------------------------------------------------------------------------------------- |
| `tenantId`           | string                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                                  |
| `clusterId`          | string                                          | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$`                            |
| `version`            | string                                          |          | at most 64 characters; matches `^[0-9]{1,3}\.[0-9]{1,3}(\.[0-9]{1,3})?(-[a-z0-9.]{1,40})?$` |
| `releaseChannel`     | one of `rapid`, `regular`, `stable`, `extended` |          |                                                                                             |
| `authorizedNetworks` | list of string                                  |          | 0-50 items                                                                                  |
| `description`        | string                                          |          | at most 256 characters                                                                      |

## delete [#delete]

`POST /api/compute/kubernetesClusters/delete`

Marks the cluster and its node pools for deletion and returns it `deleting`. The reconciler deletes the cluster at the provider (its node pools go with it); the cluster and its pools are then `deleted`, its authorization row is removed and `kubernetes.cluster.deleted` is published. Deleting a cluster that is being deleted returns it unchanged. If the provider refuses the delete the cluster is `error` with its message and the delete is retried with a growing wait (5, 10, 20 minutes and so on, at most every 6 hours); calling `delete` again tries at once. Publishes `kubernetes.cluster.deleting`. Decided as `compute:kubernetes:delete` on the cluster.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:delete` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId` | string | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |

## credentials [#credentials]

`POST /api/compute/kubernetesClusters/credentials`

Returns a short-lived kubeconfig for the caller: `{ kubeconfig, expiresAt, role }`, where `kubeconfig` is YAML text with one cluster (the control-plane endpoint and its CA), one user with a bearer token, one context and `current-context` set.

* `role` is `view` (the default), `edit` or `admin`, bound to Kubernetes' built-in `view`, `edit` and `cluster-admin` roles. The call is decided as `compute:kubernetes:connect` on the cluster; `edit` also needs `compute:kubernetes:update` and `admin` `compute:kubernetes:delete` on it, decided with the caller's own credential (`ACCESS_DENIED` otherwise).

* The provider mints the token for a Kubernetes service account derived from the caller and the role (`bc-u-`, 16 hex digits of a hash of the identity, `-view`/`-edit`/`-admin`), so each role keeps its own account and a later `view` kubeconfig never narrows an `admin` one the caller holds.

* `ttlSeconds` is 600 to 86400 (default 3600, the module's `credentialTtlSeconds`).

* The cluster must be `ready` (or `updating` after it was ready once); otherwise `INVALID_STATE`.

* The token is never stored, logged or published. Publishes `kubernetes.cluster.credentials-issued` with the role, subject, identity, ttl and expiry.

* **Kind:** mutation

* **Authorization:** `compute:kubernetes:connect` @ `compute-kubernetes-cluster`

* **Module:** `kubernetes`

* **Phase:** v1.0 (registered)

**Input**

| Field        | Type                           | Required | Constraints                                                      |
| ------------ | ------------------------------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string                         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId`  | string                         | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `role`       | one of `view`, `edit`, `admin` |          | default `"view"`                                                 |
| `ttlSeconds` | integer                        |          | 600-86400                                                        |

## versions [#versions]

`POST /api/compute/kubernetesClusters/versions`

Returns the Kubernetes versions the provider offers in a location of a pool visible to the tenant: `defaultVersion`, `validVersions`, and per release channel its default and valid versions. The pool and location are checked as for `create`. Decided as `compute:kubernetes:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:kubernetes:read` @ `compute-tenant`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `location` | string | yes      | at most 40 characters; matches `^[a-z][a-z0-9-]{1,38}[a-z0-9]$`   |


# kubernetesNodePools (/docs/reference/api/kubernetes-node-pools)

> Node pools are the groups of worker nodes of a `standard` Kubernetes cluster ([kubernetesClusters](/docs/reference/api/kubernetes-clusters)); an `autopilot` cluster has none, because the provider runs its nodes. A cluster's first node pools are created with it; these calls add, resize and remove pools later. Every call records what the tenant wants and returns at once: the node pool shows `pending`, `ready`, `updating`, `deleting`, `deleted` or `error` (with a `message`), and its cluster goes `updating` until the reconciler has converged the change, then `ready` again. A node pool the provider refuses is `error`, and so is its cluster, with the pool named in its `message`, until the pool is changed or deleted. The reconciler publishes `kubernetes.nodepool.ready`, `kubernetes.nodepool.error` and `kubernetes.nodepool.deleted` on the cluster's subject. A node pool's name is also its provider-side name.



Node pools are the groups of worker nodes of a `standard` Kubernetes cluster ([kubernetesClusters](/docs/reference/api/kubernetes-clusters)); an `autopilot` cluster has none, because the provider runs its nodes. A cluster's first node pools are created with it; these calls add, resize and remove pools later. Every call records what the tenant wants and returns at once: the node pool shows `pending`, `ready`, `updating`, `deleting`, `deleted` or `error` (with a `message`), and its cluster goes `updating` until the reconciler has converged the change, then `ready` again. A node pool the provider refuses is `error`, and so is its cluster, with the pool named in its `message`, until the pool is changed or deleted. The reconciler publishes `kubernetes.nodepool.ready`, `kubernetes.nodepool.error` and `kubernetes.nodepool.deleted` on the cluster's subject. A node pool's name is also its provider-side name.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                              | Phase | Status     |
| ------------------- | -------- | ---------------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:kubernetes:update` @ `compute-kubernetes-cluster` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:kubernetes:read` @ `compute-kubernetes-cluster`   | v1.0  | registered |
| [`update`](#update) | mutation | `compute:kubernetes:update` @ `compute-kubernetes-cluster` | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:kubernetes:update` @ `compute-kubernetes-cluster` | v1.0  | registered |

## create [#create]

`POST /api/compute/kubernetesNodePools/create`

Adds a node pool to a `standard` cluster that is not being deleted and returns it `pending` (`INVALID_STATE` for an autopilot cluster or one being deleted).

* `name` is 1 to 40 characters, `[a-z]([-a-z0-9]*[a-z0-9])?`, unique among the cluster's node pools that are not deleted (`CONFLICT`); a deleted pool's name is free again.
* `machineType` is the provider's machine type (`e2-standard-4`). `nodeCount` (0 to 1000) is nodes per zone of the pool; with `autoscaling` (`minNodes` 0 to 1000 not above `maxNodes` 1 to 1000) it is the initial count.
* Optional: `diskGiB` (10 to 65536) and `diskType` (`pd-balanced`), `spot`, `accelerators` (`{ type: 'nvidia-l4', count }`, 1 to 16 per node), `zones` (at most 10, zones of the cluster's region and of the account), Kubernetes `labels` (at most 32) and `taints` (at most 32, effect `NoSchedule`, `PreferNoSchedule` or `NoExecute`).
* A cluster has at most 10 node pools that are not deleted (`QUOTA_EXCEEDED`).
* With an `idempotencyKey` (scoped to the tenant, the caller and the cluster) a retry returns the same node pool; the same key with another request is `CONFLICT`.

Publishes `kubernetes.nodepool.created`. Decided as `compute:kubernetes:update` on the cluster.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:update` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field                  | Type                                                 | Required | Constraints                                                        |
| ---------------------- | ---------------------------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`             | string                                               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `clusterId`            | string                                               | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `name`                 | string                                               | yes      | at most 40 characters; matches `^[a-z]([-a-z0-9]{0,38}[a-z0-9])?$` |
| `machineType`          | string                                               | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9.-]{0,62}$`        |
| `nodeCount`            | integer                                              | yes      | 0-1000                                                             |
| `autoscaling`          | object                                               |          |                                                                    |
| `autoscaling.minNodes` | integer                                              | yes      | 0-1000                                                             |
| `autoscaling.maxNodes` | integer                                              | yes      | 1-1000                                                             |
| `diskGiB`              | integer                                              |          | 10-65536                                                           |
| `diskType`             | string                                               |          | matches `^[a-z][a-z0-9-]{1,40}$`                                   |
| `spot`                 | boolean                                              |          | default `false`                                                    |
| `accelerators`         | object                                               |          |                                                                    |
| `accelerators.type`    | string                                               | yes      | matches `^[a-z0-9][a-z0-9-]{0,62}$`                                |
| `accelerators.count`   | integer                                              | yes      | 1-16                                                               |
| `zones`                | list of string                                       |          | 0-10 items                                                         |
| `labels`               | map of string                                        |          |                                                                    |
| `taints`               | list of object                                       |          | 0-32 items                                                         |
| `taints[].key`         | string                                               | yes      |                                                                    |
| `taints[].value`       | string                                               |          | at most 63 characters                                              |
| `taints[].effect`      | one of `NoSchedule`, `PreferNoSchedule`, `NoExecute` | yes      |                                                                    |
| `idempotencyKey`       | string                                               |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/kubernetesNodePools/list`

Returns the cluster's node pools that are not deleted. Decided as `compute:kubernetes:read` on the cluster.

* **Kind:** read
* **Authorization:** `compute:kubernetes:read` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId` | string | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/kubernetesNodePools/update`

Resizes a node pool in place: `nodeCount`, and `autoscaling` (`null` turns it off); name at least one. A `nodeCount` outside the pool's autoscaling bounds is `INVALID_INPUT`. The machine type, disks, accelerators, zones, labels and taints never change: delete the pool and create another. The pool goes `updating` (bumping its `version`) until the provider has resized it. A pool or cluster being deleted is `INVALID_STATE`. Decided as `compute:kubernetes:update` on the cluster.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:update` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field         | Type        | Required | Constraints                                                      |
| ------------- | ----------- | -------- | ---------------------------------------------------------------- |
| `tenantId`    | string      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId`   | string      | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `nodePoolId`  | string      | yes      | at most 56 characters; matches `^knp_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `nodeCount`   | integer     |          | 0-1000                                                           |
| `autoscaling` | object,null |          |                                                                  |

## delete [#delete]

`POST /api/compute/kubernetesNodePools/delete`

Marks the node pool for deletion and returns it `deleting`; the reconciler deletes it at the provider, and it is then `deleted` and gone from `list`. Deleting a pool that is being deleted returns it unchanged. Decided as `compute:kubernetes:update` on the cluster.

* **Kind:** mutation
* **Authorization:** `compute:kubernetes:update` @ `compute-kubernetes-cluster`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                      |
| ------------ | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `clusterId`  | string | yes      | at most 56 characters; matches `^kcl_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `nodePoolId` | string | yes      | at most 56 characters; matches `^knp_[0-9a-hjkmnp-tv-z]{26,52}$` |


# kubernetesSubstrates (/docs/reference/api/kubernetes-substrates)

> A pool substrate is the managed Kubernetes cluster behind a pool whose policy has a `substrate` (`kind: 'kubernetes'`, an `account`, and `cluster: { create }` or `cluster: { adopt }`). The `kubernetes.substrates` reconciler records it here: it creates the cluster with one node pool per flavor (or adopts an existing cluster and adds the flavors' node pools to it, named `bc-<pool>-<flavor>` so they never collide with the tenant's own pools), server-side applies bcd's DaemonSet with a hosted join token, and the cluster's Nodes join the pool. Every object it applies carries the label `better-compute.dev/substrate-pool=<pool id>`; on an adopted cluster an object of the same name it did not create is never taken over or deleted. There is nothing to call to create or delete a substrate: change the pool's policy, or archive the pool to tear it down.



A pool substrate is the managed Kubernetes cluster behind a pool whose policy has a `substrate` (`kind: 'kubernetes'`, an `account`, and `cluster: { create }` or `cluster: { adopt }`). The `kubernetes.substrates` reconciler records it here: it creates the cluster with one node pool per flavor (or adopts an existing cluster and adds the flavors' node pools to it, named `bc-<pool>-<flavor>` so they never collide with the tenant's own pools), server-side applies bcd's DaemonSet with a hosted join token, and the cluster's Nodes join the pool. Every object it applies carries the label `better-compute.dev/substrate-pool=<pool id>`; on an adopted cluster an object of the same name it did not create is never taken over or deleted. There is nothing to call to create or delete a substrate: change the pool's policy, or archive the pool to tear it down.

All 1 methods are served by @better-compute/server 0.3.0.

| Method        | Kind | Authorization                                | Phase | Status     |
| ------------- | ---- | -------------------------------------------- | ----- | ---------- |
| [`get`](#get) | read | `compute:kubernetes:read` @ `compute-tenant` | v1.0  | registered |

## get [#get]

`POST /api/compute/kubernetesSubstrates/get`

Returns the pool's substrate: `poolId`, `tenantId`, `providerAccountId`, `mode` (`create` or `adopt`), the `clusterId` of its [kubernetesClusters](/docs/reference/api/kubernetes-clusters) row once recorded, `state` (`pending`, `provisioning`, `ready`, `updating`, `error`, `deleting`, `deleted`) with a `message` when there is something to fix or wait for, `nodesListed` and `nodesJoined` (the substrate's Nodes the cluster last listed, and of them the ones whose bcd joined the pool), `waitingToJoin` (`{ nodes, since }`: how many listed Nodes have no joined bcd, and since when none has joined; after 15 minutes the `message` says what to check, while the state stays `ready`), and `upgrade` (`{ version, since }`) while an `upgrades: 'drain'` upgrade waits for bookings to end. It never returns the join token, its id or a credential. A pool without a substrate is `NOT_FOUND`. Decided as `compute:kubernetes:read` on the pool owner's tenant.

* **Kind:** read
* **Authorization:** `compute:kubernetes:read` @ `compute-tenant`
* **Module:** `kubernetes`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |


# loadBalancers (/docs/reference/api/load-balancers)

> Load balancers on a provider account (DigitalOcean regional and global first). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.loadBalancers`. Reading takes `compute:load-balancers:read`, everything else `compute:load-balancers:manage`, on `provider-load-balancer/{id}`. A load balancer books one `loadBalancers` quota on the pool (no quota row grants none) and bills `vps.load-balancer.hours` from `ready` until deleted.



Load balancers on a provider account (DigitalOcean regional and global first). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.loadBalancers`. Reading takes `compute:load-balancers:read`, everything else `compute:load-balancers:manage`, on `provider-load-balancer/{id}`. A load balancer books one `loadBalancers` quota on the pool (no quota row grants none) and bills `vps.load-balancer.hours` from `ready` until deleted.

What a load balancer reaches is computed from the tenant's own rows and sent to the provider as explicit machine ids, never a tag, so it never reaches another tenant's machine. A `regional` one targets the tenant's instances in its pool and region (`targets: { pool: true }`) or named instances of the tenant (`targets: { instanceIds }`); a `global` one targets the tenant's regional load balancers (`targets: { loadBalancerIds }`). A regional load balancer is in one VPC and the provider balances only to machines in it: it lives on one subnet of a tenant network (`subnetId`, or the subnet its named instances share) and reaches only instances on that subnet, or, without one, on the account's default network and reaches only instances on no tenant network. An instance counts once it has a machine and until it is being deleted. Deleting an instance updates its load balancers at once; one that gains a machine is added by the periodic pass, within five minutes.

All 5 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                                      | Phase | Status     |
| ------------------- | -------- | ------------------------------------------------------------------ | ----- | ---------- |
| [`create`](#create) | mutation | `compute:load-balancers:manage` @ `compute-provider-load-balancer` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:load-balancers:read` @ `compute-tenant` + items           | v1.0  | registered |
| [`get`](#get)       | read     | `compute:load-balancers:read` @ `compute-provider-load-balancer`   | v1.0  | registered |
| [`update`](#update) | mutation | `compute:load-balancers:manage` @ `compute-provider-load-balancer` | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:load-balancers:manage` @ `compute-provider-load-balancer` | v1.0  | registered |

## create [#create]

`POST /api/compute/loadBalancers/create`

Makes a load balancer with its `forwarding` rules (1 to 15; an `https` or `http2` entry names a `certificateId` of the tenant on the same account, or an `https` one passes TLS through with `tlsPassthrough`), an optional `healthCheck`, `stickySessions`, `sourceRanges` and `redirectHttpToHttps`, and (global only) the `domains` it serves. It returns `pending`, booking one `loadBalancers`; no room, or no quota row, is `QUOTA_EXCEEDED`. Another tenant's instance is `ACCESS_DENIED`, another region or account `INVALID_INPUT`, named instances on two subnets (or not on `subnetId`) `INVALID_INPUT`, another tenant's subnet `NOT_FOUND`, another tenant's certificate or load balancer `NOT_FOUND`. Named instances are also decided as `compute:instances:update` on each, with the caller's own credential. The provider is asked once the certificates it names are ready. Publishes `providerLoadBalancer.created`, then `providerLoadBalancer.ready` with its `address`, or `providerLoadBalancer.error`.

* **Kind:** mutation
* **Authorization:** `compute:load-balancers:manage` @ `compute-provider-load-balancer`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                            | Type                                          | Required | Constraints                                                        |
| -------------------------------- | --------------------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`                       | string                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`                      | string                                        |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`                         | string                                        | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`                           | string                                        | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `description`                    | string                                        |          | at most 256 characters                                             |
| `scope`                          | one of `regional`, `global`                   |          | default `"regional"`                                               |
| `region`                         | string                                        |          | at most 40 characters; matches `^[a-z][a-z0-9-]{0,38}[a-z0-9]$`    |
| `subnetId`                       | string                                        |          | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `targets`                        | object                                        | yes      |                                                                    |
| `targets.pool`                   | any                                           |          |                                                                    |
| `targets.instanceIds`            | list of string                                |          | 1-100 items                                                        |
| `targets.loadBalancerIds`        | list of string                                |          | 1-100 items                                                        |
| `forwarding`                     | list of object                                | yes      | 1-15 items                                                         |
| `forwarding[].entryProtocol`     | one of `http`, `https`, `http2`, `tcp`, `udp` | yes      |                                                                    |
| `forwarding[].entryPort`         | integer                                       | yes      | 1-65535                                                            |
| `forwarding[].targetProtocol`    | one of `http`, `https`, `http2`, `tcp`, `udp` | yes      |                                                                    |
| `forwarding[].targetPort`        | integer                                       | yes      | 1-65535                                                            |
| `forwarding[].certificateId`     | string                                        |          | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `forwarding[].tlsPassthrough`    | boolean                                       |          |                                                                    |
| `healthCheck`                    | object                                        |          |                                                                    |
| `healthCheck.protocol`           | one of `http`, `https`, `tcp`                 | yes      |                                                                    |
| `healthCheck.port`               | integer                                       | yes      | 1-65535                                                            |
| `healthCheck.path`               | string                                        |          | at most 255 characters                                             |
| `healthCheck.intervalSeconds`    | integer                                       |          | 3-300                                                              |
| `healthCheck.timeoutSeconds`     | integer                                       |          | 3-300                                                              |
| `healthCheck.healthyThreshold`   | integer                                       |          | 2-10                                                               |
| `healthCheck.unhealthyThreshold` | integer                                       |          | 2-10                                                               |
| `stickySessions`                 | object                                        |          |                                                                    |
| `stickySessions.cookieName`      | string                                        | yes      | at most 40 characters; matches `^[A-Za-z0-9_-]{1,40}$`             |
| `stickySessions.ttlSeconds`      | integer                                       | yes      | 1-604800                                                           |
| `sourceRanges`                   | object                                        |          |                                                                    |
| `sourceRanges.allow`             | list of string                                |          | 0-50 items                                                         |
| `sourceRanges.deny`              | list of string                                |          | 0-50 items                                                         |
| `redirectHttpToHttps`            | boolean                                       |          |                                                                    |
| `domains`                        | list of object                                |          | 0-20 items                                                         |
| `domains[].name`                 | string                                        | yes      | at most 253 characters                                             |
| `domains[].certificateId`        | string                                        |          | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `idempotencyKey`                 | string                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/loadBalancers/list`

Returns the tenant's load balancers in the order they were made (at most 1000), optionally of one pool or project. Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:load-balancers:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/loadBalancers/get`

Returns one load balancer: its settings, targets, `address` (and `addressV6`) once assigned, and its state.

* **Kind:** read
* **Authorization:** `compute:load-balancers:read` @ `compute-provider-load-balancer`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                      |
| ---------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `loadBalancerId` | string | yes      | at most 56 characters; matches `^plb_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/loadBalancers/update`

Replaces each part given (`forwarding`, `healthCheck`, `targets`, `stickySessions`, `sourceRanges`, `redirectHttpToHttps`, `domains`) in place and returns it `updating` until the provider has applied it; publishes `providerLoadBalancer.updated` with what `changed`. The scope, region, pool and subnet never change: make another load balancer. Named instances must be on its subnet (or, without one, on no tenant network), else `INVALID_INPUT`.

* **Kind:** mutation
* **Authorization:** `compute:load-balancers:manage` @ `compute-provider-load-balancer`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                            | Type                                          | Required | Constraints                                                        |
| -------------------------------- | --------------------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`                       | string                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `loadBalancerId`                 | string                                        | yes      | at most 56 characters; matches `^plb_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `targets`                        | object                                        |          |                                                                    |
| `targets.pool`                   | any                                           |          |                                                                    |
| `targets.instanceIds`            | list of string                                |          | 1-100 items                                                        |
| `targets.loadBalancerIds`        | list of string                                |          | 1-100 items                                                        |
| `forwarding`                     | list of object                                |          | 1-15 items                                                         |
| `forwarding[].entryProtocol`     | one of `http`, `https`, `http2`, `tcp`, `udp` | yes      |                                                                    |
| `forwarding[].entryPort`         | integer                                       | yes      | 1-65535                                                            |
| `forwarding[].targetProtocol`    | one of `http`, `https`, `http2`, `tcp`, `udp` | yes      |                                                                    |
| `forwarding[].targetPort`        | integer                                       | yes      | 1-65535                                                            |
| `forwarding[].certificateId`     | string                                        |          | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `forwarding[].tlsPassthrough`    | boolean                                       |          |                                                                    |
| `healthCheck`                    | object                                        |          |                                                                    |
| `healthCheck.protocol`           | one of `http`, `https`, `tcp`                 | yes      |                                                                    |
| `healthCheck.port`               | integer                                       | yes      | 1-65535                                                            |
| `healthCheck.path`               | string                                        |          | at most 255 characters                                             |
| `healthCheck.intervalSeconds`    | integer                                       |          | 3-300                                                              |
| `healthCheck.timeoutSeconds`     | integer                                       |          | 3-300                                                              |
| `healthCheck.healthyThreshold`   | integer                                       |          | 2-10                                                               |
| `healthCheck.unhealthyThreshold` | integer                                       |          | 2-10                                                               |
| `stickySessions`                 | object                                        |          |                                                                    |
| `stickySessions.cookieName`      | string                                        | yes      | at most 40 characters; matches `^[A-Za-z0-9_-]{1,40}$`             |
| `stickySessions.ttlSeconds`      | integer                                       | yes      | 1-604800                                                           |
| `sourceRanges`                   | object                                        |          |                                                                    |
| `sourceRanges.allow`             | list of string                                |          | 0-50 items                                                         |
| `sourceRanges.deny`              | list of string                                |          | 0-50 items                                                         |
| `redirectHttpToHttps`            | boolean                                       |          |                                                                    |
| `domains`                        | list of object                                |          | 0-20 items                                                         |
| `domains[].name`                 | string                                        | yes      | at most 253 characters                                             |
| `domains[].certificateId`        | string                                        |          | at most 58 characters; matches `^pcert_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/loadBalancers/delete`

Marks the load balancer for deletion and returns it `deleting`; the reconciler deletes it at the provider, refunds its quota and publishes `providerLoadBalancer.deleted`. A regional load balancer a global one of the tenant targets is `INVALID_STATE` (reason `inUse`).

* **Kind:** mutation
* **Authorization:** `compute:load-balancers:manage` @ `compute-provider-load-balancer`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                      |
| ---------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `loadBalancerId` | string | yes      | at most 56 characters; matches `^plb_[0-9a-hjkmnp-tv-z]{26,52}$` |


# metal (/docs/reference/api/metal)

> The metal group enrolls machines by their BMCs and moves them through the provision state machine: inspect, commission, deploy, rescue, release and retire. Each verb checks the move against the provision state table, records it in one transaction and queues the metal reconciler, which runs the workflow of the new state; no verb touches the machine itself. Every method returns the node's metal view, which reports whether BMC credentials are set but never returns them. See [Bare metal](/docs/guides/bare-metal) and [BMC drivers and credentials](/docs/guides/bare-metal/bmc).



The metal group enrolls machines by their BMCs and moves them through the provision state machine: inspect, commission, deploy, rescue, release and retire. Each verb checks the move against the provision state table, records it in one transaction and queues the metal reconciler, which runs the workflow of the new state; no verb touches the machine itself. Every method returns the node's metal view, which reports whether BMC credentials are set but never returns them. See [Bare metal](/docs/guides/bare-metal) and [BMC drivers and credentials](/docs/guides/bare-metal/bmc).

All 15 methods are served by @better-compute/server 0.3.0.

| Method                      | Kind     | Authorization                              | Phase | Status     |
| --------------------------- | -------- | ------------------------------------------ | ----- | ---------- |
| [`enroll`](#enroll)         | mutation | `compute:nodes:provision` @ `compute-pool` | v0.2  | registered |
| [`manage`](#manage)         | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`inspect`](#inspect)       | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`provide`](#provide)       | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`deploy`](#deploy)         | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`undeploy`](#undeploy)     | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`rescue`](#rescue)         | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`unrescue`](#unrescue)     | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`abort`](#abort)           | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`recover`](#recover)       | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`retire`](#retire)         | mutation | `compute:nodes:provision` @ `compute-node` | v0.2  | registered |
| [`power`](#power)           | mutation | `compute:nodes:power` @ `compute-node`     | v0.2  | registered |
| [`bulk`](#bulk)             | mutation | `compute:nodes:provision` @ `compute-pool` | v1.0  | registered |
| [`confirmMac`](#confirmmac) | mutation | `compute:nodes:provision` @ `compute-node` | v1.0  | registered |
| [`discovered`](#discovered) | read     | `compute:sites:read` @ `compute-site`      | v1.0  | registered |

## enroll [#enroll]

`POST /api/compute/metal/enroll`

Creates a metal node from a BMC address, driver and credentials in the given pool, in provision state `enrolling`, and queues the reconciler to verify BMC access and inspect the machine out of band.

* Decided as `compute:nodes:provision` on the pool, since the node does not exist yet.
* `bmc.credentials` is a `secret://` reference or an inline username and password. Inline credentials go to the secret broker before anything is written, and only the reference is stored; without a secret broker they are `INVALID_CONFIG`.
* The node's driver bindings come from the BMC (power, management, virtual media), then the hardware profile (`hardwareProfile`, else the pool's `policy.hardwareProfile`). An unknown hardware profile is `INVALID_INPUT`.
* `bmc.driverOptions` is a JSON object of at most 1 KiB for the BMC driver (e.g. `{ "cipherSuite": 3 }` for older IPMI BMCs), sent on every BMC request and validated by the driver. Keys that look like credentials are `INVALID_INPUT`: credentials travel only as `bmc.credentials`.
* `addresses` are the machine's own IP literals (at most 16). In an agentless pool the site's agents can reach the machine through them at once; out-of-band inspection replaces them only when it reports some (IPMI reports none).
* A driver that cannot report MACs (IPMI) needs `macs`, or `productUuid` (the SMBIOS UUID its BMC reports). With the UUID alone, the first boot PXE-boots the machine once with no boot entry, and PXE discovery proposes the MAC its site agent then sees (see `discovered` and `confirmMac`). A firmware placeholder UUID (all zeros, all ones, `03000200-0400-0500-0006-000700080009`) cannot stand in for the MACs.
* `NOT_FOUND` when the pool does not exist or the tenant does not own it; `INVALID_INPUT` when the pool is not in `siteId`.

Each call creates a new node; there is no idempotency key. Publishes `node.enrolled`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-pool`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field               | Type                     | Required | Constraints                                                     |
| ------------------- | ------------------------ | -------- | --------------------------------------------------------------- |
| `tenantId`          | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`      |
| `siteId`            | string                   | yes      | 1-128 characters                                                |
| `poolId`            | string                   | yes      | 1-128 characters                                                |
| `bmc`               | object                   | yes      |                                                                 |
| `bmc.address`       | string                   | yes      | 1-512 characters                                                |
| `bmc.driver`        | string                   | yes      | matches `^[a-z][a-z0-9-]{0,63}$`                                |
| `bmc.credentials`   | string,object            | yes      | at most 2048 characters; matches `^(secret\|vault\|env)://\S+$` |
| `bmc.tlsSpkiSha256` | string                   |          | matches `^[0-9A-Fa-f]{64}$`                                     |
| `bmc.driverOptions` | object                   |          |                                                                 |
| `macs`              | list of any              |          | 0-16 items                                                      |
| `addresses`         | list of string           |          | 0-16 items                                                      |
| `productUuid`       | string                   |          | 1-64 characters                                                 |
| `arch`              | one of `x86_64`, `arm64` |          | default `"x86_64"`                                              |
| `uefi`              | boolean                  |          | default `true`                                                  |
| `hardwareProfile`   | string                   |          | 1-64 characters                                                 |

## manage [#manage]

`POST /api/compute/metal/manage`

Moves a `ready` node back to `enrolling` to re-verify BMC access; the reconciler then chains re-inspection and re-commissioning back to `ready`.

`INVALID_STATE` from any state but `ready`, `NOT_FOUND` when the tenant has no such metal node, and `CONFLICT` when the node changed concurrently. Publishes `node.provision.enrolling`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## inspect [#inspect]

`POST /api/compute/metal/inspect`

Moves a `ready` node to `inspecting`, where the machine boots the ramdisk and reports its inventory; commissioning follows, and the node returns to `ready`.

`INVALID_STATE` from any state but `ready`; `CONFLICT` when the node changed concurrently. Publishes `node.provision.inspecting`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## provide [#provide]

`POST /api/compute/metal/provide`

Moves a `ready` node to `commissioning`, which runs DCGM diagnostics, a quick wipe and a power-off before the node returns to `ready`.

`INVALID_STATE` from any state but `ready`; `CONFLICT` when the node changed concurrently. Publishes `node.provision.commissioning`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## deploy [#deploy]

`POST /api/compute/metal/deploy`

Moves a `ready` node to `deploying` with the given profile version recorded on it; the metal reconciler then starts a workflow of that profile.

The call records `profileId` and `profileVersion` without running the profile; run `profiles.validate` first to catch problems before the workflow starts. When the workflow succeeds the node becomes `deployed` and takes the profile's `role`. `INVALID_STATE` from any state but `ready`; `CONFLICT` when the node changed concurrently. Publishes `node.provision.deploying`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`         | string  | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `profileId`      | string  | yes      | 1-128 characters                                                  |
| `profileVersion` | integer | yes      | at least 1                                                        |

## undeploy [#undeploy]

`POST /api/compute/metal/undeploy`

Moves a `deployed` or `rescue` node to `releasing`, which wipes its disks with the chosen mode and powers it off before it returns to `ready`.

`wipe` is `none`, `quick` (the default), `secure-erase` or `crypto-erase`. Unless the mode is `none`, the node reaches `ready` only when the wipe result is attested; otherwise it fails with `UNVERIFIED_WIPE`. `INVALID_STATE` from other states. Publishes `node.provision.releasing`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type                                                   | Required | Constraints                                                       |
| ---------- | ------------------------------------------------------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string                                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string                                                 | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `wipe`     | one of `none`, `quick`, `secure-erase`, `crypto-erase` |          | default `"quick"`                                                 |

## rescue [#rescue]

`POST /api/compute/metal/rescue`

Moves a `deployed` node to `rescue`, whose workflow boots the rescue ramdisk for repairs from outside the installed OS.

`INVALID_STATE` from any state but `deployed`. On a pool with `agent: 'none'` the node fails with `UNSUPPORTED`, since such a machine cannot boot the ramdisk. Publishes `node.provision.rescue`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## unrescue [#unrescue]

`POST /api/compute/metal/unrescue`

Starts the `metal-unrescue` workflow, which boots a rescued node back into its installed OS; the node stays in `rescue` until that workflow succeeds, then becomes `deployed`.

* `INVALID_STATE` outside `rescue`.
* A second call while the unrescue workflow runs returns the node unchanged.
* `CONFLICT` while the rescue ramdisk is still booting.
* `INVALID_CONFIG` when the provision module is not registered.

Publishes `node.provision.unrescuing` with the `workflowId`. If the workflow fails, the node goes to `failed`, and `recover` returns it to `deployed`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## abort [#abort]

`POST /api/compute/metal/abort`

Stops the node's running workflow and moves the node to `failed`, recording where it failed, an `ABORTED` error with the optional `reason`, and maintenance on.

Legal from `enrolling`, `inspecting`, `commissioning`, `deploying` and `releasing`; `INVALID_STATE` otherwise. The aborted workflow still runs its cleanup steps. Publishes `node.provision.failed`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reason`   | string |          | 1-512 characters                                                  |

## recover [#recover]

`POST /api/compute/metal/recover`

Moves a `failed` node to the last stable point for the state it failed in: `enrolling` after enrolling, inspecting or commissioning, `releasing` after deploying or releasing, `deployed` after rescue, and `retiring` after retiring.

It clears `failedFrom` and `lastError` but leaves maintenance on. `INVALID_STATE` when the node is not `failed`. Publishes `node.provision.enrolling`, `node.provision.releasing`, `node.provision.deployed` or `node.provision.retiring`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## retire [#retire]

`POST /api/compute/metal/retire`

Moves a node to `retiring`, whose workflow crypto-erases the disks (on pools whose machines run `bcd`), powers the machine off and revokes its identity before the node becomes `retired`, which is terminal.

Legal from `adopting`, `adopted`, `enrolling`, `ready` and `failed`; `INVALID_STATE` otherwise, so undeploy a deployed machine first. Publishes `node.provision.retiring`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## power [#power]

`POST /api/compute/metal/power`

Queues a `bmc.power` command to the site agent of the node's site and returns its `commandId`; the provision state does not change.

* Decided as `compute:nodes:power` on the node.
* `action` is `on`, `off`, `graceful-off`, `reset`, `graceful-restart`, `cycle` or `nmi`.
* `INVALID_STATE` when the node has no BMC credentials, while it is `inspecting`, `commissioning`, `deploying` or `releasing` (a workflow controls its power then), or when no site agent serves its site.

The command expires five minutes after it is queued. Its idempotency key derives from the request id, so a retry with the same `X-Request-Id` header returns the same command.

* **Kind:** mutation
* **Authorization:** `compute:nodes:power` @ `compute-node`
* **Module:** `metal`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type                                                                            | Required | Constraints                                                       |
| ---------- | ------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string                                                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string                                                                          | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `action`   | one of `on`, `off`, `graceful-off`, `reset`, `graceful-restart`, `cycle`, `nmi` | yes      |                                                                   |

## bulk [#bulk]

`POST /api/compute/metal/bulk`

Runs one lifecycle verb on many nodes of a pool at once (`manage`, `inspect`, `provide`, `undeploy`, `rescue` or `retire`), as one batch; returns every node and, when the verb sends destructive commands, the batch.

* Each node is decided on its own, exactly as if the verb were called for it (`metal.<verb>`); one refusal refuses the whole call before anything changes. Every node must be in `poolId` (`INVALID_INPUT` otherwise), listed once, 1 to 10,000 of them.
* When the verb's steps send destructive commands, a quorum batch (`commandBatches`) is created over them with `reason`, and each node's commands carry it: with approver keys they wait for its approval; without, it is recorded for audit.
* `wipe` applies to `undeploy` only.

Decided as `compute:nodes:provision` on the pool, then per node.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-pool`
* **Module:** `metal`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type                                                                  | Required | Constraints                                                |
| ---------- | --------------------------------------------------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string                                                                | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `poolId`   | string                                                                | yes      | 1-128 characters                                           |
| `verb`     | one of `manage`, `inspect`, `provide`, `undeploy`, `rescue`, `retire` | yes      |                                                            |
| `nodeIds`  | list of string                                                        | yes      | 1-10000 items; no repeats                                  |
| `wipe`     | one of `none`, `quick`, `secure-erase`, `crypto-erase`                |          | default `"quick"`                                          |
| `reason`   | string                                                                | yes      | 1-512 characters                                           |

## confirmMac [#confirmmac]

`POST /api/compute/metal/confirmMac`

Binds the PXE MAC of a machine enrolled without MACs, usually the candidate PXE discovery proposed (its `discoveredMac`), or any MAC the operator knows to be the machine's. The waiting boot then goes on from that MAC.

* Decided as `compute:nodes:provision` on the node. Publishes `node.pxe-mac.confirmed` with the MAC and whether it was the discovered candidate.

* `INVALID_STATE` when the node already has MACs or is retired. `CONFLICT` when another machine of the site has the MAC.

* Until a MAC is confirmed, a machine known only by its UUID never gets a boot entry or a discovery token. A pool whose `policy.trustProvisioningNetwork` is `true` binds the candidate at once instead. Use it only on a provisioning network no untrusted host can reach. Its ramdisk's reported UUID and NICs are then checked against the machine before anything is written, and a mismatch fails the workflow.

* **Kind:** mutation

* **Authorization:** `compute:nodes:provision` @ `compute-node`

* **Module:** `metal`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `mac`      | string | yes      | matches `^[0-9A-Fa-f]{2}([:-][0-9A-Fa-f]{2}){5}$`                 |

## discovered [#discovered]

`POST /api/compute/metal/discovered`

Lists the machines the site's agents saw PXE-booting from a MAC with no boot entry in the last 24 hours, newest first. The agents never answer or enroll them.

* Decided as `compute:sites:read` on the site, and only the tenant that owns the site sees it (`NOT_FOUND` otherwise): the list describes the site's network.

* Each machine has its `mac`, the `productUuid` its DHCP request carried (as sent: SMBIOS stores the first three fields byte-swapped), its `arch`, when it was first and last seen, a request `count` summed over agents, the DHCP `relays` it came through, and the `agentNodeIds` that saw it.

* `nodeId` names the enrolled machine with that MAC. `candidateFor` names the machine enrolled without MACs whose candidate it is, awaiting `confirmMac`.

* The list alone binds nothing: any host on the network can claim any MAC and UUID. A MAC becomes a machine's candidate only when metal has just power-cycled that machine into PXE with its BMC. The MAC must be first seen within 10 minutes after that cycle, and be the only MAC its product UUID (in any byte order) was seen from in 24 hours. No other machine of the site may share the UUID, and the MAC must not be enrolled already. This publishes `node.pxe-mac.discovered`.

* At most 256 machines per agent, as each agent's last status report lists them.

* **Kind:** read

* **Authorization:** `compute:sites:read` @ `compute-site`

* **Module:** `metal`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `siteId`   | string | yes      | 1-128 characters                                           |


# networks (/docs/reference/api/networks)

> Networks are a tenant's private networks (VPCs) on a provider account, with their subnets and firewall rules ([firewallRules](/docs/reference/api/firewall-rules)). A network belongs to one provider pool, so to that pool's account, and the provider must support tenant networks (Google Cloud does; a provider that does not is `UNSUPPORTED`). The calls record what the tenant wants and return at once. The `vps.networks` reconciler then converges the provider: it creates the VPC, then the subnets and the firewall rules, then tags the network's instances, and removes what the tenant deleted, rules first, then subnets, then the network. `state` shows where it is: `pending`, `ready`, `deleting`, `deleted` or `error` (with a `message`). The network's `state` covers the VPC and its subnets (`ready` when all are; `error` with the failing resource in `message`); each subnet and each rule also has its own `state`, and a rule the provider refuses is `error` on the rule without taking the network out of `ready`. The reconciler publishes `network.ready`, `network.error`, `network.subnet.ready`, `network.rule.ready`, `network.rule.error` and `network.deleted`. An `error` is retried by the periodic pass, which runs every five minutes and also re-applies what drifted at the provider (a firewall deleted by hand).



Networks are a tenant's private networks (VPCs) on a provider account, with their subnets and firewall rules ([firewallRules](/docs/reference/api/firewall-rules)). A network belongs to one provider pool, so to that pool's account, and the provider must support tenant networks (Google Cloud does; a provider that does not is `UNSUPPORTED`). The calls record what the tenant wants and return at once. The `vps.networks` reconciler then converges the provider: it creates the VPC, then the subnets and the firewall rules, then tags the network's instances, and removes what the tenant deleted, rules first, then subnets, then the network. `state` shows where it is: `pending`, `ready`, `deleting`, `deleted` or `error` (with a `message`). The network's `state` covers the VPC and its subnets (`ready` when all are; `error` with the failing resource in `message`); each subnet and each rule also has its own `state`, and a rule the provider refuses is `error` on the rule without taking the network out of `ready`. The reconciler publishes `network.ready`, `network.error`, `network.subnet.ready`, `network.rule.ready`, `network.rule.error` and `network.deleted`. An `error` is retried by the periodic pass, which runs every five minutes and also re-applies what drifted at the provider (a firewall deleted by hand).

Only instances that own their machine are tagged: not deleting, lost or failed, and an attached one once it is claimed and while it has no `OwnershipLost`, `AttachConflict` or `DuplicateMachine` condition; the provider is told the instance id and leaves a machine labelled for another instance alone. Tags are set after the rules, and a machine whose tags were set within the last hour is skipped, so a network with more machines than one run can tag gets its rules at once and its tags over the next runs; a tag removed by hand comes back within the hour.

The provider-side name of a network, a subnet and a rule is `bc-` and the resource's own id, lower-cased. A machine on the network carries the tag `bc-` and 16 hex digits of the sha256 of its instance id, so a rule can target single instances.

Every tenant's networks on an account live in that account's one provider project, so they share its quotas (VPC networks, subnetworks and firewall rules per project): one tenant's networks use up what the others on the account can get. To bound that, a tenant has at most 10 networks that are not deleted, and a network at most 16 subnets and 100 firewall rules (`QUOTA_EXCEEDED`, with the cap in `details.limit`). The project's own quotas still apply on top, and a refusal of the provider shows as `error` with its message.

All 11 methods are served by @better-compute/server 0.3.0.

| Method                          | Kind     | Authorization                                      | Phase | Status     |
| ------------------------------- | -------- | -------------------------------------------------- | ----- | ---------- |
| [`create`](#create)             | mutation | `compute:networks:create` @ `compute-network`      | v1.0  | registered |
| [`list`](#list)                 | read     | `compute:networks:read` @ `compute-tenant` + items | v1.0  | registered |
| [`get`](#get)                   | read     | `compute:networks:read` @ `compute-network`        | v1.0  | registered |
| [`update`](#update)             | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |
| [`delete`](#delete)             | mutation | `compute:networks:delete` @ `compute-network`      | v1.0  | registered |
| [`addSubnet`](#addsubnet)       | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |
| [`removeSubnet`](#removesubnet) | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |
| [`adopt`](#adopt)               | mutation | `compute:providers:manage` @ `compute-fleet`       | v1.0  | registered |
| [`natGateway`](#natgateway)     | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |
| [`peer`](#peer)                 | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |
| [`routes`](#routes)             | mutation | `compute:networks:update` @ `compute-network`      | v1.0  | registered |

## create [#create]

`POST /api/compute/networks/create`

Records a network on the pool's provider account and returns it `pending`, with no subnets yet.

* The pool must be offered to the tenant, a `provider` pool, and its account's provider must support networks; otherwise the call is `UNSUPPORTED` (`NOT_FOUND` for a pool the tenant cannot see).
* `name` is 1 to 42 characters, `[a-z]([-a-z0-9]*[a-z0-9])?`, unique among the tenant's networks that are not deleted (`CONFLICT` otherwise; the name of a deleted network is free again).
* A tenant that already has 10 networks that are not deleted (one being deleted still counts) is `QUOTA_EXCEEDED`.
* Decided as `compute:networks:create` on the network it would create (`compute-network`, under the tenant), whose attributes are `providerAccountId`, `poolId` and `projectId`. A refusal leaves nothing behind.
* On a pool whose billing mode is `metered`, the tenant's budgets are checked after the decision, as for `instances.create`: when the pool asks for it or the tenant's offer is `stop-new`, `drain` or `suspend`, a spent budget is `SPEND_LIMIT_REACHED`. A root caller creating a network for a tenant is checked against that tenant's budgets. A suspended offer or a region its `allowedRegions` exclude refuses the create on any pool.

The network id derives from the tenant, the caller and `idempotencyKey`. A retry with the same key returns the same network; the same key with another pool, name, project or description is `CONFLICT`. The caller owns the network. Publishes `network.created`.

* **Kind:** mutation
* **Authorization:** `compute:networks:create` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`      | string |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`         | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`           | string | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `description`    | string |          | at most 256 characters                                             |
| `cidr`           | string |          |                                                                    |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/networks/list`

Returns the tenant's networks with their subnets, oldest first; `poolId` and `projectId` narrow it, and `includeDeleted: true` adds deleted ones. At most 500 are returned. Decided as `compute:networks:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:networks:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/networks/get`

Returns one network with its subnets (`pending`, `ready`, `deleting`, `deleted` or `error`). A network of another tenant is `NOT_FOUND` here, and is refused by the decision (`ACCESS_DENIED`) before that. Decided as `compute:networks:read` on the network.

* **Kind:** read
* **Authorization:** `compute:networks:read` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/networks/update`

Changes the network's `description`. The name, pool and account never change, because the provider cannot rename a network. A network being deleted is `INVALID_STATE`. Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field         | Type   | Required | Constraints                                                      |
| ------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`    | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId`   | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `description` | string | yes      | at most 256 characters                                           |

## delete [#delete]

`POST /api/compute/networks/delete`

Marks the network, its subnets and its rules for deletion and returns it `deleting`, along with its routes, peerings (also those another network holds with it) and NAT gateways. The reconciler deletes the routes, peerings and gateways, then the rules, then the subnets, then the network, and then the network is `deleted` and publishes `network.deleted`.

* `INVALID_STATE` while instances that are not deleted are on the network, with their ids in `details.instanceIds`: delete or move them first. Likewise while load balancers are on one of its subnets (`details.loadBalancerIds`).
* `INVALID_STATE` while [filesystems](/docs/reference/api/filesystems) that are not deleted are on the network, with their ids in `details.filesystemIds`: delete them first.
* `INVALID_STATE` while shared provider volumes that are not deleted are reached over the network, with their ids in `details.volumeIds`: delete them first.
* Deleting a network that is being deleted returns it unchanged.

Publishes `network.deleting`. Decided as `compute:networks:delete` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:delete` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$` |

## addSubnet [#addsubnet]

`POST /api/compute/networks/addSubnet`

Adds a subnet and returns the network with it `pending`.

* `cidr` is an IPv4 range inside 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16 (RFC 1918), with a prefix from /8 to /29, written as its network address (`10.0.0.0/20`). Anything else is `INVALID_INPUT`.
* It must not overlap another subnet of the network, in any region: that is `CONFLICT`, and the database refuses it too.
* `region` must be a region of the zones the account is configured with (`us-central1` for the zone `us-central1-a`); otherwise `INVALID_INPUT`, with the allowed regions in `details`.
* The name is unique in the network; the name of a deleted subnet is free again.
* A network that already has 16 subnets (not counting removed ones) is `QUOTA_EXCEEDED`.
* A network being deleted is `INVALID_STATE`.

Publishes `network.subnet.ready` once the provider has the subnet. Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type    | Required | Constraints                                                        |
| --------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`      | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `networkId`     | string  | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `name`          | string  | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `region`        | string  | yes      | matches `^[a-z][a-z0-9-]{1,38}[a-z0-9]$`                           |
| `zone`          | string  |          | matches `^[a-z][a-z0-9-]{1,62}[a-z0-9]$`                           |
| `cidr`          | string  | yes      |                                                                    |
| `privateAccess` | boolean |          |                                                                    |

## removeSubnet [#removesubnet]

`POST /api/compute/networks/removeSubnet`

Marks a subnet for deletion and returns the network with the subnet `deleting`. `INVALID_STATE` while instances that are not deleted use the subnet, with their ids in `details.instanceIds`, or load balancers are on it (`details.loadBalancerIds`), and while shared provider volumes of the network that are not deleted are in the subnet's region (a share lives in one of the network's VPCs there), with their ids in `details.volumeIds`. Its range is free for a new subnet at once. Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                       |
| ----------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `networkId` | string | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `subnetId`  | string | yes      | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$` |

## adopt [#adopt]

`POST /api/compute/networks/adopt`

Registers a VPC that already exists in the pool's provider account as a network of the tenant, so attached VMs whose NIC is on it are linked to it and get its firewall rules; returns the network `pending`, with `externalNetwork` set. Only the platform (root callers) adopts.

* **Input.** `externalNetwork` is the VPC's provider-side name, 1 to 63 characters `[a-z]([-a-z0-9]*[a-z0-9])?`; a `bc-` name (a network better-compute made) is `INVALID_INPUT`. `default` is shared with every VM of the project: adopting it needs `sharedTagScoped: true` (`INVALID_INPUT` with `details.field: 'sharedTagScoped'` otherwise), which the network records and shows. `name` and `description` are as for `create`, and the 10-network cap applies.

* **The VPC.** better-compute never creates, changes or deletes an adopted VPC: the reconciler skips it, provisions only the subnets and rules added to the network, and `delete` removes those and leaves the VPC. Its provider-side name is `externalNetwork`, not `bc-` and the id, also for instances created on it.

* **Firewall rules.** A rule on an adopted network must name `targets.instanceIds` (`INVALID_INPUT` on `firewallRules.create` and `update` otherwise), so it always targets instance network tags and never the whole VPC. A rule without targets that reaches the reconciler anyway goes to `error` instead of being applied.

* **Attach.** `instances.attach` links a machine whose NIC is on the VPC (by its name, or a reference ending in it) to the network, as it does for networks better-compute made.

* **Idempotency.** The same adoption again (same pool, VPC, name, flag and description) returns the network. A VPC already adopted in the tenant under another request is `CONFLICT` with `details.networkId`; one adopted by another tenant is `CONFLICT` unless both adoptions are `sharedTagScoped`.

* **IAM.** Decided as `compute:providers:manage` on the fleet (`compute-fleet`, in the root tenant); anyone but the platform is `ACCESS_DENIED`. The adopting identity owns the network.

* **Events.** `network.adopted` with `poolId`, `providerAccountId`, `name`, `externalNetwork` and `sharedTagScoped`.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-fleet`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field             | Type    | Required | Constraints                                                        |
| ----------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`        | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `poolId`          | string  | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `externalNetwork` | string  | yes      | at most 63 characters; matches `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$` |
| `name`            | string  | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `sharedTagScoped` | boolean |          | default `false`                                                    |
| `description`     | string  |          | at most 256 characters                                             |

## natGateway [#natgateway]

`POST /api/compute/networks/natGateway`

Turns a subnet's NAT gateway on (`enabled: true`) or off (`enabled: false`) and returns the network; the gateway appears in `natGateways` (`pending`, then `ready` with its public egress `addresses`). A NAT gateway carries the subnet's outbound traffic, so its machines need no public address of their own (isolated pools, and DOKS isolated worker nodes). Only providers with NAT gateways take it (DigitalOcean does; Google Cloud is `UNSUPPORTED` here).

* One gateway per subnet: asking again returns the network unchanged, and turning off a subnet without one is a no-op.
* Turning it off is `INVALID_STATE`, with the route names in `details.routes`, while a route of the subnet sends traffic to it (`networks.routes`): change those routes first.
* The subnet must belong to the network and not be being removed (`NOT_FOUND`, `INVALID_STATE`); a network being deleted is `INVALID_STATE`.
* Removing the subnet or deleting the network removes the gateway first: the reconciler deletes the subnet only once the gateway is gone.

Publishes `network.natGateway.ready` (and `network.natGateway.error`) with `edgeId` and `subnetId`. Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type    | Required | Constraints                                                       |
| ----------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`  | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `networkId` | string  | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `subnetId`  | string  | yes      | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `enabled`   | boolean | yes      |                                                                   |

## peer [#peer]

`POST /api/compute/networks/peer`

Peers a subnet of the network (`subnetId`) with a subnet of another network of the tenant (`peerNetworkId`, `peerSubnetId`), so their machines reach each other on private addresses; `enabled: false` removes the peering. Two subnets of the same network may be peered too: on DigitalOcean every subnet is a VPC of its own, and two of them do not reach each other otherwise. Only providers with peerings take it (`UNSUPPORTED` otherwise).

* The peering appears in `peerings` on both networks, each seen from its own side (`subnetId` is that network's subnet). The same pair asked again, from either network, is the same peering; either network can remove it.
* Both networks must be the tenant's (another tenant's network is `NOT_FOUND`), on the same provider account (`INVALID_INPUT`), and not being deleted (`INVALID_STATE`). A subnet is never peered with itself (`INVALID_INPUT`).
* A network takes part in at most 16 peerings (`QUOTA_EXCEEDED`, with `details.limit` and the network in `details.networkId`).
* Removing either subnet, or deleting either network, removes the peering first; the subnet goes once the peering is gone.

Publishes `network.peering.ready` (and `network.peering.error`) on the network that asked first. Decided as `compute:networks:update` on the network, and, on the caller's own credential, on the other network too.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type    | Required | Constraints                                                       |
| --------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`      | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `networkId`     | string  | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `subnetId`      | string  | yes      | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `peerNetworkId` | string  | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `peerSubnetId`  | string  | yes      | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `enabled`       | boolean |          | default `true`                                                    |

## routes [#routes]

`POST /api/compute/networks/routes`

Replaces the static routes of a subnet with `routes` (at most 32) and returns the network; the routes appear in `routes`. A route kept exactly as it was stays; a changed one is replaced (deleted at the provider, then made anew); one left out is deleted; an empty list removes them all. Each route has a `name` (unique in the subnet), a `destination` IPv4 CIDR written as its network address (`0.0.0.0/0` for everything), and exactly one `nextHop`:

* `{ natGateway: true }`: the subnet's NAT gateway, which must exist (`INVALID_STATE` otherwise). The reconciler makes the route once the gateway is `ready`.
* `{ instanceId }`: an instance of the tenant on this subnet (`NOT_FOUND`, or `INVALID_INPUT` when it is on another subnet). The route waits until the instance has a machine; an instance that is gone later puts the route in `error`. Decided, besides the network, as `compute:instances:update` on the instance, on the caller's own credential.
* `{ address }`: a host address inside the subnet (`INVALID_INPUT` otherwise).

Routes are a preview at DigitalOcean (VPC Routes): the account needs `previews: true`, or the route goes to `error` with the provider's refusal. Only providers with static routes take a non-empty list (`UNSUPPORTED` otherwise).

Publishes `network.route.ready` (and `network.route.error`). Decided as `compute:networks:update` on the network.

* **Kind:** mutation
* **Authorization:** `compute:networks:update` @ `compute-network`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                         | Type           | Required | Constraints                                                        |
| ----------------------------- | -------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`                    | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `networkId`                   | string         | yes      | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `subnetId`                    | string         | yes      | at most 57 characters; matches `^snet_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `routes`                      | list of object | yes      | 0-32 items                                                         |
| `routes[].name`               | string         | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `routes[].destination`        | string         | yes      |                                                                    |
| `routes[].nextHop`            | object         | yes      |                                                                    |
| `routes[].nextHop.natGateway` | any            |          |                                                                    |
| `routes[].nextHop.instanceId` | string         |          | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `routes[].nextHop.address`    | string         |          |                                                                    |


# nodes (/docs/reference/api/nodes)

> Nodes are machines that joined the control plane, as their tenant sees them: a tenant sees the nodes the authorization projection holds for it. Every method except `list` is decided on `compute-node/{nodeId}`. Mutations take an optional `version`; when the node has moved past it, the call is `CONFLICT`. See [Operating nodes](/docs/guides/nodes/operating-nodes).



Nodes are machines that joined the control plane, as their tenant sees them: a tenant sees the nodes the authorization projection holds for it. Every method except `list` is decided on `compute-node/{nodeId}`. Mutations take an optional `version`; when the node has moved past it, the call is `CONFLICT`. See [Operating nodes](/docs/guides/nodes/operating-nodes).

All 14 methods are served by @better-compute/server 0.3.0.

| Method                              | Kind     | Authorization                                    | Phase | Status     |
| ----------------------------------- | -------- | ------------------------------------------------ | ----- | ---------- |
| [`list`](#list)                     | read     | `compute:nodes:read` @ `compute-tenant` + items  | v0.1  | registered |
| [`get`](#get)                       | read     | `compute:nodes:read` @ `compute-node`            | v0.1  | registered |
| [`events`](#events)                 | read     | `compute:nodes:read` @ `compute-node`            | v0.1  | registered |
| [`cordon`](#cordon)                 | mutation | `compute:nodes:cordon` @ `compute-node`          | v0.1  | registered |
| [`uncordon`](#uncordon)             | mutation | `compute:nodes:cordon` @ `compute-node`          | v0.1  | registered |
| [`drain`](#drain)                   | mutation | `compute:nodes:drain` @ `compute-node`           | v0.1  | registered |
| [`label`](#label)                   | mutation | `compute:nodes:update` @ `compute-node`          | v0.1  | registered |
| [`taint`](#taint)                   | mutation | `compute:nodes:update` @ `compute-node`          | v0.1  | registered |
| [`adopt`](#adopt)                   | mutation | `compute:nodes:provision` @ `compute-node`       | v0.1  | registered |
| [`revokeIdentity`](#revokeidentity) | mutation | `compute:nodes:revoke-identity` @ `compute-node` | v0.1  | registered |
| [`delete`](#delete)                 | mutation | `compute:nodes:delete` @ `compute-node`          | v0.2  | registered |
| [`issueSshCert`](#issuesshcert)     | mutation | `compute:nodes:ssh-root` @ `compute-node`        | v0.3  | registered |
| [`revokeSshCert`](#revokesshcert)   | mutation | `compute:nodes:ssh-root` @ `compute-node`        | v0.3  | registered |
| [`listSshCerts`](#listsshcerts)     | read     | `compute:nodes:read`                             | —     | registered |

## list [#list]

`POST /api/compute/nodes/list`

Lists the nodes visible to the tenant in id order, filtered by pool, site, `cordoned` or `provisionState`.

The call is decided once on `compute-tenant/{tenantId}`, then each node is checked for `compute:nodes:read`; nodes the caller may not read are left out, so a page can hold fewer than `limit` items. Pages hold up to 50 nodes.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-tenant` + items
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field            | Type                                                                                                                                                               | Required | Constraints                                                           |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | --------------------------------------------------------------------- |
| `tenantId`       | string                                                                                                                                                             | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`            |
| `poolId`         | string                                                                                                                                                             |          | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$` |
| `siteId`         | string                                                                                                                                                             |          | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$`     |
| `cordoned`       | boolean                                                                                                                                                            |          |                                                                       |
| `provisionState` | one of `adopting`, `adopted`, `enrolling`, `inspecting`, `commissioning`, `ready`, `deploying`, `deployed`, `releasing`, `rescue`, `failed`, `retiring`, `retired` |          |                                                                       |
| `limit`          | integer                                                                                                                                                            |          | 1-50; default `50`                                                    |
| `cursor`         | string                                                                                                                                                             |          | at most 256 characters; matches `^[A-Za-z0-9._:-]{1,256}$`            |

## get [#get]

`POST /api/compute/nodes/get`

Returns one node with its labels, conditions and session (epoch, gateway and connection times).

A node the tenant cannot see is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |

## events [#events]

`POST /api/compute/nodes/events`

Returns a node's events from the event outbox, oldest first.

`type` keeps only events whose type starts with the given prefix, such as `node.` or `security.`. Pages hold up to 200 events (100 by default). A node the tenant cannot see is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:nodes:read` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `type`     | string  |          | at most 64 characters; matches `^[a-z][a-z0-9.-]{0,63}$`          |
| `limit`    | integer |          | 1-200; default `100`                                              |
| `cursor`   | string  |          | at most 41 characters; matches `^\d{1,20}:\d{1,20}$`              |

## cordon [#cordon]

`POST /api/compute/nodes/cordon`

Marks the node cordoned so placement puts no new work on it; existing allocations stay.

Cordoning a cordoned node returns it unchanged and publishes nothing. Otherwise the node's version goes up, its reconciler is queued, and `node.cordoned` is published.

* **Kind:** mutation
* **Authorization:** `compute:nodes:cordon` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## uncordon [#uncordon]

`POST /api/compute/nodes/uncordon`

Clears `cordoned` and ends a drain in progress, so the node is placeable again.

Decided as `compute:nodes:cordon`. A node that is neither cordoned nor draining is returned unchanged. Otherwise the `draining` reason is removed and `node.uncordoned` is published.

* **Kind:** mutation
* **Authorization:** `compute:nodes:cordon` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## drain [#drain]

`POST /api/compute/nodes/drain`

Cordons the node, adds the `draining` NoSchedule reason and hands the node to the Drain controller.

The Drain controller releases nothing in this release: it waits until the node holds no live allocation, then removes `draining` and publishes `node.drained`. The node stays cordoned until `uncordon`. Draining a node that is already draining returns it unchanged. A new drain publishes `node.draining`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:drain` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## label [#label]

`POST /api/compute/nodes/label`

Sets and removes operator labels on the node and returns the node with its labels.

`set` takes up to 64 labels and `remove` up to 64 keys. The call is `INVALID_INPUT` when it names nothing, names a key in both `set` and `remove`, or sets a key that inventory or the system owns. `remove` removes only operator labels. The node's version goes up and `node.labeled` is published.

* **Kind:** mutation
* **Authorization:** `compute:nodes:update` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type           | Required | Constraints                                                                          |
| ---------- | -------------- | -------- | ------------------------------------------------------------------------------------ |
| `tenantId` | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                           |
| `nodeId`   | string         | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$`                    |
| `set`      | map of string  |          | at most 64 entries; keys match `^[A-Za-z0-9](?:[A-Za-z0-9._/-]{0,126}[A-Za-z0-9])?$` |
| `remove`   | list of string |          | 0-64 items; no repeats                                                               |
| `version`  | integer        |          | at least 1                                                                           |

## taint [#taint]

`POST /api/compute/nodes/taint`

Adds and removes operator NoSchedule taints, stored in `noScheduleReasons` as `taint:<key>`.

`add` and `remove` take up to 32 keys each. The call is `INVALID_INPUT` when it names nothing or names a key in both lists. Only operator taints change; conditions, `unreachable` and `draining` belong to their controllers. When the result equals the current reasons the node is returned unchanged; otherwise `node.tainted` is published.

* **Kind:** mutation
* **Authorization:** `compute:nodes:update` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type           | Required | Constraints                                                       |
| ---------- | -------------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string         | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `add`      | list of string |          | 0-32 items; no repeats                                            |
| `remove`   | list of string |          | 0-32 items; no repeats                                            |
| `version`  | integer        |          | at least 1                                                        |

## adopt [#adopt]

`POST /api/compute/nodes/adopt`

Re-runs acceptance checks on a node in `adopting` or `adopted`, moving it to `adopting`.

Decided as `compute:nodes:provision`. Any other provision state is `INVALID_STATE`. The node leaves placement until acceptance passes. Publishes `node.adopting`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:provision` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## revokeIdentity [#revokeidentity]

`POST /api/compute/nodes/revokeIdentity`

Revokes every live certificate of the node and raises its identity generation, so no certificate of the old generation can renew.

Returns the node and `revokedCertificates`, the number of certificates revoked; the revocation reason is `reason`, or `identity revoked`. Each call raises the generation again. The machine comes back only by joining again. Publishes `security.node.identity-revoked`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:revoke-identity` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## delete [#delete]

`POST /api/compute/nodes/delete`

Tombstones a node that is `retired`, `adopting` or `enrolling` (or failed from `adopting` or `enrolling`), or an `adopted` node that is cordoned, and has no open allocation. An adopted host has no retire workflow: cordon it, wait until it holds nothing, then delete it. For example, after its TPM or disk was replaced, deleting it lets the machine join again under a new identity.

Any other state, or an open allocation, is `INVALID_STATE`; a node that changed pools during the call is `CONFLICT`. The node's devices stop being bookable, its scalar resources are removed, its certificates are revoked, its hardware key and attestor bindings are freed so the machine can join again as a new node, and its pending commands are cancelled. The node leaves its pool and the projection, so nobody sees it; audit rows and past allocations keep referencing it.

A retry after the delete returns `deleted: false`. Publishes `node.deleted`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:delete` @ `compute-node`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 64 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |
| `reason`   | string  |          | at most 256 characters; matches `^[^\u0000-\u001f\u007f]*$`       |

## issueSshCert [#issuesshcert]

`POST /api/compute/nodes/issueSshCert`

Issues a short-lived break-glass OpenSSH user certificate for one node, bound to the principal `<account>@<nodeId>`.

Decided as `compute:nodes:ssh-root`. The method comes from the tunnels module (v0.3) and is `UNSUPPORTED` when no compute SSH CA is configured. It needs an MFA session by default (`ACCESS_DENIED` with reason `mfa-required`), and a session that ends within a minute is `ACCESS_DENIED`. `publicKey` must be an Ed25519 or ECDSA P-256 key, and `principal` (default `root`) one of the configured accounts; otherwise `INVALID_INPUT`.

The lifetime is `ttlSeconds` (60 to 3,600, default 900), never past the caller's session. Issuance is rate-limited per identity (20 per hour by default, then `RATE_LIMITED`). Every certificate is recorded and `node.ssh-cert.issued` is published.

* **Kind:** mutation
* **Authorization:** `compute:nodes:ssh-root` @ `compute-node`
* **Module:** `tunnels`
* **Phase:** v0.3 (registered)

**Input**

| Field        | Type    | Required | Constraints                                                |
| ------------ | ------- | -------- | ---------------------------------------------------------- |
| `tenantId`   | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `nodeId`     | string  | yes      | at most 57 characters; matches `^node_[0-9a-z]{26,52}$`    |
| `publicKey`  | string  | yes      | at most 4096 characters                                    |
| `principal`  | string  |          | matches `^[a-z_][a-z0-9_-]{0,31}$`; default `"root"`       |
| `ttlSeconds` | integer |          | 60-3600; default `900`                                     |

## revokeSshCert [#revokesshcert]

`POST /api/compute/nodes/revokeSshCert`

Revokes a break-glass SSH certificate before it expires; the node's revocation list carries it from the next NodeSpec on.

Decided as `compute:nodes:ssh-root` and hosted by the tunnels module (v0.3). Revoking again returns the same revocation. A serial not issued for this node is `NOT_FOUND`. Publishes `node.ssh-cert.revoked`.

* **Kind:** mutation
* **Authorization:** `compute:nodes:ssh-root` @ `compute-node`
* **Module:** `tunnels`
* **Phase:** v0.3 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `nodeId`   | string | yes      | at most 57 characters; matches `^node_[0-9a-z]{26,52}$`    |
| `serial`   | string | yes      | matches `^[0-9]{1,20}$`                                    |
| `reason`   | string |          | at most 500 characters                                     |

## listSshCerts [#listsshcerts]

`POST /api/compute/nodes/listSshCerts`

Lists the break-glass SSH certificates the tenant issued for one node, newest first, with each one's principal, issuer, lifetime and state; never the certificate itself.

Decided as `compute:nodes:read` on `compute-node/{nodeId}` and hosted by the tunnels module. `state` is `active`, `expired` or `revoked`; a revoked certificate also carries `revokedAt`, `revokedBy` (an identity, or `iam` when re-authorization revoked it) and `revokeReason` when one was given.

Only unexpired certificates are listed unless `includeExpired` is true; expired certificates are kept for one day, then removed by an hourly clean-up. Pages hold 1 to 500 certificates (100 by default), and `nextCursor` is set while more remain. A node the tenant cannot see is `NOT_FOUND`, and a malformed `cursor` is `INVALID_INPUT`.

* **Kind:** read
* **Authorization:** `compute:nodes:read`
* **Module:** `tunnels`
* **Phase:** not in PLAN.md §9.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                |
| ---------------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `nodeId`         | string  | yes      | at most 57 characters; matches `^node_[0-9a-z]{26,52}$`    |
| `includeExpired` | boolean |          | default `false`                                            |
| `limit`          | integer |          | 1-500; default `100`                                       |
| `cursor`         | string  |          | at most 128 characters                                     |


# pools (/docs/reference/api/pools)

> A pool groups the nodes of one site under one GPU model, one sharing mode and one policy, and every reservation and quota belongs to exactly one pool. The tenant that creates a pool owns it; other tenants see it only once the owner offers it to them with a quota. Only the owner can offer, unoffer, update or archive a pool; anyone else gets `NOT_FOUND`. See [Pools](/docs/guides/capacity/pools).



A pool groups the nodes of one site under one GPU model, one sharing mode and one policy, and every reservation and quota belongs to exactly one pool. The tenant that creates a pool owns it; other tenants see it only once the owner offers it to them with a quota. Only the owner can offer, unoffer, update or archive a pool; anyone else gets `NOT_FOUND`. See [Pools](/docs/guides/capacity/pools).

All 13 methods are served by @better-compute/server 0.3.0.

| Method                              | Kind     | Authorization                                                                         | Phase | Status     |
| ----------------------------------- | -------- | ------------------------------------------------------------------------------------- | ----- | ---------- |
| [`list`](#list)                     | read     | `compute:pools:read` @ `compute-tenant` + items                                       | v0.1  | registered |
| [`get`](#get)                       | read     | `compute:pools:read` @ `compute-pool`                                                 | v0.1  | registered |
| [`create`](#create)                 | mutation | `compute:pools:create` @ `compute-fleet` or `compute:pools:create` @ `compute-tenant` | v0.1  | registered |
| [`capacity`](#capacity)             | read     | `compute:capacity:read` @ `compute-pool`                                              | v0.1  | registered |
| [`offer`](#offer)                   | mutation | `compute:pools:share` @ `compute-pool`                                                | v0.1  | registered |
| [`unoffer`](#unoffer)               | mutation | `compute:pools:share` @ `compute-pool`                                                | v0.1  | registered |
| [`updateOffer`](#updateoffer)       | mutation | `compute:pools:share` @ `compute-pool`                                                | v1.0  | registered |
| [`offers`](#offers)                 | read     | `compute:pools:read` @ `compute-tenant`                                               | v1.0  | registered |
| [`update`](#update)                 | mutation | `compute:pools:manage` @ `compute-pool`                                               | v0.2  | registered |
| [`archive`](#archive)               | mutation | `compute:pools:manage` @ `compute-pool`                                               | v0.2  | registered |
| [`bindMachines`](#bindmachines)     | mutation | `compute:pools:manage` @ `compute-pool`                                               | v1.0  | registered |
| [`unbindMachines`](#unbindmachines) | mutation | `compute:pools:manage` @ `compute-pool`                                               | v1.0  | registered |
| [`machines`](#machines)             | read     | `compute:pools:read` @ `compute-pool`                                                 | v1.0  | registered |

## list [#list]

`POST /api/compute/pools/list`

Returns the pools the tenant owns or is offered, each with `owned`, `archived`, the tenant's own quota rows and, when its offer lists them, the `flavors` it may use.

Archived pools appear only for their owner. After the coarse decision on the tenant, each pool is checked as `compute:pools:read` on the pool, and only the allowed ones are returned.

* **Kind:** read
* **Authorization:** `compute:pools:read` @ `compute-tenant` + items
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## get [#get]

`POST /api/compute/pools/get`

Returns one pool as the tenant sees it, with `owned`, `archived` and the tenant's own quota rows.

The call is `NOT_FOUND` when the pool is not owned by or offered to the tenant, or is archived and the tenant is not its owner.

* **Kind:** read
* **Authorization:** `compute:pools:read` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## create [#create]

`POST /api/compute/pools/create`

Creates a pool on a site the tenant owns, with the owner's quota rows and an empty rollup; returns the pool at version 1.

* It is decided as `compute:pools:create` on the tenant (`compute-tenant`), the alternative its catalog row lists next to `compute-fleet`.
* `sharing` defaults to `exclusive` and `agent` to `bcd` (`none` for `provider` pools). A `provider` pool needs `providerAccountId` and `agent: 'none'`, and other classes refuse `providerAccountId`; both are `INVALID_INPUT`. The account must be one of the pool's own tenant: another tenant's (the root's included) is `NOT_FOUND`, worded as for an account that does not exist. Tenants a pool is offered to are not shown its `providerAccountId` in `get` and `list`.
* `ownerQuota` defaults to `gpu` at 2,147,483,647. Owners of `mig`, `timeslice` or `mps` pools add `gpu.mig` or `gpu.slot` rows themselves, because a missing row grants 0.
* The policy is checked as the nodes will get it (v1.1). Each module's `pool-policy` validator runs, e.g. provision's host units: hugepages, kernel modules, a `kubernetes` join set given all together or not at all, and sharing replicas up to 48. A policy nodes can't take is `INVALID_INPUT`.
* On a platform (non-`custom`) site, a `kubernetes` join set also needs `platformOperated: true`: the cluster's admins get root on the machines, which keep their platform trust.
* With `idempotencyKey`, the pool id is derived from the tenant, the caller and the key. A retry returns the stored pool without a second IAM decision; the same key with a different site, class, sharing mode, agent or provider account is `CONFLICT`.
* `policy.workloadRoles: { aws }` (AWS A7) lists the IAM roles, as ARN globs (`?` and at most 8 `*` in the role path and name; the account is literal), that instances of a hosted pool with `policy.workloadIdentity: true` may name as `workloadIdentity.awsRoleArn`; at most 64. A non-empty list without `policy.workloadIdentity: true`, or anything else, is `INVALID_INPUT`. (The plan called this field `workloadIdentity.aws.roleArns`; `workloadIdentity` was already the pool's boolean switch.)
* `policy.billing: { mode }` sets how provider instances in the pool are billed: `metered` (priced per resource), `platform-fee` (one managed-VM line, as a tenant's own cloud account pays) or `none`. Without it a provider pool owned by a tenant other than the root pays the platform fee and every other pool is metered. The explicit mode outranks that default, so only the platform (a caller of the IAM root tenant) sets it: any other caller gets `ACCESS_DENIED`, and a mode other than those three is `INVALID_INPUT`. The mode is read when an instance's usage interval opens.
* `policy.replacePreempted` (provider pools): `true` recreates an instance in place, on a new machine under the same id, when the provider reclaims its Spot machine; unset or `false` leaves it `lost`. Replacements back off and stop after three in a row that never reach `running`. A value that is not a boolean is `INVALID_INPUT` (`policy.replacePreempted must be true or false`). See [Spot preemption](/docs/guides/vps/instances#spot-preemption).
* A site the tenant does not own is `NOT_FOUND`.

Emits `pool.created`.

* **Kind:** mutation
* **Authorization:** `compute:pools:create` @ `compute-fleet` or `compute:pools:create` @ `compute-tenant`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field               | Type                                                           | Required | Constraints                                                        |
| ------------------- | -------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`          | string                                                         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `siteId`            | string                                                         | yes      | at most 57 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `class`             | one of `metal-host`, `metal-dedicated`, `vps-host`, `provider` | yes      |                                                                    |
| `gpuModel`          | string                                                         |          | at most 32 characters; matches `^[A-Za-z0-9-]{1,32}$`              |
| `sharing`           | one of `exclusive`, `mig`, `timeslice`, `mps`                  |          | default `"exclusive"`                                              |
| `agent`             | one of `bcd`, `none`                                           |          |                                                                    |
| `policy`            | object                                                         |          |                                                                    |
| `ownerQuota`        | map of integer                                                 |          | keys match `^[A-Za-z][A-Za-z0-9._-]{0,63}$`                        |
| `providerAccountId` | string                                                         |          | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `idempotencyKey`    | string                                                         |          | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`     |

## capacity [#capacity]

`POST /api/compute/pools/capacity`

Returns one pool's rollup rows, one per resource; the same answer as `capacity.byPool`.

The call is `NOT_FOUND` when the pool is not visible to the tenant.

* **Kind:** read
* **Authorization:** `compute:capacity:read` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## offer [#offer]

`POST /api/compute/pools/offer`

Offers a pool the tenant owns to another tenant with a quota, or updates an existing offer and its quota; returns the pool as that tenant sees it.

* `quota` creates or resizes the offered tenant's quota rows. A new maximum below what is in use is raised to the amount in use.
* `includeDescendants` and `priority` (default 0) are stored with the offer; `offers` lists the offers a tenant inherits through them.
* `flavors` (1 to 64 flavor names) lists the flavors the tenant may create, and attach machines of, on the pool: `instances.create`, `instances.attach` (the flavor whose offer is the machine's type), `instances.resize` and `sandboxes.create` (the template's flavor) refuse any other with `QUOTA_EXCEEDED` (`details.reason: 'flavor-not-offered'`), for every caller. Leaving it out keeps the stored list; `updateOffer({ flavors: null })` clears it, which allows every flavor again.
* `quota` may also hold `instances` and `sandboxes`, counted one per instance and per sandbox (see `quotas`).
* `onBudgetExceeded` (`notify`, `stop-new`, `drain` or `suspend`) and `allowedRegions` (1 to 16 region names) are stored with the offer; leaving one out keeps what the offer already has, so re-offering to change the quota does not clear them. See `updateOffer` for what the policies do. Anything else is `INVALID_INPUT`.
* `INVALID_STATE` when the pool is archived, when a `timeslice` or `mps` pool or a pool holding `tenant-owned` nodes is offered to anyone but its owner, or when a `policy.singleTenant` pool is offered with `includeDescendants` or is already offered to another tenant.
* The pool's IAM authz row is written for the offered tenant, with the attributes `gpuModel`, `class`, `sharing`, `region` and `owned`.

Emits `pool.offered`.

* **Kind:** mutation
* **Authorization:** `compute:pools:share` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field                | Type                                            | Required | Constraints                                                       |
| -------------------- | ----------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`           | string                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`             | string                                          | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `toTenantId`         | string                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `quota`              | map of integer                                  | yes      | keys match `^[A-Za-z][A-Za-z0-9._-]{0,63}$`                       |
| `includeDescendants` | boolean                                         |          | default `false`                                                   |
| `priority`           | integer                                         |          | -1000-1000                                                        |
| `onBudgetExceeded`   | one of `notify`, `stop-new`, `drain`, `suspend` |          |                                                                   |
| `allowedRegions`     | list of string                                  |          | 1-16 items                                                        |
| `flavors`            | list of string                                  |          | 1-64 items                                                        |

## unoffer [#unoffer]

`POST /api/compute/pools/unoffer`

Withdraws a pool's offer from a tenant and returns whether an offer was removed.

That tenant's quota rows on the pool drop to what is in use, so live reservations keep their bookings while nothing new fits, and its authz row for the pool is removed. A pool cannot be unoffered from its owner (`INVALID_INPUT`). Calling it again answers `removed: false`. Emits `pool.unoffered` only when an offer was removed.

* **Kind:** mutation
* **Authorization:** `compute:pools:share` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`     | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `toTenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |

## updateOffer [#updateoffer]

`POST /api/compute/pools/updateOffer`

Changes the budget policy and regions of an existing offer, or suspends or lifts it, and returns the pool as the offered tenant sees it. It is the owner's call, decided as `compute:pools:share` on the pool like `offer`; a tenant that is only offered the pool gets `NOT_FOUND`.

* **Fields.** `onBudgetExceeded`, `allowedRegions`, `suspended` and `flavors` are each optional, and a request naming none is `INVALID_INPUT`. `flavors` replaces the offer's flavor allowlist (see `offer`); `null` clears it. `NOT_FOUND` when the pool is not offered to `toTenantId`; a pool has no offer to its owner (`INVALID_INPUT`).
* **Budget policies.** When a budget of the tenant is spent, better-iam's alert applies the policy of each offer to that tenant. `notify` only publishes `quota.budget-exceeded`. `stop-new` and `drain` also make admissions check the tenant's budget, and `drain` cancels the tenant's reservations on the pool that have not started. `suspend` stops spend on the pool: it marks the offer suspended, publishes `quota.budget-suspended`, sets the tenant's running instances in the pool to `desiredState: 'stopped'` with the condition `BudgetSuspended` (the reconciler then stops the machines), and refuses the tenant's new work on the pool with `SPEND_LIMIT_REACHED` (`details.reason: 'offer-suspended'`): creating or attaching an instance, creating a network or reservation, and starting an instance. The stop does not depend on host wiring: the `vps.suspension` reconciler stops, every minute, the running instances of every suspended offer (a stop that failed is retried on the next pass), and the host's budget reaction (vps's `sandboxIamReactions`, in `iamReactions`) only makes it immediate. Sandbox warm pools of a suspended offer are emptied and not refilled. Other sandbox-managed instances follow their sandbox and are not stopped directly.
* **Suspension.** `suspended: true` suspends the offer by hand, with the same effect as a `suspend` policy: new work is refused and the `vps.suspension` reconciler stops the tenant's running instances in the pool within a minute. `suspended: false` lifts it and restarts nothing: stopped instances stay stopped until the tenant starts them, and `BudgetSuspended` clears on that start. A new alert that finds the budget still spent suspends the offer again.
* **Regions.** With `allowedRegions`, new work on a pool whose site is in another region is `ACCESS_DENIED` (`details.reason: 'region-not-allowed'`). The site's `iamRegion` is compared, else its `region`; a site with neither match is refused. It applies to instances, reservations, networks and sandboxes created on the pool.

Emits `pool.offer-updated`.

* **Kind:** mutation
* **Authorization:** `compute:pools:share` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v1.0 (registered)

**Input**

| Field              | Type                                            | Required | Constraints                                                       |
| ------------------ | ----------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`         | string                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`           | string                                          | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `toTenantId`       | string                                          | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `onBudgetExceeded` | one of `notify`, `stop-new`, `drain`, `suspend` |          |                                                                   |
| `allowedRegions`   | list of string                                  |          | 1-16 items                                                        |
| `suspended`        | boolean                                         |          |                                                                   |
| `flavors`          | array,null                                      |          | 1-64 items                                                        |

## offers [#offers]

`POST /api/compute/pools/offers`

Returns the offers that reach the tenant, as `items`: its own, and those of its ancestors made with `includeDescendants`. Each item names the pool (`poolId`, and `pool` with its owner, site, class and GPU model), `offeredTo` (the tenant the offer was made to), `inherited` and `depth` (0 for the tenant's own offer, 1 for its parent's, and so on), `includeDescendants`, `priority`, the offer's `flavors`, `allowedRegions`, `onBudgetExceeded` and `suspended`, and the tenant's own quota rows on the pool (never an ancestor's: an inherited offer's limits and usage stay with the tenant it was made to, so an inherited item usually has none).

* **Order.** By `priority`, highest first, then nearest first, then pool id. One item per pool: the tenant's own offer shadows an inherited one, and a nearer ancestor's a farther one's. Archived pools and pools the tenant owns are left out.

* **Ancestry.** The tenant tree is IAM's: compute asks the host's `capacity({ tenantAncestors })` (`iamTenantAncestors` from `@better-compute/iam`, with a root credential), with no transaction open. Without it, only the tenant's own offers are listed.

* **Admission.** An inherited offer is a listing only: creating on the pool needs an offer to the tenant itself (`pools.offer`), with its own quota rows.

* Decided as `compute:pools:read` on the tenant. A read: nothing is written.

* **Kind:** read

* **Authorization:** `compute:pools:read` @ `compute-tenant`

* **Module:** `capacity`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## update [#update]

`POST /api/compute/pools/update`

Replaces the policy, the GPU model, or both, of a pool the tenant owns, and returns the pool with its version incremented.

* The new `policy` replaces the old one as a whole, and is checked as on `create`: the `pool-policy` validators, and `platformOperated` for a platform site's Kubernetes join. A request with neither field is `INVALID_INPUT`. Class, site, sharing mode and agent cannot change.
* `policy.billing` is root's, as on `create`: a non-root caller may resend the pool's stored `billing` but not add, change or remove it (`ACCESS_DENIED`).
* `policy.replacePreempted` must be a boolean, as on `create` (`INVALID_INPUT`). The new value applies to the next preemption.
* Turning `policy.singleTenant` on is `INVALID_STATE` while more than one tenant, or any tenant with `includeDescendants`, holds an offer.
* Changing `gpuModel` is `INVALID_STATE` (`details.tenantId`) while GPUs are charged on the pool to a tenant that has a per-model tenant quota row (`gpu.<model>` on `*`), so every refund reaches the row its charge took.
* A new `migLayout` in a `mig` pool never repartitions booked GPUs; each node takes the new layout once its GPUs are idle. A change to `prewarmImages` refreshes the desired state of the site's agents.
* The pool's authz rows are refreshed for the owner and every offered tenant.

Emits `pool.updated`.

* **Kind:** mutation
* **Authorization:** `compute:pools:manage` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `policy`   | object |          |                                                                   |
| `gpuModel` | string |          | at most 32 characters; matches `^[A-Za-z0-9-]{1,32}$`             |

## archive [#archive]

`POST /api/compute/pools/archive`

Retires a pool the tenant owns: it removes every offer and stops new work, while live reservations keep their bookings until they end.

Each offered tenant's quota rows drop to what is in use and its authz row for the pool is removed. After archiving, only the owner sees the pool, with `archived: true`. Archiving an archived pool returns it unchanged. Emits `pool.archived` with the list of tenants whose offers were removed.

* **Kind:** mutation
* **Authorization:** `compute:pools:manage` @ `compute-pool`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |

## bindMachines [#bindmachines]

`POST /api/compute/pools/bindMachines`

Binds machines that the `vps.discovery` worker saw in a `provider` pool's account to the pool; returns `bound`, the machines bound to the pool now, and `skipped`, each with `externalId` and `reason`. Tenants the pool is offered to can then list them (`machines`) and attach them (`instances.attach`).

* **Input.** `externalIds` lists 1 to 100 machine ids as `providerAccounts.machines` shows them; duplicates count once.

* **Skips.** `not-found`: discovery has not seen the machine in the pool's account (run `providerAccounts.discover`). `bound-elsewhere`: it is bound to another pool; unbind it there first. `gone`: discovery marked it gone. `deleting`: the provider is deleting it. `held`: an instance of another pool holds it (one of this pool may). `other-region`: for Compute Engine accounts (`gcp`), the machine's region (or its zone's) is not the region of the pool's site; other providers' sites name regions freely and are not compared.

* **Idempotency.** The machine rows are locked while the call runs. A machine already bound to the pool is listed in `bound` again and changes nothing. Discovery passes, detaching and deleting never clear a binding, so a machine whose instance goes returns to its pool. A binding to a pool archived since counts as none: the machine can be bound and attached elsewhere.

* **IAM.** Decided as `compute:pools:manage` on the pool. Only the platform (root callers) and the tenant that owns the pool's provider account bind; a tenant the pool is offered to is `ACCESS_DENIED`.

* **Errors.** `NOT_FOUND` for a pool the tenant neither owns nor is offered, `INVALID_INPUT` for one that is not a `provider` pool, `INVALID_STATE` for a provider pool without an account.

* **Events.** `pool.machines.bound` with `poolId`, `accountId` and the `externalIds` this call bound (none is published when it bound nothing new).

* **Kind:** mutation

* **Authorization:** `compute:pools:manage` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field         | Type           | Required | Constraints                                                       |
| ------------- | -------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`    | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`      | string         | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `externalIds` | list of string | yes      | 1-100 items                                                       |

## unbindMachines [#unbindmachines]

`POST /api/compute/pools/unbindMachines`

Takes machines out of the pool; returns `unbound` and `skipped`, with the input of `bindMachines`. An instance that holds an unbound machine keeps it.

* A machine discovery has not seen is skipped as `not-found`, one bound to another pool as `bound-elsewhere`. A machine that is not bound is listed as unbound and changes nothing.

* Decided as `compute:pools:manage` on the pool; only the platform and the account's tenant unbind (`ACCESS_DENIED` otherwise). Publishes `pool.machines.unbound` with the `externalIds` this call unbound.

* **Kind:** mutation

* **Authorization:** `compute:pools:manage` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field         | Type           | Required | Constraints                                                       |
| ------------- | -------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`    | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`      | string         | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `externalIds` | list of string | yes      | 1-100 items                                                       |

## machines [#machines]

`POST /api/compute/pools/machines`

Lists the machines bound to the pool that the provider still reports, ordered by zone and machine id; returns `items`, a `cursor` when more remain, and `discoveredAt`, the time of the account's last successful discovery pass.

* **Input.** `filter` is `free` (the default: no instance holds the machine), `attached` (one does) or `all`. `limit` is 1 to 500 (default 100).

* **Output.** Each machine is shown as `providerAccounts.machines` shows it, with `poolId`. A machine held by an instance of another tenant is listed without that instance's `instanceId` and `tenantId`, except to the platform. Tenants the pool is offered to are not shown `accountId`, as `get` does not show them `providerAccountId`.

* **IAM.** Decided as `compute:pools:read` on the pool: its owner and the tenants it is offered to read it. Any other tenant gets `NOT_FOUND`.

* **Kind:** read

* **Authorization:** `compute:pools:read` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type                             | Required | Constraints                                                       |
| ---------- | -------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string                           | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`   | string                           | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `filter`   | one of `free`, `attached`, `all` |          | default `"free"`                                                  |
| `cursor`   | string                           |          | at most 1024 characters                                           |
| `limit`    | integer                          |          | 1-500; default `100`                                              |


# profiles (/docs/reference/api/profiles)

> The profiles group stores and checks the versioned recipes that provisioning workflows run: steps, drivers, an image, user data and host config. A version becomes immutable the first time a workflow uses it. Lookups see the tenant's own profiles, public ones, and the read-only profiles that modules ship under the tenant `builtin`. See [Profiles and images](/docs/guides/bare-metal/profiles-and-images).



The profiles group stores and checks the versioned recipes that provisioning workflows run: steps, drivers, an image, user data and host config. A version becomes immutable the first time a workflow uses it. Lookups see the tenant's own profiles, public ones, and the read-only profiles that modules ship under the tenant `builtin`. See [Profiles and images](/docs/guides/bare-metal/profiles-and-images).

All 7 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                      | Phase | Status     |
| ----------------------- | -------- | -------------------------------------------------- | ----- | ---------- |
| [`create`](#create)     | mutation | `compute:profiles:manage` @ `compute-tenant`       | v0.2  | registered |
| [`get`](#get)           | read     | `compute:profiles:read` @ `compute-profile`        | v0.2  | registered |
| [`validate`](#validate) | read     | `compute:profiles:read` @ `compute-profile`        | v0.2  | registered |
| [`render`](#render)     | read     | `compute:profiles:read` @ `compute-profile`        | v0.2  | registered |
| [`list`](#list)         | read     | `compute:profiles:read` @ `compute-tenant` + items | v0.2  | registered |
| [`update`](#update)     | mutation | `compute:profiles:manage` @ `compute-profile`      | v0.2  | registered |
| [`delete`](#delete)     | mutation | `compute:profiles:manage` @ `compute-profile`      | v0.2  | registered |

## create [#create]

`POST /api/compute/profiles/create`

Stores a new profile version from a profile document, `private` unless `visibility` says `public`.

* Decided as `compute:profiles:manage` on the tenant.

* The document is checked like `defineProfile`; any problem is `INVALID_INPUT`, with `details.path` naming the first offending field.

* `CONFLICT` when that id and version already exist; create the next version instead.

* **Kind:** mutation

* **Authorization:** `compute:profiles:manage` @ `compute-tenant`

* **Module:** `provision`

* **Phase:** v0.2 (registered)

**Input**

| Field        | Type                       | Required | Constraints                                                |
| ------------ | -------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId`   | string                     | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `document`   | object                     | yes      |                                                            |
| `visibility` | one of `private`, `public` |          | default `"private"`                                        |

## get [#get]

`POST /api/compute/profiles/get`

Returns one profile version, the latest when `profileVersion` is omitted.

The lookup prefers the tenant's own profile, then a public stored one, then one a module ships. `NOT_FOUND` when none matches.

* **Kind:** read
* **Authorization:** `compute:profiles:read` @ `compute-profile`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                                 |
| ---------------- | ------- | -------- | --------------------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                  |
| `profileId`      | string  | yes      | at most 128 characters; matches `^[a-z0-9](?:[a-z0-9._-]{0,126}[a-z0-9])?$` |
| `profileVersion` | integer |          | 1-1000000                                                                   |

## validate [#validate]

`POST /api/compute/profiles/validate`

Expands a profile version against the registered steps and returns `valid` with every issue found, such as a `use` that does not resolve or a `with` its step rejects.

Problems in the profile are returned as `issues` rather than thrown; an unknown profile is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:profiles:read` @ `compute-profile`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                                 |
| ---------------- | ------- | -------- | --------------------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                  |
| `profileId`      | string  | yes      | at most 128 characters; matches `^[a-z0-9](?:[a-z0-9._-]{0,126}[a-z0-9])?$` |
| `profileVersion` | integer |          | 1-1000000                                                                   |

## render [#render]

`POST /api/compute/profiles/render`

Returns the profile document and the expanded steps a workflow of this version would run, with their phases.

`INVALID_INPUT` when the profile does not resolve, with the problems in `details.issues`; `NOT_FOUND` for an unknown profile.

* **Kind:** read
* **Authorization:** `compute:profiles:read` @ `compute-profile`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                                 |
| ---------------- | ------- | -------- | --------------------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                  |
| `profileId`      | string  | yes      | at most 128 characters; matches `^[a-z0-9](?:[a-z0-9._-]{0,126}[a-z0-9])?$` |
| `profileVersion` | integer |          | 1-1000000                                                                   |

## list [#list]

`POST /api/compute/profiles/list`

Returns a page of the tenant's stored profiles, public ones and those modules ship, ordered by id and version.

`limit` defaults to 50 and is at most 200; pass the returned `cursor` to read the next page.

* **Kind:** read
* **Authorization:** `compute:profiles:read` @ `compute-tenant` + items
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                |
| ---------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `limit`    | integer |          | 1-200; default `50`                                        |
| `cursor`   | string  |          | at most 256 characters                                     |

## update [#update]

`POST /api/compute/profiles/update`

Replaces the document of one of the tenant's profile versions, or changes its visibility, and returns the profile.

* Pass `document`, `visibility` or both; neither is `INVALID_INPUT`.

* A version a workflow has used keeps its document: `INVALID_STATE`. Its visibility can still change.

* A new document must keep the profile's id and version (`INVALID_INPUT`).

* `expectedVersion` is the profile's row `version`; a mismatch is `CONFLICT`.

* `NOT_FOUND` when the tenant has no such version.

* **Kind:** mutation

* **Authorization:** `compute:profiles:manage` @ `compute-profile`

* **Module:** `provision`

* **Phase:** v0.2 (registered)

**Input**

| Field             | Type                       | Required | Constraints                                                                 |
| ----------------- | -------------------------- | -------- | --------------------------------------------------------------------------- |
| `tenantId`        | string                     | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                  |
| `profileId`       | string                     | yes      | at most 128 characters; matches `^[a-z0-9](?:[a-z0-9._-]{0,126}[a-z0-9])?$` |
| `profileVersion`  | integer                    | yes      | 1-1000000                                                                   |
| `document`        | any                        |          |                                                                             |
| `visibility`      | one of `private`, `public` |          |                                                                             |
| `expectedVersion` | integer                    |          | at least 1                                                                  |

## delete [#delete]

`POST /api/compute/profiles/delete`

Hides one version, or every version when `profileVersion` is omitted, from new workflows and returns the number deleted.

Workflows already started keep the profile document they snapshotted. Profiles that modules ship are not stored, so deleting one is `NOT_FOUND`, as is a profile the tenant does not have.

* **Kind:** mutation
* **Authorization:** `compute:profiles:manage` @ `compute-profile`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                                 |
| ---------------- | ------- | -------- | --------------------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`                  |
| `profileId`      | string  | yes      | at most 128 characters; matches `^[a-z0-9](?:[a-z0-9._-]{0,126}[a-z0-9])?$` |
| `profileVersion` | integer |          | 1-1000000                                                                   |


# projects (/docs/reference/api/projects)

> Projects group a tenant's work without creating more tenants. A project is app-owned, lives in one tenant, and its creator becomes its owner in the authorization projection; membership is a Better IAM relationship on `compute-project/{id}`. See [Tenancy](/docs/guides/iam/tenancy).



Projects group a tenant's work without creating more tenants. A project is app-owned, lives in one tenant, and its creator becomes its owner in the authorization projection; membership is a Better IAM relationship on `compute-project/{id}`. See [Tenancy](/docs/guides/iam/tenancy).

All 4 methods are served by @better-compute/server 0.3.0.

| Method                | Kind     | Authorization                                      | Phase | Status     |
| --------------------- | -------- | -------------------------------------------------- | ----- | ---------- |
| [`create`](#create)   | mutation | `compute:projects:manage` @ `compute-tenant`       | v0.1  | registered |
| [`get`](#get)         | read     | `compute:projects:read` @ `compute-project`        | v0.1  | registered |
| [`list`](#list)       | read     | `compute:projects:read` @ `compute-tenant` + items | v0.1  | registered |
| [`archive`](#archive) | mutation | `compute:projects:manage` @ `compute-project`      | v0.2  | registered |

## create [#create]

`POST /api/compute/projects/create`

Creates an `active` project with a DNS-label name that is unique in the tenant; returns the project.

* Decided with `compute:projects:manage` on `compute-tenant/{tenantId}`.

* Idempotent per tenant, caller identity and `idempotencyKey`: the project id derives from them, and the same request again returns the existing project without a second IAM decision. The same key with a different name is `CONFLICT`.

* A second project with the same name in the tenant is `CONFLICT`.

* Emits `project.created`.

* **Kind:** mutation

* **Authorization:** `compute:projects:manage` @ `compute-tenant`

* **Module:** `server`

* **Phase:** v0.1 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                             |
| ---------------- | ------ | -------- | ----------------------------------------------------------------------- |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `name`           | string | yes      | at most 63 characters; matches `^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$` |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`          |

## get [#get]

`POST /api/compute/projects/get`

Returns one project of the tenant by id.

Decided with `compute:projects:read` on `compute-project/{projectId}`. `NOT_FOUND` when the project is not in the tenant.

* **Kind:** read
* **Authorization:** `compute:projects:read` @ `compute-project`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `projectId` | string | yes      | at most 64 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$` |

## list [#list]

`POST /api/compute/projects/list`

Lists the tenant's projects the caller may read, ordered by id, optionally filtered by `state` (`active` or `archived`); returns a page with a cursor.

Decided with `compute:projects:read` on the tenant, then per project. Items the caller may not read are dropped from the page, so a page can hold fewer than `limit` items (at most 50, the default) while a cursor is still returned.

* **Kind:** read
* **Authorization:** `compute:projects:read` @ `compute-tenant` + items
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type                        | Required | Constraints                                                |
| ---------- | --------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string                      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `state`    | one of `active`, `archived` |          |                                                            |
| `limit`    | integer                     |          | 1-50; default `50`                                         |
| `cursor`   | string                      |          | at most 256 characters; matches `^[A-Za-z0-9._:-]{1,256}$` |

## archive [#archive]

`POST /api/compute/projects/archive`

Marks a project `archived` and increments its version; returns the project.

* Decided with `compute:projects:manage` on `compute-project/{projectId}`.

* Archiving an archived project returns it unchanged and emits nothing.

* Emits `project.archived`. `NOT_FOUND` when the project is not in the tenant.

* **Kind:** mutation

* **Authorization:** `compute:projects:manage` @ `compute-project`

* **Module:** `server`

* **Phase:** v0.2 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `projectId` | string | yes      | at most 64 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$` |


# providerAccounts (/docs/reference/api/provider-accounts)

> Provider accounts connect a tenant to an external provider registered with `vps({ providers })`, such as a cloud or a dedicated-server host. Each account's `config` is validated by its provider's manifest and holds secret references, never secret values. Provider pools create instances through an account, and `bootstrap` turns one of its machines into a bcd host. See [Providers](/docs/guides/vps/providers).



Provider accounts connect a tenant to an external provider registered with `vps({ providers })`, such as a cloud or a dedicated-server host. Each account's `config` is validated by its provider's manifest and holds secret references, never secret values. Provider pools create instances through an account, and `bootstrap` turns one of its machines into a bcd host. See [Providers](/docs/guides/vps/providers).

All 23 methods are served by @better-compute/server 0.3.0.

| Method                                          | Kind     | Authorization                                                                                        | Phase | Status     |
| ----------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------- | ----- | ---------- |
| [`create`](#create)                             | mutation | `compute:providers:manage` @ `compute-fleet` or `compute:providers:manage` @ `compute-tenant`        | v0.3  | registered |
| [`list`](#list)                                 | read     | `compute:providers:read` @ `compute-tenant` + items                                                  | v0.3  | registered |
| [`federationSetup`](#federationsetup)           | read     | `compute:providers:read` @ `compute-tenant`                                                          | v1.0  | registered |
| [`connectSetup`](#connectsetup)                 | read     | `compute:providers:read` @ `compute-tenant`                                                          | v1.0  | registered |
| [`connect`](#connect)                           | mutation | `compute:providers:manage` @ `compute-fleet` or `compute:providers:manage` @ `compute-tenant`        | v1.0  | registered |
| [`get`](#get)                                   | read     | `compute:providers:read` @ `compute-provider-account`                                                | v0.3  | registered |
| [`offers`](#offers)                             | read     | `compute:providers:read` @ `compute-provider-account`                                                | v0.3  | registered |
| [`check`](#check)                               | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`findings`](#findings)                         | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`billing`](#billing)                           | read     | `compute:providers:manage` @ `compute-provider-account`                                              | v1.0  | registered |
| [`update`](#update)                             | mutation | `compute:providers:manage` @ `compute-provider-account`                                              | v0.3  | registered |
| [`delete`](#delete)                             | mutation | `compute:providers:manage` @ `compute-provider-account`                                              | v0.3  | registered |
| [`bootstrap`](#bootstrap)                       | mutation | `compute:providers:manage` @ `compute-provider-account`                                              | v0.3  | registered |
| [`machines`](#machines)                         | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`discover`](#discover)                         | mutation | `compute:providers:manage` @ `compute-provider-account`                                              | v1.0  | registered |
| [`reservations`](#reservations)                 | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`bindReservation`](#bindreservation)           | mutation | `compute:providers:manage` @ `compute-provider-account` or `compute:flavors:manage` @ `compute-pool` | v1.0  | registered |
| [`unbindReservation`](#unbindreservation)       | mutation | `compute:providers:manage` @ `compute-provider-account`                                              | v1.0  | registered |
| [`reservationOfferings`](#reservationofferings) | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`purchaseReservation`](#purchasereservation)   | mutation | `compute:providers:purchase` @ `compute-provider-account`                                            | v1.0  | registered |
| [`purchases`](#purchases)                       | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`clusters`](#clusters)                         | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |
| [`purchaseProposals`](#purchaseproposals)       | read     | `compute:providers:read` @ `compute-provider-account`                                                | v1.0  | registered |

## create [#create]

`POST /api/compute/providerAccounts/create`

Stores a provider account for the tenant after validating its `config` with the provider's manifest; returns the account.

* **Idempotency.** `idempotencyKey` is optional. With a key, the account id derives from the tenant, the caller and the key, and a repeat returns the stored account without a second IAM decision; a different `provider` or `name` under the same key is `CONFLICT`. A name the tenant already uses for another account is also `CONFLICT`.

* **IAM.** Decided as `compute:providers:manage` on the tenant.

* **Secret references.** An account of the root (platform) tenant may name any `secret://`, `vault://` or `env://` reference. An account of any other tenant, for every provider, may name only `secret://<its tenant>/…` references (secrets it put with `secrets.put`), plus the `vault://` paths the host allows that tenant with `vps({ tenantVaultPaths })` (an entry ending in `/` allows every path under it). It may never name `env://`, which reads the control plane's own environment, nor another tenant's or the platform's secrets: the account would run on the platform's credentials. The same rule holds when the provider resolves the account's credentials: a stored account that names a reference it may not use (written before this rule, or around the API) fails that call with `ACCESS_DENIED` and the value is never read, and a tenant's `secret://` reference is resolved for that tenant, so the store refuses a secret whose stored tenant is another.

* **Errors.** `INVALID_INPUT` for an unknown provider (`details.known` lists the registered ones), a config the manifest rejects (`details.issues`), or a field ending in `token`, `secret`, `password` or `key` whose value is not a `secret://`, `vault://` or `env://` reference. `INVALID_CONFIG` for a secret reference the account may not use (see above; `details.path` names the field, `details.reason` is `secretRefScope`), or when the provider refuses to build a client for the account: a Google Cloud account of a tenant other than the root (platform) tenant must name its own service-account key in `credentials`, or instead sign in keyless with `federationAudience` (its workload identity pool provider, `//iam.googleapis.com/projects/<n>/locations/global/workloadIdentityPools/<pool>/providers/<provider>`; see `federationSetup`), because without either it would run on the control plane's own identity (the metadata server or the host's token source).

* **Endpoints (address policy).** A provider whose endpoints the account's config chooses (Proxmox's `url`; Hetzner when the host overrides its API URL) has each endpoint resolved before the account is stored: an endpoint that is not `https`, carries credentials, is an internal name (`localhost`, `*.internal`, `*.local`, `*.svc`, `*.cluster.local`, a single label) or resolves to any private, link-local (the metadata server included), loopback, CGNAT, multicast, reserved, documentation, unique-local or cluster address (`vps({ addressPolicy: { denyCidrs } })`) is `INVALID_CONFIG` with `details.reason` `endpoint-scheme`, `endpoint-address` (and `details.category`; the resolved address is never shown) or `endpoint-unresolved`; a resolver outage is `UNAVAILABLE`. The account's client then connects only through a fetch that checks the address again and connects to the one it checked (DNS pinning) and never follows redirects. The host may exempt ranges (`allowCidrs`), names (`allowHosts`) or its own platform accounts (`platformAccounts: 'exempt'`); a tenant's account is never exempt by account. Google Cloud accounts are not affected: their hosts come from host options only.

* **Pass-through (`billing`).** `billing.passThrough` (at most 50 rules of `{ skuPattern, meter, multiplier }`) passes the account's provider costs through to tenants (D11). The `vps.costs` worker reads the account's cost rows (for Google Cloud, the BigQuery billing export named by `billingExport`) and attributes each to a tenant by its `better-compute-instance` label (the instance's tenant), else its `better-compute-tenant` label. A regular-usage row attributed to a tenant other than the platform (and, for a row of an instance, an instance root bills: not `billing: none`, in a pool whose billing mode is `metered`), in the deployment currency (`vps({ billing: { currency } })`), whose SKU id or description contains `skuPattern` (case-insensitive; the first matching rule wins) and whose net cost (cost + credits) is positive becomes one usage event on `meter` for that tenant: `quantity` the row's usage, `cost` = net cost × `multiplier`, `occurredAt` the usage start (or, for a row whose usage month has already closed, `vps({ billing: { closeGraceHours } })` hours after the month ends, default 6, the first hour of the current UTC month: a late charge on the next statement, with the same key; the stored row keeps its usage start), tags `region`, `provider`, `source: cost` and the instance's `pool`, key `costx:<accountId>:<rowId>`. It is written as a `cost` usage window and recorded by the normal usage flush, once: a later correction of the row is published as `billing.cost-corrected` with `passedThrough: true` for a root credit note, never billed again. Rules are allowed only on the root (platform) tenant's accounts (another tenant's account is `INVALID_INPUT`: its costs are its own and are never passed through) and only for providers that declare which SKUs compute meters itself. `meter` must be a `reported` meter on the IAM root tenant (create it on the root tenant with `pricing: 'reported'`, through better-iam's `billing.createMeter` or the IAM port's `ensureMeters`: `billing.syncMeters` creates only the catalog's rate-card meters; otherwise `INVALID_INPUT`, and `UNSUPPORTED` when the IAM port cannot list meters); `multiplier` is 1 to 10. A `skuPattern` that matches a SKU family compute meters itself (for Compute Engine, `core`, `ram`, `gpu` and `instance` SKUs, and persistent disk and Hyperdisk capacity, local SSD storage (`SSD backed Local Storage`) and external IP charges) is `INVALID_CONFIG` (`details.reason` `meteredSku`), and a row of such a SKU is never passed through whatever the pattern, so nothing is billed twice.

* **Keyless Google Cloud accounts.** For an account with `federationAudience`, `create` and `update` exchange tokens with Google before the account is stored, with no transaction open, and refuse it with `INVALID_CONFIG` and `details.reason`: `credentialsAndFederation` (both `credentials` and `federationAudience`), `noWorkloadIssuer` (the host runs no workload issuer), `federationAudience` (not a workload identity pool provider's resource name), `federationTrustsAnySubject` (the pool provider also accepts a subject that is not this account's: its attribute condition must be exactly `federationSetup`'s), `federationRefused` (Google STS refuses the account's own subject), `impersonationRefused` (the account may not impersonate the `impersonate` service account) or `projectAccess` (the resulting token cannot read the project). An audience another provider account already uses, of this tenant or any other, is `CONFLICT` with the message `Another provider account already uses this gcp identity` and no `details`. Google being unreachable, slow or throttling is `UNAVAILABLE` or `RATE_LIMITED`, retryable: nothing is stored, and a retry with the same `idempotencyKey` runs the checks again.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-fleet` or `compute:providers:manage` @ `compute-tenant`

* **Module:** `vps`

* **Phase:** v0.3 (registered)

**Input**

| Field                              | Type           | Required | Constraints                                                    |
| ---------------------------------- | -------------- | -------- | -------------------------------------------------------------- |
| `tenantId`                         | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`     |
| `provider`                         | string         | yes      | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`        |
| `name`                             | string         | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`   |
| `config`                           | any            |          | default `{}`                                                   |
| `idempotencyKey`                   | string         |          | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$` |
| `billing`                          | object         |          |                                                                |
| `billing.passThrough`              | list of object |          | 0-50 items                                                     |
| `billing.passThrough[].skuPattern` | string         | yes      | 2-128 characters                                               |
| `billing.passThrough[].meter`      | string         | yes      | at most 64 characters; matches `^[a-z][a-z0-9_.-]{0,63}$`      |
| `billing.passThrough[].multiplier` | number         | yes      | 1-10                                                           |

## list [#list]

`POST /api/compute/providerAccounts/list`

Returns the tenant's provider accounts, sorted by name.

Each `config` holds only secret references.

* **Kind:** read
* **Authorization:** `compute:providers:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## federationSetup [#federationsetup]

`POST /api/compute/providerAccounts/federationSetup`

Returns what a tenant sets up at its cloud to give a provider account keyless sign-in (Google Cloud workload identity federation), for the account it is about to create: `{ issuer, jwksUri, subject, attributeCondition, allowedAudienceHint }`.

* **Input.** `tenantId`, `provider` and `name`: the `provider` and `name` the tenant will give `create`. Nothing is stored and no provider is called.

* **Output.** `issuer` is the control plane's workload issuer and `jwksUri` its keys (`<issuer>/jwks.json`). `subject` is `compute-account:<tenantId>/<provider>/<name>`, exactly the subject `create` gives the account (vps builds it in one place). `attributeCondition` is the CEL condition `assertion.sub == "<subject>"` to set on the tenant's workload identity pool provider, so it trusts that account and no other. `allowedAudienceHint` is the shape of the pool provider's resource name as an `https://` URL, with `<PROJECT_NUMBER>`, `<POOL>` and `<PROVIDER>` left for the tenant to fill in. See [Keyless GCP accounts](/docs/guides/vps/providers#keyless-gcp-accounts).

* **IAM.** Decided as `compute:providers:read` on the tenant.

* **Errors.** `INVALID_INPUT` for an invalid `name` or `provider`, or an unknown provider (`details.known` lists the registered ones), as in `create`. `INVALID_CONFIG` (`details.reason` `noWorkloadIssuer`) when the host runs no workload issuer.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-tenant`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                  |
| ---------- | ------ | -------- | ------------------------------------------------------------ |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`   |
| `provider` | string | yes      | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`      |
| `name`     | string | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$` |

## connectSetup [#connectsetup]

`POST /api/compute/providerAccounts/connectSetup`

Returns the provider's authorization URL for connecting an account with OAuth (`{ authorizeUrl }`): the person's browser goes there, signs in at the provider and comes back to `redirectUri` with a code and the caller's `state`. See [Connect a DigitalOcean account](/docs/operations/digitalocean).

* **Input.** `tenantId`, `provider`, `redirectUri` (one the platform registered for the provider's OAuth app, exactly as written) and `state` (16 to 256 URL-safe characters: the caller's anti-forgery value, which it keeps and compares on the way back). Nothing is stored and no provider is called.

* **IAM.** Decided as `compute:providers:read` on the tenant.

* **Errors.** `UNSUPPORTED` for a provider without a connect flow (DigitalOcean has one only when the host registers at least one redirect URI). `INVALID_INPUT` for an unknown provider, an invalid `state`, or a redirect URI the platform did not register.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-tenant`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field         | Type   | Required | Constraints                                                |
| ------------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId`    | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `provider`    | string | yes      | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`    |
| `redirectUri` | string | yes      | at most 2048 characters                                    |
| `state`       | string | yes      | matches `^[A-Za-z0-9._~-]{16,256}$`                        |

## connect [#connect]

`POST /api/compute/providerAccounts/connect`

Stores a provider account whose credential comes from an OAuth authorization code; returns the account, as `create` does.

* **Input.** `tenantId`, `provider`, `name`, `code` (the code the redirect carried; single use), `redirectUri` (the one `connectSetup` was given), `config` (the rest of the account's config, without the credential field: naming it is `INVALID_INPUT`) and an optional `idempotencyKey`.

* **Flow.** With no transaction open, the provider trades the code for a credential (DigitalOcean: a refresh token), which is sealed in the deployment's secret store for the tenant and named in the provider's credential field (`refreshToken`); the account's config is then checked as in `create` and the account stored. The credential is never stored elsewhere, logged or answered. When the account is not stored (a refused check, a name the tenant already uses, a failed transaction, or a request that never reaches the store) the sealed credential is deleted again.

* **Idempotency.** As `create`: with a key, a repeat returns the stored account without spending another code; a different `provider` or `name` under the same key is `CONFLICT`. A retry without the key needs a new code.

* **IAM.** Decided as `compute:providers:manage` on the tenant.

* **Errors.** `UNSUPPORTED` for a provider without a connect flow. `INVALID_INPUT` for a code the provider refuses, a redirect URI the platform did not register, or a config naming the credential field. `INVALID_CONFIG` when the host runs no secret broker, or as in `create`. `CONFLICT` for a name the tenant already uses.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-fleet` or `compute:providers:manage` @ `compute-tenant`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                    |
| ---------------- | ------ | -------- | -------------------------------------------------------------- |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`     |
| `provider`       | string | yes      | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`        |
| `name`           | string | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`   |
| `code`           | string | yes      | matches `^[A-Za-z0-9._~+/=-]{1,512}$`                          |
| `redirectUri`    | string | yes      | at most 2048 characters                                        |
| `config`         | any    |          | default `{}`                                                   |
| `idempotencyKey` | string |          | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$` |

## get [#get]

`POST /api/compute/providerAccounts/get`

Returns one provider account of the tenant.

When its provider describes it, the account carries `onboarding`: what the tenant still sets up at its cloud. An `aws` account shows its `mode` (`assume-role`, `web-identity`, `access-key` or `ambient`); a role the control plane assumes also shows the platform-assigned `externalId` its trust must require and, when the host names it, the `platformPrincipalArn` to trust; a keyless (`webIdentityRoleArn`) account shows the `subject` and `audience` its role must trust. It never holds a secret.

An account of another tenant, or none, is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:providers:read` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## offers [#offers]

`POST /api/compute/providerAccounts/offers`

Asks the provider, through the account, for the machine types it offers; returns them as `items`.

The optional `query` filters by `kind` (`metal` or `vps`), `region`, `gpuModel`, `minGpus`, `minVcpu` and `minMemoryGiB`. The call reaches the provider live. A missing account is `NOT_FOUND`; a stored account naming a secret reference its tenant may not use is `ACCESS_DENIED` (see `create`).

* **Kind:** read
* **Authorization:** `compute:providers:read` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field                | Type                  | Required | Constraints                                                        |
| -------------------- | --------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`           | string                | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`          | string                | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `query`              | object                |          |                                                                    |
| `query.kind`         | one of `metal`, `vps` |          |                                                                    |
| `query.region`       | string                |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`       |
| `query.gpuModel`     | string                |          | at most 32 characters; matches `^[A-Za-z0-9-]{1,32}$`              |
| `query.minGpus`      | integer               |          | 0-64                                                               |
| `query.minVcpu`      | integer               |          | 0-1024                                                             |
| `query.minMemoryGiB` | integer               |          | 0-16384                                                            |

## check [#check]

`POST /api/compute/providerAccounts/check`

Asks the provider what a stored account's credential may do, read-only at the provider; returns `{ accountId, provider, checkedAt, ok, missing, optional?, details?, rateLimit? }`.

* **Output.** `ok` is true when the credential may do everything instances need (`missing` is empty). `missing` lists the permissions instances need that it lacks, `optional` those only optional features need (costs, metrics, security findings), in the provider's own names (DigitalOcean: token scopes such as `droplet:read`). `details` is the provider's own detail (DigitalOcean: the account's status and per-family results), never a secret. `rateLimit` is the credential's request rate limit at the provider as its answers to the check counted it, `{ limit, remaining, resetAt? }` (epoch ms; DigitalOcean: the token's hourly limit, which every process using the token spends from); it is left out when the provider does not report one or reports one that is not well-formed.

* **Reads.** The stored config is validated and its secret references checked again, exactly as for a client. A provider with no probe but an account check answers `ok` when the check passes. The call stops reaching the provider when the request is cancelled.

* **IAM.** Decided as `compute:providers:read` on the provider account.

* **Errors.** `NOT_FOUND` for a missing account. `UNSUPPORTED` for a provider that cannot check an account. `INVALID_CONFIG` when the provider refuses the credential outright (a revoked DigitalOcean token, or an OAuth grant that must be connected again, `details.reason` `oauthGrant`); `UNAVAILABLE` or `RATE_LIMITED` when the provider cannot be asked (an OAuth refresh token another process may still be rotating is `UNAVAILABLE`, `oauthRace`, the first time).

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## findings [#findings]

`POST /api/compute/providerAccounts/findings`

Returns what the provider says an operator should look at on a stored account (DigitalOcean: its latest security scan), read-only at the provider: `{ accountId, provider, readAt, items }`.

* **Items.** Each is `{ id, kind, severity, title, detail?, at?, group? }`: `id` is stable across reads (the provider's rule id) and unique in one answer, `kind` says what it is about (`security`), `severity` is `critical`, `warning` or `info`, and `group` joins findings that read as one. Malformed findings and repeated ids are left out, and at most 200 are kept. A part of the account the credential may not read is itself a finding (DigitalOcean without `security:read`: `security:unreadable`), never an empty answer that looks clean. Never a secret.

* **Reads.** With the account's own credential, as `check`; the call stops reaching the provider when the request is cancelled. The admin panel reads it at most every ten minutes per account.

* **IAM.** Decided as `compute:providers:read` on the provider account.

* **Errors.** `NOT_FOUND` for a missing account; `UNSUPPORTED` for a provider that reports no findings; the provider's errors as in `check`.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## billing [#billing]

`POST /api/compute/providerAccounts/billing`

Returns the account's own bill at its provider (DigitalOcean: its balance, month to date, invoices and billing history), read-only at the provider: `{ accountId, provider, readAt, currency, monthToDateMicros?, balanceMicros?, monthToDateUsageMicros?, generatedAt?, invoices, history }`. This is what the provider charges the account, not what compute meters; the platform's margins and leaks are `billing.margins` and `billing.leaks`.

* **Output.** Amounts are integer micros of `currency`. `balanceMicros` is the balance as of the last invoice (negative is credit). `invoices` (`{ id, period, amountMicros }`, `period` `YYYY-MM`) are the final invoices, newest first, at most 120; `history` (`{ at, type, description, amountMicros }`) the newest billing history entries (invoices, payments, credits), newest first, at most 50. Malformed entries are left out and text is bounded. Never a secret.

* **Reads.** With the account's own credential (DigitalOcean: scope `billing:read`), as `check`; the call stops reaching the provider when the request is cancelled.

* **IAM.** Decided as `compute:providers:manage` on the provider account: the history names payments, so reading the bill takes the right to manage the account, not only to read it.

* **Errors.** `NOT_FOUND` for a missing account; `UNSUPPORTED` for a provider that cannot read an account's bill; `ACCESS_DENIED` when the credential may not read billing; the provider's other errors as in `check`.

* **Kind:** read

* **Authorization:** `compute:providers:manage` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/providerAccounts/update`

Replaces the account's `config`, validated as in `create`, and increments its version; returns the account. `billing` replaces the pass-through rules (see `create`); absent keeps them, `{}` clears them. Rows already passed through are never passed through again, whatever the new rules say. New rules apply to the account's rows not passed through yet whose usage started within the cost lookback (`vps({ costs: { lookbackDays } })`, default 3): whether each may be passed through is decided again in the same transaction.

The provider and the name cannot change. The next provider call builds a new client from the new config. A missing account is `NOT_FOUND`; an invalid config is `INVALID_INPUT`, and a secret reference the account may not use, or a config the provider refuses to build a client for, is `INVALID_CONFIG`, as in `create`. A keyless Google Cloud account's new config is checked with Google as in `create`, with the same `INVALID_CONFIG` reasons, `CONFLICT` for an audience another account uses, and retryable `UNAVAILABLE` or `RATE_LIMITED`; a refused update changes nothing.

* **Kind:** mutation
* **Authorization:** `compute:providers:manage` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field                              | Type           | Required | Constraints                                                        |
| ---------------------------------- | -------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`                         | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`                        | string         | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `config`                           | any            |          |                                                                    |
| `billing`                          | object         |          |                                                                    |
| `billing.passThrough`              | list of object |          | 0-50 items                                                         |
| `billing.passThrough[].skuPattern` | string         | yes      | 2-128 characters                                                   |
| `billing.passThrough[].meter`      | string         | yes      | at most 64 characters; matches `^[a-z][a-z0-9_.-]{0,63}$`          |
| `billing.passThrough[].multiplier` | number         | yes      | 1-10                                                               |

## delete [#delete]

`POST /api/compute/providerAccounts/delete`

Deletes a provider account that no pool, no instance and no network uses; returns `deleted: true`.

Archived pools, deleted instances and deleted networks do not count: deleted instances keep their history but drop the reference, and deleted networks, with their subnets and rules, are removed with the account. A pool that is not archived, an instance that is not deleted, or a network that is not deleted (named in the message, ids in `details.networkIds`) on the account makes the call `INVALID_STATE`. A missing account is `NOT_FOUND`.

* **Kind:** mutation
* **Authorization:** `compute:providers:manage` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## bootstrap [#bootstrap]

`POST /api/compute/providerAccounts/bootstrap`

Records a request to turn one of the account's machines into a bcd host of a `vps-host` or `metal-host` pool; returns the bootstrap, which the `vps.bootstraps` reconciler then drives to `joined` or `failed`.

* **Input.** Name exactly one of `machineId` (a machine the account already has; starts in `identifying`) or `offerId` (a machine to claim or order; starts in `claiming`). The pool must be owned by the tenant, not archived, and run the `bcd` agent.

* **Idempotency.** `idempotencyKey` is required. A repeat returns the bootstrap's current status; a different request under the same key is `CONFLICT`.

* **Errors.** `INVALID_CONFIG` when `vps()` has no `bootstrap` options or the host has no PKI. `UNSUPPORTED` when the provider cannot identify and bootstrap machines. `NOT_FOUND` for a missing account or a pool the tenant does not own, `INVALID_STATE` for an archived pool, `INVALID_INPUT` for a pool of another class or agent.

* **Events.** `provider.bootstrap.started`; the reconciler later publishes `provider.bootstrap.installed`, `provider.bootstrap.joined` or `provider.bootstrap.failed`. The join token it mints is single-use, bound to the machine, and never returned or stored.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v0.3 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`      | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `poolId`         | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `machineId`      | string |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `offerId`        | string |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## machines [#machines]

`POST /api/compute/providerAccounts/machines`

Lists the machines the `vps.discovery` worker last saw in the account, ordered by zone and machine id; returns `items`, a `cursor` when more remain, and `discoveredAt`, the time of the account's last successful pass.

* **Input.** `filter` is `free` (the default: machines no instance holds), `attached` (machines attached to, or created as, an instance) or `all`. `includeGone` also lists machines the provider stopped reporting, with `goneAt`. `limit` is 1 to 500 (default 100).

* **Output.** Each machine has its state, zone, region, machine type, vCPU, memory, GPUs, labels, networks, addresses and any reservation it consumes. An attached or created machine has `instanceId` and the instance's `tenantId`, unless that instance is another tenant's (the account's pool is offered to it): then both are left out, except for the platform (root callers). A machine bound to a pool (`pools.bindMachines`) has `poolId`.

* **Errors.** An account of another tenant, or none, is `NOT_FOUND`. The worker lists accounts every 5 minutes by default (`vps({ discovery: { every } })`) and never changes a machine.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field         | Type                             | Required | Constraints                                                        |
| ------------- | -------------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`    | string                           | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`   | string                           | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `filter`      | one of `free`, `attached`, `all` |          | default `"free"`                                                   |
| `includeGone` | boolean                          |          | default `false`                                                    |
| `cursor`      | string                           |          | at most 1024 characters                                            |
| `limit`       | integer                          |          | 1-500; default `100`                                               |

## discover [#discover]

`POST /api/compute/providerAccounts/discover`

Makes the account's next discovery pass due now and wakes the worker; returns `queued: true`.

* **Input.** With `wait: true`, the answer is held, for at most 30 seconds, until a pass that started after the call has finished, and then carries that pass's time as `discoveredAt`. A timeout is not an error: the answer then has no `discoveredAt`.

* **Errors.** `UNSUPPORTED` when the account's provider cannot list its machines. An account of another tenant, or none, is `NOT_FOUND`.

* **Events.** The pass publishes `provider.machine.discovered`, `provider.machine.changed` (with `changes`) and `provider.machine.gone`.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type    | Required | Constraints                                                        |
| ----------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string  | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `wait`      | boolean |          | default `false`                                                    |

## reservations [#reservations]

`POST /api/compute/providerAccounts/reservations`

Returns the account's provider capacity reservations (for example GCE reservations) as the `vps.reservations` worker last saw them, as `items`.

* **Input.** `includeGone` also lists reservations the provider no longer reports (`goneAt` is set); by default they are left out.

* **Fields.** Each item is the provider's reservation (`id`, `name`, `zone`, `state`, `machineType`, `vcpu`, `memoryGiB`, `gpus`, `total`, `inUse`, `specificOnly`, `shared`, `startsAt`, `endsAt`, `offerId`) plus `pending` (instances admitted against it that the last scan may not have counted yet), `free` (`total - inUse - pending`, never below 0), `observedAt`, and `poolId` and `flavorId` while it is bound.

* **Freshness.** The worker asks each account's provider every `vps({ reservations: { every } })` (default `5m`); accounts whose provider cannot list reservations have none. The call itself does not reach the provider.

* **Events.** The worker publishes `provider.reservation.detected`, `changed`, `ready` (a scheduled or creating reservation became ready) and `gone`.

* **Errors.** `NOT_FOUND` for a missing account.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field         | Type    | Required | Constraints                                                        |
| ------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`    | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`   | string  | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `includeGone` | boolean |          | default `false`                                                    |

## bindReservation [#bindreservation]

`POST /api/compute/providerAccounts/bindReservation`

Makes a reservation usable from a provider pool: creates a flavor in the pool whose offer is the reservation's, and remembers the binding; returns `poolId` and `flavorId`.

* **Input.** The pool must be a provider pool of this account that the tenant owns. The reservation must be `ready` and not gone: a `scheduled` future reservation is listed but cannot be bound, because the provider fulfils it under a new id; bind it once it is ready. The flavor is named `flavorName`, default `rsv-<reservation name>`, and takes the reservation's `vcpu`, `memoryGiB` and GPU count; its disk is the account's `config.diskGiB` (default 100).

* **Idempotency.** `idempotencyKey` is required. Binding a reservation to the pool it is already bound to returns the existing binding; binding it to another pool is `CONFLICT`, as is a flavor name the pool already uses.

* **Admission.** `instances.create` on the flavor locks the reservation and takes one free slot, so concurrent creates cannot over-commit it. Between two scans a create counts the instances it admitted that the provider has not reported yet (`pending`). With no free slot, or a reservation that is not `ready` (a reservation that went away or is not ready cannot be bound, so this only happens after a binding was made and the reservation then changed), the create fails with `CAPACITY_UNAVAILABLE` and `details` naming `reservationId`, and `total`, `inUse`, `pending` or `state`.

* **Errors.** `NOT_FOUND` for a missing account, reservation, or a pool the tenant does not own. `INVALID_INPUT` for a pool that is not a provider pool of this account. `INVALID_STATE` for an archived pool, a gone reservation, or one that is not `ready` (`details` names `reservationId` and `state`).

* **IAM.** Decided as `compute:providers:manage` on the provider account.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-provider-account` or `compute:flavors:manage` @ `compute-pool`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`      | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reservationId`  | string | yes      | at most 300 characters; matches `^[A-Za-z0-9._/:-]{1,300}$`        |
| `poolId`         | string | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `flavorName`     | string |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`       |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## unbindReservation [#unbindreservation]

`POST /api/compute/providerAccounts/unbindReservation`

Removes a reservation's binding; returns `reservationId`.

The flavor `bindReservation` created is deleted if no instance uses it, and archived otherwise (instances that use it keep running; no new instance can use it). Unbinding a reservation that is not bound succeeds. The worker does the same when a bound reservation is no longer reported, and then publishes `provider.reservation.gone` with `poolId` and `flavorId`. A missing account or reservation is `NOT_FOUND`.

* **Kind:** mutation
* **Authorization:** `compute:providers:manage` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                        |
| --------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`     | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reservationId` | string | yes      | at most 300 characters; matches `^[A-Za-z0-9._/:-]{1,300}$`        |

## reservationOfferings [#reservationofferings]

`POST /api/compute/providerAccounts/reservationOfferings`

Lists capacity the account could buy ahead of time; returns `{ items }`, each `{ id, instanceType, zone, count, startsAt, endsAt, upfrontPrice, currency }`. A provider read: nothing is bought.

* **Input.** `query` names the `instanceType`, `count` (1 to 1000) and `durationHours` (1 to 4368), optionally bounded by `earliestStart` and `latestStart` (epoch ms).

* **Who.** Root (platform) tenant only, on the platform's own provider accounts, like `purchaseReservation`: anyone else, and root on a tenant's account, is `ACCESS_DENIED`. Decided as `compute:providers:read` on the provider account.

* **Provider calls.** The call reaches the provider live, with no transaction open.

* **AWS.** The items are On-Demand Capacity Reservations the account could create now, in each of its zones that offers the type (`upfrontPrice` 0: an ODCR bills on-demand rates for every reserved instance while it is active, used or not), and Capacity Blocks for ML on sale in the account's zones (1 to 64 instances). Offering ids describe themselves: `cb:<zone>/<type>/<count>/<hours>/<start epoch seconds>/<upfront cents>/<offering id>` for a block, `odcr:<zone>/<type>/<count>/<hours>` for an ODCR.

* **Errors.** `NOT_FOUND` for a missing account. `UNSUPPORTED` when the provider sells nothing ahead of time. `INVALID_INPUT` for an instance type the provider does not know.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field                 | Type    | Required | Constraints                                                        |
| --------------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`            | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`           | string  | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `query`               | object  | yes      |                                                                    |
| `query.instanceType`  | string  | yes      | at most 64 characters; matches `^[A-Za-z0-9._-]{1,64}$`            |
| `query.count`         | integer | yes      | 1-1000                                                             |
| `query.durationHours` | integer | yes      | 1-4368                                                             |
| `query.earliestStart` | integer |          | 0-140737488355328                                                  |
| `query.latestStart`   | integer |          | 0-140737488355328                                                  |

## purchaseReservation [#purchasereservation]

`POST /api/compute/providerAccounts/purchaseReservation`

Buys one offering; returns the purchase `{ id, accountId, offeringId, state, attempts, reservationId?, reservation?, error?, createdAt, updatedAt }`. &#x2A;*This spends money.**

* **Who.** Root (platform) tenant only, decided as `compute:providers:purchase` on the provider account; anyone else is `ACCESS_DENIED`, and so is a purchase from a provider account the root tenant does not own (capacity is bought only in the platform's own accounts).

* **Confirmation.** The request must carry `confirm: true`, else `CONFIRMATION_REQUIRED` (`details.reason` `confirm-spend`, with `offeringId` and `accountId`); retry with `confirm: true` and the same `idempotencyKey`. An agent acting for a person is held back until the person approves; the approval names what is bought (on AWS: instance type and count, zone, start and end, and the up-front price, or that an ODCR bills on-demand rates for its reserved instance-hours).

* **Idempotency.** `idempotencyKey` is required and scoped to (tenant, caller identity): a retry with the same key returns the same purchase and never buys twice, including two requests racing on one key. The same key for another account or offering is `CONFLICT`.

* **Exactly once.** The purchase is recorded in the request's transaction and bought by the `vps.purchases` worker outside any transaction, with a key of its own that the provider makes idempotent. An attempt that ends without the provider's answer (a timeout, a dropped connection) is retried with that key, and the provider is told an earlier attempt may have bought: on AWS a Capacity Block (whose purchase call has no idempotency token) is then looked for by what it is, and is not bought again until 15 minutes passed without it showing.

* **States.** `state` moves `requested` → `purchasing` → `purchased` (the reservation bought, which the reservations worker then lists so it can be bound), `failed` (`error`: the provider refused, nothing was bought) or `unknown` (the attempts ran out while one may have bought: check the account's reservations). A mutation's poll settles on any of the last three. Events: `provider.purchase.requested`, `completed`, `failed`, `unknown`.

* **Account gate.** The provider must be able to buy for this account: an AWS account must set `allowPurchases` (off by default) and be a platform account. Otherwise the request is `UNSUPPORTED` (`details.reason` `purchases-unavailable`) and nothing is recorded.

* **AWS.** `offeringId` is an id `reservationOfferings` listed (`cb:…` or `odcr:…`). A Capacity Block is bought only while EC2 still lists it at no more than the price in its id (else `failed` with `CONFLICT`, `price-changed`, or `CAPACITY_UNAVAILABLE`, `offering-gone`).

* **Kind:** mutation

* **Authorization:** `compute:providers:purchase` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                        |
| ---------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`      | string  | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `offeringId`     | string  | yes      | at most 300 characters; matches `^[A-Za-z0-9._/:-]{1,300}$`        |
| `idempotencyKey` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `confirm`        | boolean |          |                                                                    |

## purchases [#purchases]

`POST /api/compute/providerAccounts/purchases`

Returns the account's purchases, newest first (at most 500), as `items`; root (platform) tenant only, decided as `compute:providers:read` on the provider account.

* **Kind:** read
* **Authorization:** `compute:providers:read` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## clusters [#clusters]

`POST /api/compute/providerAccounts/clusters`

Lists the managed Kubernetes clusters the account's provider can see, whoever created them, live from the provider; returns `items`. A pool's `policy.kubernetes` names one of them to join its machines to it (for EKS, `policy.kubernetes.eks`).

* **Input.** `tenantId` and `accountId`. Nothing is stored; no transaction is open while the provider is read, and the listing has 55 seconds.

* **Output.** Each item has `name`, `id` (on EKS `<region>/<name>`), `location`, `state` (`pending`, `ready`, `updating`, `deleting`, `error`), `version` and `endpoint` when the provider reports them, and `managed` (`true` when better-compute created it: on AWS the tag `better-compute:managed=true`). A ready EKS cluster also has `eks`: `{ name, apiServerEndpoint, certificateAuthority, cidr }`, the nodeadm settings `k8s-join` mode `eks` takes. Only these plain fields are returned: no token or kubeconfig. On AWS at most 100 clusters of the account's region are listed.

* **IAM.** Decided as `compute:providers:read` on the account.

* **Errors.** An account of another tenant, or none, is `NOT_FOUND`. `UNSUPPORTED` when the provider cannot list clusters. Provider failures pass through (`ACCESS_DENIED` when the account's identity may not list or describe clusters, or fails the account guards; `UNAVAILABLE` or `RATE_LIMITED`, retryable).

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |

## purchaseProposals [#purchaseproposals]

`POST /api/compute/providerAccounts/purchaseProposals`

Returns the account's purchase proposals, newest first (at most 500), as `items`, each `{ id, accountId, reservationId, poolId, tenantId, state, reason?, startsAt, endsAt, query?, offering?, description?, error?, attempts, purchase?, createdAt, updatedAt }`. Nothing is bought: a proposal is capacity a person may decide to buy with `purchaseReservation`.

* **Who.** Root (platform) tenant only, decided as `compute:providers:read` on the provider account; anyone else is `ACCESS_DENIED`. Proposals name other tenants' reservations and what the platform could spend.

* **Where they come from.** The `vps.purchase-proposals` worker (every 15 minutes, `vps({ purchaseProposals: { every } })`) looks at capacity reservations that are `pending` on a provider pool of one of the platform's own provider accounts and start within the next 48 hours. It asks the provider (`reservationOfferings`, a read) for whole machines of the pool's GPU flavor that wastes the fewest GPUs (a plain offer: not a reservation's or a Spot one), starting in the day before the reservation (not before now) and lasting until it ends, in whole days up to 14 days and whole weeks up to 182 days. It proposes the covering offering (it starts after now, no later than the reservation, and ends no earlier) that starts latest, then the cheapest. Each reservation gets one proposal; `provider.purchase.proposed` is published when an offering is proposed.

* **States.** `proposed` (`offering`, its `description`, and `purchase { id, state }` once root bought it), `none` (`reason`: `no-flavor` when the pool has no GPU flavor to buy, `no-offering` when nothing covers the reservation, `unsupported` when the provider sells nothing ahead of time), or `error` (`error { code, message }`: the provider read failed; tried again every 15 minutes, at most 5 attempts in all).

* **AWS.** The proposed offering is a Capacity Block for ML (`cb:…`); an ODCR, which starts at once, is never proposed. Buy it with `purchaseReservation` and `confirm: true` (the account must set `allowPurchases`); bind the bought reservation to the pool with `bindReservation` once the reservations worker lists it.

* **Kind:** read

* **Authorization:** `compute:providers:read` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |


# providerAddresses (/docs/reference/api/provider-addresses)

> Reserved public addresses on a provider account (DigitalOcean reserved IPv4 and IPv6 first), assigned to the account's [instances](/docs/reference/api/instances). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.addresses`. Reading takes `compute:addresses:read`, everything else `compute:addresses:manage`, on `provider-address/{id}`. An address books one `addresses` quota on the pool and bills `vps.address.hours` from `ready` until deleted. A DigitalOcean reserved IP carries no name or tag, so the row is the ownership record: one address is recorded by at most one live row, and it keeps the provider's fingerprint from the reserve where there is one (a reserved IPv6's reservation time; a reserved IPv4 has none). An address released at the provider outside better-compute (gone, mismatched, or handed to a later reserve) goes to `error` for good: never re-adopted, assigned or released, and its delete makes no provider call.



Reserved public addresses on a provider account (DigitalOcean reserved IPv4 and IPv6 first), assigned to the account's [instances](/docs/reference/api/instances). Every call records what the tenant wants and returns at once; the `vps.provider-resources` reconciler converges the provider through `ProviderImpl.addresses`. Reading takes `compute:addresses:read`, everything else `compute:addresses:manage`, on `provider-address/{id}`. An address books one `addresses` quota on the pool and bills `vps.address.hours` from `ready` until deleted. A DigitalOcean reserved IP carries no name or tag, so the row is the ownership record: one address is recorded by at most one live row, and it keeps the provider's fingerprint from the reserve where there is one (a reserved IPv6's reservation time; a reserved IPv4 has none). An address released at the provider outside better-compute (gone, mismatched, or handed to a later reserve) goes to `error` for good: never re-adopted, assigned or released, and its delete makes no provider call.

All 6 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                           | Phase | Status     |
| ----------------------- | -------- | ------------------------------------------------------- | ----- | ---------- |
| [`create`](#create)     | mutation | `compute:addresses:manage` @ `compute-provider-address` | v1.0  | registered |
| [`list`](#list)         | read     | `compute:addresses:read` @ `compute-tenant` + items     | v1.0  | registered |
| [`get`](#get)           | read     | `compute:addresses:read` @ `compute-provider-address`   | v1.0  | registered |
| [`assign`](#assign)     | mutation | `compute:addresses:manage` @ `compute-provider-address` | v1.0  | registered |
| [`unassign`](#unassign) | mutation | `compute:addresses:manage` @ `compute-provider-address` | v1.0  | registered |
| [`delete`](#delete)     | mutation | `compute:addresses:manage` @ `compute-provider-address` | v1.0  | registered |

## create [#create]

`POST /api/compute/providerAddresses/create`

Reserves an address (`family` `ipv4` or `ipv6`) in a region the account uses and returns it `pending`, booking one `addresses`. The name is unique in the tenant (`CONFLICT`); no room, or no quota row, is `QUOTA_EXCEEDED`. The same `idempotencyKey` returns the same address. With `prefix` (an IPv4 CIDR or the provider's id of a BYOIP prefix on the account) the address is drawn from the account's own address space instead of the provider's pool: IPv4 only (`INVALID_INPUT`), only for the provider account's owner tenant (`ACCESS_DENIED`, reason `byoipNotOwner`), and on DigitalOcean only with the account's `previews` enabled; a prefix that is not the account's, not active or in another region sends the address to `error`. A reserve whose answer was lost is never sent again (it could reserve a second address): the address goes to `error` saying so, and an operator checks the account. Publishes `providerAddress.created`, then `providerAddress.ready` with its `address`, or `providerAddress.error`.

* **Kind:** mutation
* **Authorization:** `compute:addresses:manage` @ `compute-provider-address`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                  | Required | Constraints                                                        |
| ---------------- | --------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string                | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`      | string                |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`         | string                | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`           | string                | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `description`    | string                |          | at most 256 characters                                             |
| `region`         | string                | yes      | at most 40 characters; matches `^[a-z][a-z0-9-]{0,38}[a-z0-9]$`    |
| `family`         | one of `ipv4`, `ipv6` |          | default `"ipv4"`                                                   |
| `prefix`         | string                |          | at most 43 characters                                              |
| `idempotencyKey` | string                | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/providerAddresses/list`

Returns the tenant's addresses in the order they were made (at most 1000), optionally of one pool or project. Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:addresses:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/providerAddresses/get`

Returns one address: the IP, its state, the instance it is wanted on and `assignmentState` (`none`, `pending`, `ready`, `deleting`, `error` with `assignmentMessage`).

* **Kind:** read
* **Authorization:** `compute:addresses:read` @ `compute-provider-address`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `addressId` | string | yes      | at most 58 characters; matches `^paddr_[0-9a-hjkmnp-tv-z]{26,52}$` |

## assign [#assign]

`POST /api/compute/providerAddresses/assign`

Puts the address on an instance of the tenant on the same provider account and region and returns it with `assignmentState: 'pending'`; the reconciler assigns it and publishes `providerAddress.assigned`. Another tenant's instance is `ACCESS_DENIED`, another region `INVALID_INPUT`, an address on another instance `CONFLICT` (unassign it first). The reconciler never takes an address over from a machine it did not put it on. An address whose reservation is gone (`error`, lost) is `INVALID_STATE`. Also decided as `compute:instances:update` on the instance, with the caller's own credential.

* **Kind:** mutation
* **Authorization:** `compute:addresses:manage` @ `compute-provider-address`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                        |
| ------------ | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `addressId`  | string | yes      | at most 58 characters; matches `^paddr_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## unassign [#unassign]

`POST /api/compute/providerAddresses/unassign`

Takes the address off its instance and keeps it reserved; the reconciler unassigns it and publishes `providerAddress.unassigned`. An unassigned address is returned as it is.

* **Kind:** mutation
* **Authorization:** `compute:addresses:manage` @ `compute-provider-address`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `addressId` | string | yes      | at most 58 characters; matches `^paddr_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/providerAddresses/delete`

Marks the address for deletion and returns it `deleting`; the reconciler releases it at the provider, refunds its quota and publishes `providerAddress.deleted`. An assigned address is `INVALID_STATE` (unassign it first). An address whose IP the provider has since handed to another reservation is not released: the row is finished with a warning in the log.

* **Kind:** mutation
* **Authorization:** `compute:addresses:manage` @ `compute-provider-address`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                        |
| ----------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `addressId` | string | yes      | at most 58 characters; matches `^paddr_[0-9a-hjkmnp-tv-z]{26,52}$` |


# providerSnapshots (/docs/reference/api/provider-snapshots)

> Snapshots of [provider volumes](/docs/reference/api/provider-volumes), made with `providerVolumes.snapshot` and converged by the `vps.provider-resources` reconciler. Reading takes `compute:volumes:read`, deleting and restoring `compute:volumes:manage`, on `provider-snapshot/{id}`. A snapshot books no quota and bills `vps.snapshot.gib.hours` (the GiB the provider reports) from `ready` until deleted.



Snapshots of [provider volumes](/docs/reference/api/provider-volumes), made with `providerVolumes.snapshot` and converged by the `vps.provider-resources` reconciler. Reading takes `compute:volumes:read`, deleting and restoring `compute:volumes:manage`, on `provider-snapshot/{id}`. A snapshot books no quota and bills `vps.snapshot.gib.hours` (the GiB the provider reports) from `ready` until deleted.

All 4 methods are served by @better-compute/server 0.3.0.

| Method                | Kind     | Authorization                                          | Phase | Status     |
| --------------------- | -------- | ------------------------------------------------------ | ----- | ---------- |
| [`list`](#list)       | read     | `compute:volumes:read` @ `compute-tenant` + items      | v1.0  | registered |
| [`get`](#get)         | read     | `compute:volumes:read` @ `compute-provider-snapshot`   | v1.0  | registered |
| [`delete`](#delete)   | mutation | `compute:volumes:manage` @ `compute-provider-snapshot` | v1.0  | registered |
| [`restore`](#restore) | mutation | `compute:volumes:manage` @ `compute-provider-snapshot` | v1.0  | registered |

## list [#list]

`POST /api/compute/providerSnapshots/list`

Returns the tenant's snapshots in the order they were made (at most 1000), optionally of one volume. Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:volumes:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId`       | string  |          | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/providerSnapshots/get`

Returns one snapshot: its volume, name, region, size and state.

* **Kind:** read
* **Authorization:** `compute:volumes:read` @ `compute-provider-snapshot`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                        |
| ------------ | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `snapshotId` | string | yes      | at most 58 characters; matches `^psnap_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/providerSnapshots/delete`

Marks the snapshot for deletion and returns it `deleting`; the reconciler deletes it at the provider and publishes `providerVolume.snapshot.deleted`. Deleting a snapshot being deleted returns it. A volume restored from it is not affected.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-snapshot`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                        |
| ------------ | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `snapshotId` | string | yes      | at most 58 characters; matches `^psnap_[0-9a-hjkmnp-tv-z]{26,52}$` |

## restore [#restore]

`POST /api/compute/providerSnapshots/restore`

Creates a new volume from a `ready` snapshot (`INVALID_STATE` otherwise), in the snapshot's pool and region, and returns it `pending`, as [`providerVolumes.create`](/docs/reference/api/provider-volumes) does: `sizeGiB` is at least the snapshot's (`INVALID_INPUT`; absent, the snapshot's size) and books `diskGiB`, the name is unique in the tenant, and the same `idempotencyKey` returns the same volume. The volume keeps the snapshot's filesystem: `filesystem` is `INVALID_INPUT` (a format could wipe the restored data). Only a volume snapshot this deployment made restores (the provider refuses any other, `INVALID_STATE`). A snapshot of a shared volume does not restore yet: `UNSUPPORTED` (`details.reason` `sharedRestore`). Decided on the snapshot.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-snapshot`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                | Type    | Required | Constraints                                                        |
| -------------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`           | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `snapshotId`         | string  | yes      | at most 58 characters; matches `^psnap_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `name`               | string  | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `description`        | string  |          | at most 256 characters                                             |
| `sizeGiB`            | integer |          | 1-16384                                                            |
| `deleteWithInstance` | boolean |          | default `false`                                                    |
| `idempotencyKey`     | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |


# providerVolumes (/docs/reference/api/provider-volumes)

> Block volumes and shared volumes (network file shares) on a provider account (DigitalOcean Block Storage and Network File Storage first), next to the account's [instances](/docs/reference/api/instances). Every call records what the tenant wants and returns at once, usually `pending`; the `vps.provider-resources` reconciler then converges the provider through `ProviderImpl.volumes` without blocking on provider actions, and writes back what it saw (`state` is `pending`, `ready`, `updating`, `deleting`, `deleted` or `error`, with the provider's `message`). Reading takes `compute:volumes:read`, everything else `compute:volumes:manage`, on `provider-volume/{id}`. A volume books its `sizeGiB` on the tenant's `diskGiB` quota for the pool and bills from `ready` until deleted: a block volume `vps.volume.gib.hours`, a shared volume `vps.share.gib.hours`. Once created, a volume is only read: one deleted at the provider outside better-compute goes to `error` and is never created again (its data is gone; delete it). See the [provider volumes guide](/docs/guides/vps/provider-volumes).



Block volumes and shared volumes (network file shares) on a provider account (DigitalOcean Block Storage and Network File Storage first), next to the account's [instances](/docs/reference/api/instances). Every call records what the tenant wants and returns at once, usually `pending`; the `vps.provider-resources` reconciler then converges the provider through `ProviderImpl.volumes` without blocking on provider actions, and writes back what it saw (`state` is `pending`, `ready`, `updating`, `deleting`, `deleted` or `error`, with the provider's `message`). Reading takes `compute:volumes:read`, everything else `compute:volumes:manage`, on `provider-volume/{id}`. A volume books its `sizeGiB` on the tenant's `diskGiB` quota for the pool and bills from `ready` until deleted: a block volume `vps.volume.gib.hours`, a shared volume `vps.share.gib.hours`. Once created, a volume is only read: one deleted at the provider outside better-compute goes to `error` and is never created again (its data is gone; delete it). See the [provider volumes guide](/docs/guides/vps/provider-volumes).

All 8 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                        | Phase | Status     |
| ----------------------- | -------- | ---------------------------------------------------- | ----- | ---------- |
| [`create`](#create)     | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |
| [`list`](#list)         | read     | `compute:volumes:read` @ `compute-tenant` + items    | v1.0  | registered |
| [`get`](#get)           | read     | `compute:volumes:read` @ `compute-provider-volume`   | v1.0  | registered |
| [`attach`](#attach)     | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |
| [`detach`](#detach)     | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |
| [`resize`](#resize)     | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |
| [`snapshot`](#snapshot) | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |
| [`delete`](#delete)     | mutation | `compute:volumes:manage` @ `compute-provider-volume` | v1.0  | registered |

## create [#create]

`POST /api/compute/providerVolumes/create`

Records a volume in a pool on a provider account and returns it `pending`, booking its `sizeGiB` (1 to 16384) on `diskGiB`.

* `region` must be one the account uses (`INVALID_INPUT`); a pool whose provider offers no volumes is `UNSUPPORTED`. `filesystem` (`ext4`, `xfs` or `none`) formats it at create; it is never changed later.
* `kind` is `block` (the default: a disk on one instance at a time) or `shared` (a network file share many instances of one tenant network mount at once). A shared volume needs `networkId`, a network of the tenant on the same provider account, provisioned, not being deleted (`INVALID_STATE`), with a subnet in `region` (`INVALID_INPUT` without one, `INVALID_STATE` while it is not ready); `networkId` on a block volume is `INVALID_INPUT`. A shared volume takes no `filesystem` and no `deleteWithInstance: true` (`INVALID_INPUT`). A provider that makes no shared volumes is `UNSUPPORTED` (`details.reason` `sharedVolumes`).
* What the provider would refuse before any request is refused here, before quota is booked: for example a DigitalOcean share under 50 GiB is `INVALID_INPUT` (`details.reason` `size`).
* `instanceId` attaches it once both exist: the instance must be the tenant's (`ACCESS_DENIED`), on the same account and region (`INVALID_INPUT`), for a shared volume on its network (`INVALID_INPUT`), and not being deleted (`INVALID_STATE`). The caller also needs `compute:instances:update` on that instance, decided on its own credential.
* `deleteWithInstance: true` deletes the volume with the instance it is attached to (on DigitalOcean, the Droplet's selective destroy names it); otherwise deleting the instance only detaches it.
* The name is unique in the tenant among volumes that are not deleted (`CONFLICT`). A tenant without room on `diskGiB`, or without a `diskGiB` quota row, is `QUOTA_EXCEEDED` (fail closed). The same `idempotencyKey` with the same input returns the same volume; with other input it is `CONFLICT`.

Publishes `providerVolume.created`, then `providerVolume.ready` (with its `device`, the path the guest sees) or `providerVolume.error`. A create the provider refused (an account volume limit, a name taken by a volume that is not this deployment's) is `error` and is retried by the periodic pass; deleting it makes no provider call.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field                | Type                         | Required | Constraints                                                        |
| -------------------- | ---------------------------- | -------- | ------------------------------------------------------------------ |
| `tenantId`           | string                       | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`          | string                       |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `poolId`             | string                       | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`               | string                       | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `description`        | string                       |          | at most 256 characters                                             |
| `region`             | string                       | yes      | at most 40 characters; matches `^[a-z][a-z0-9-]{0,38}[a-z0-9]$`    |
| `sizeGiB`            | integer                      | yes      | 1-16384                                                            |
| `kind`               | one of `block`, `shared`     |          | default `"block"`                                                  |
| `networkId`          | string                       |          | at most 56 characters; matches `^net_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `filesystem`         | one of `ext4`, `xfs`, `none` |          |                                                                    |
| `deleteWithInstance` | boolean                      |          | default `false`                                                    |
| `instanceId`         | string                       |          | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `idempotencyKey`     | string                       | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## list [#list]

`POST /api/compute/providerVolumes/list`

Returns the tenant's volumes in the order they were made (at most 1000), optionally of one pool or project, with their attachment (a shared volume: every attachment, as `attachments`). Deleted ones only with `includeDeleted`.

* **Kind:** read
* **Authorization:** `compute:volumes:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `includeDeleted` | boolean |          | default `false`                                                   |

## get [#get]

`POST /api/compute/providerVolumes/get`

Returns one volume: its `kind`, state, observed size, `device`, and the attachment (`instanceId`, `state`, `desiredState`) when it has one. A shared volume shows its `networkId`, every attachment as `attachments`, and `mount` (`source`, the NFS source `<host>:<path>` a guest mounts, and `type` `nfs4`) once the provider reports one.

* **Kind:** read
* **Authorization:** `compute:volumes:read` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId` | string | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |

## attach [#attach]

`POST /api/compute/providerVolumes/attach`

Attaches the volume to an instance of the tenant on the same provider account and region, and returns it with the attachment `pending`; the reconciler attaches it at the provider without a reboot and publishes `providerVolume.attached`. Attaching it again to the same instance returns it unchanged. A block volume on another instance is `CONFLICT` (detach it first), one being detached `INVALID_STATE`. A shared volume attaches to any number of instances, each at most once; the instance must be on the volume's network (`INVALID_INPUT`), and one it is being detached from is `INVALID_STATE` until that finishes. Also decided as `compute:instances:update` on the instance, with the caller's own credential.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId`   | string | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `instanceId` | string | yes      | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## detach [#detach]

`POST /api/compute/providerVolumes/detach`

Detaches the volume from an instance and returns it. `instanceId` names the instance: it is required for a shared volume (`INVALID_INPUT` without it); for a block volume, naming an instance it is not on changes nothing. Then the reconciler asks the provider even while an attach is still running there, and publishes `providerVolume.detached` once the provider has let it go. A volume not attached to that instance (or to anything) is returned as it is. A `deleteWithInstance` volume whose instance is being deleted is `INVALID_STATE`: it goes with the machine.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId`   | string | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `instanceId` | string |          | at most 57 characters; matches `^inst_[0-9a-hjkmnp-tv-z]{26,52}$` |

## resize [#resize]

`POST /api/compute/providerVolumes/resize`

Grows the volume to `sizeGiB`, booking the difference on `diskGiB`, and returns it; the reconciler resizes it at the provider (`updating`, then `ready`) and publishes `providerVolume.resized`. A smaller size is `INVALID_INPUT`; the same size returns it unchanged. The filesystem is not grown: the guest does that.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId` | string  | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `sizeGiB`  | integer | yes      | 1-16384                                                           |

## snapshot [#snapshot]

`POST /api/compute/providerVolumes/snapshot`

Records a snapshot of the volume named `name` and returns it `pending` (a [provider snapshot](/docs/reference/api/provider-snapshots)); a retry with the same `idempotencyKey` returns the same snapshot. A volume has at most 25 snapshots that are not deleted (`QUOTA_EXCEEDED`); the name is unique among them (`CONFLICT`). Decided on the volume.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type   | Required | Constraints                                                        |
| ---------------- | ------ | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `volumeId`       | string | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `name`           | string | yes      | at most 42 characters; matches `^[a-z]([-a-z0-9]{0,40}[a-z0-9])?$` |
| `idempotencyKey` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## delete [#delete]

`POST /api/compute/providerVolumes/delete`

Marks the volume for deletion and returns it `deleting`; the reconciler deletes it at the provider, refunds its `diskGiB` and publishes `providerVolume.deleted`. A volume any instance still has attached is `INVALID_STATE` (detach it first; a shared volume from every instance). A volume never created at the provider, or whose create was refused, is finished without a provider call; one whose create answer was lost is looked up by name (never created to be deleted). Deleting a volume being deleted returns it.

* **Kind:** mutation
* **Authorization:** `compute:volumes:manage` @ `compute-provider-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `volumeId` | string | yes      | at most 57 characters; matches `^pvol_[0-9a-hjkmnp-tv-z]{26,52}$` |


# quotas (/docs/reference/api/quotas)

> A quota caps how much of one resource a tenant may hold in one pool at the same time, as a `max` and a `used` counter per tenant, pool and resource. A missing row grants 0; `pools.create` and `pools.offer` create the rows, and admissions charge them with `QUOTA_EXCEEDED` when a request does not fit.



A quota caps how much of one resource a tenant may hold in one pool at the same time, as a `max` and a `used` counter per tenant, pool and resource. A missing row grants 0; `pools.create` and `pools.offer` create the rows, and admissions charge them with `QUOTA_EXCEEDED` when a request does not fit.

* **Tenant-level rows.** A row whose `poolId` is `*` caps the tenant's total over every pool (`vcpu`, `memoryGiB`, `diskGiB`, `gpu` and other device classes, and `gpu.<gpuModel>` for the GPUs of pools of one model). A missing tenant row limits nothing. Every charge and refund on any of the tenant's pools also moves the tenant row, all or nothing: a refusal there is `QUOTA_EXCEEDED` with `details.poolId: '*'` and `details.scope: 'tenant'`, and no pool row changes.
* **Counted resources.** `instances` counts one per instance (`instances.create` and `instances.attach`, hosted or provider) and `sandboxes` one per sandbox; a sandbox's instance is not counted as an instance, and warm instances count as neither. Unlike scalars, a missing row limits nothing, on a pool or tenant-wide. Each instance records what its admission took and gives exactly that back when it is deleted, so a row set later counts the work admitted from then on.
* **Flavor allowlists.** A row may list `flavors`: an admission that would consume the row (create, attach, resize, sandbox) with another flavor is `QUOTA_EXCEEDED` with `details.reason: 'flavor-not-allowed'`. Reservations and volumes have no flavor and are not restricted. See [Quotas and usage](/docs/guides/capacity/quotas-and-usage).

All 2 methods are served by @better-compute/server 0.3.0.

| Method        | Kind     | Authorization                             | Phase | Status     |
| ------------- | -------- | ----------------------------------------- | ----- | ---------- |
| [`get`](#get) | read     | `compute:quotas:read` @ `compute-tenant`  | v0.1  | registered |
| [`set`](#set) | mutation | `compute:quotas:manage` @ `compute-fleet` | v0.2  | registered |

## get [#get]

`POST /api/compute/quotas/get`

Returns the tenant's quota rows, each with `max`, `used`, `available` and, when the row lists them, `flavors`; optionally only those of one pool, or with `poolId: '*'` only the tenant-level rows.

Rows are ordered by pool, then resource. `available` is `max` minus `used`, never below 0.

* **Kind:** read
* **Authorization:** `compute:quotas:read` @ `compute-tenant`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                             |
| ---------- | ------ | -------- | ----------------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `poolId`   | string |          | at most 57 characters; matches `^(\*\|pool_[0-9a-hjkmnp-tv-z]{26,52})$` |

## set [#set]

`POST /api/compute/quotas/set`

Sets one tenant's maximum for one resource in one pool, or tenant-wide with `poolId: '*'`, creating the row if needed, and returns the row with the maximum that took effect.

* It is decided as `compute:quotas:manage` on `compute-fleet`, which resolves only in the root tenant, so only callers in the root tenant get past the tenant pre-check. `tenantId` names the tenant whose quota is set.
* A maximum below the current `used` is raised to `used`. To shrink a quota under live reservations, set it again once they have released.
* A new tenant-level row of a scalar starts at the tenant's current total of it over its pool rows (`gpu.<model>`: its GPUs on pools of that model); a counted resource starts at 0.
* `flavors` sets the row's flavor allowlist; `null` clears it, and leaving it out keeps it.
* `NOT_FOUND` when the pool does not exist.

Emits `quota.updated` with the resource and the requested `max` (and `flavors` when given); a tenant-level row's event has subject `tenant/<tenantId>` and `scope: 'tenant'`.

* **Kind:** mutation
* **Authorization:** `compute:quotas:manage` @ `compute-fleet`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type       | Required | Constraints                                                             |
| ---------- | ---------- | -------- | ----------------------------------------------------------------------- |
| `tenantId` | string     | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `poolId`   | string     | yes      | at most 57 characters; matches `^(\*\|pool_[0-9a-hjkmnp-tv-z]{26,52})$` |
| `resource` | string     | yes      | at most 64 characters; matches `^[A-Za-z][A-Za-z0-9._-]{0,63}$`         |
| `max`      | integer    | yes      | at least 0                                                              |
| `flavors`  | array,null |          | 1-64 items                                                              |


# releases (/docs/reference/api/releases)

> Releases are signed bcd builds the platform may roll out to its nodes. A release is registered from a signed manifest, which must verify under the release key pins given to `rolloutModule`, and rollouts can then target its version. Both methods decide `compute:fleet:admin` on the fleet scope. See [Rollouts](/docs/guides/rollouts).



Releases are signed bcd builds the platform may roll out to its nodes. A release is registered from a signed manifest, which must verify under the release key pins given to `rolloutModule`, and rollouts can then target its version. Both methods decide `compute:fleet:admin` on the fleet scope. See [Rollouts](/docs/guides/rollouts).

All 2 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                           | Phase | Status     |
| ----------------------- | -------- | --------------------------------------- | ----- | ---------- |
| [`register`](#register) | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`list`](#list)         | read     | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |

## register [#register]

`POST /api/compute/releases/register`

Verifies a signed bcd release manifest under the configured release pins and records the release; returns it with its per-platform binaries.

* **Replay.** Registering a manifest whose version is already recorded with the same sequence returns the stored release; the same version with another sequence is `CONFLICT`.

* **Errors.** `INVALID_INPUT` when the manifest does not verify (`details.reason` says why), when its sequence is not above the latest registered one (a rollback of the release channel, `details.reason` is `rollback`), when it lists several binaries for one platform, or when it lists no Linux binary.

* **Kind:** mutation

* **Authorization:** `compute:fleet:admin` @ `compute-fleet`

* **Module:** `rollout`

* **Phase:** v0.4 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `manifest` | object | yes      |                                                            |

## list [#list]

`POST /api/compute/releases/list`

Returns every registered release, highest sequence first.

The list is not filtered by tenant: releases belong to the platform.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `rollout`
* **Phase:** v0.4 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |


# reservationGrants (/docs/reference/api/reservation-grants)

> Reservation grants commit slots of a provider capacity reservation to one tenant for a term. Root makes them: the reservation is the platform's (a GCE reservation on a root provider account, bound to a pool with `providerAccounts.bindReservation`), and a grant hands some of its slots to a tenant the pool is offered to, from `startsAt` to `endsAt`. Times are epoch milliseconds.



Reservation grants commit slots of a provider capacity reservation to one tenant for a term. Root makes them: the reservation is the platform's (a GCE reservation on a root provider account, bound to a pool with `providerAccounts.bindReservation`), and a grant hands some of its slots to a tenant the pool is offered to, from `startsAt` to `endsAt`. Times are epoch milliseconds.

While a grant is in its term, its slots are the grantee's. `instances.create` on the reservation's flavor puts the grantee's instance into its grant while the grant has a slot its instances do not hold; once the grant is full, the instance takes an ordinary slot of the reservation if one is free. Nobody else can take a granted slot, even while the grantee leaves it idle: others share `total - in use - unused granted slots`. A grant that names a `projectId` serves only that project's instances. A reservation that is physically full refuses everyone, the grantee included, with `CAPACITY_UNAVAILABLE`.

Billing: the grantee is billed `vm.grant.slot.hours` for every committed slot, every hour of the term, used or not, once each hour has ended (the `vps.grants` reconciler writes them as usage, tagged with the reservation's region, the pool and the provider). Its instances inside the grant record only the informational `vm.grant.usage.hours`, never compute meters, so nothing is billed twice; an instance beyond its grant is billed at the reservation rate (`vm.reserved.*`). When a grant ends, its billing stops at `endsAt` and its running instances are billed at the reservation rate from then on, unless another grant of the tenant on the same reservation is live then (a renewal), which takes them on up to its free slots. Likewise a grant that starts while its tenant already runs instances on the reservation outside any grant takes them in, oldest first, from its start; instances attached with `billing: 'none'` are never taken in.

While a grant is `scheduled` or `active`, `providerAccounts.unbindReservation` and `providerAccounts.delete` refuse with `INVALID_STATE` (`details.grantIds`); deleting the account removes its ended and cancelled grants, once `vps.grants` has metered every committed slot-hour of the ended ones (until then `providerAccounts.delete` is `INVALID_STATE` with `details.grantIds`; retry after the reconciler's next pass past the hour the grant ended in). When a tenant is deleted, its active grants end and its scheduled ones are cancelled.

A grant's stopped instances keep their slots for the grantee: others never get a slot the grant pays for.

When the provider stops reporting the reservation (the `vps.reservations` scan marks it gone), its active grants end at that moment and its scheduled ones are cancelled, as `end` would, with `reason: 'reservation-gone'` on their `reservation-grant.ended` / `.cancelled` events. Its instances are no longer reserved: they are billed on-demand from then on. When a scan sees the reservation's `total` drop below the slots its live grants hold (at their peak overlap), it publishes `reservation-grant.capacity-short` `{ accountId, reservationId, total, grantedSlots, grantIds }` in the platform's tenant, once per change of `total`. The grants keep billing: root decides which to end.

States: `scheduled` until `startsAt`, `active` during the term, then `ended`; `cancelled` when ended before it started. Each change publishes a `reservation-grant.*` event (`created`, `active`, `ended`, `cancelled`) in the grantee's tenant, with `{ grantId, poolId, projectId?, slots, startsAt, endsAt }`.

All 4 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                           | Phase | Status     |
| ------------------- | -------- | ------------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:providers:manage` @ `compute-provider-account` | v1.0  | registered |
| [`get`](#get)       | read     | `compute:quotas:read` @ `compute-tenant`                | v1.0  | registered |
| [`list`](#list)     | read     | `compute:quotas:read` @ `compute-tenant`                | v1.0  | registered |
| [`end`](#end)       | mutation | `compute:providers:manage` @ `compute-provider-account` | v1.0  | registered |

## create [#create]

`POST /api/compute/reservationGrants/create`

Root only. Commits `slots` of a reservation to `grantTenantId` (optionally one of its projects) for `[startsAt, endsAt)` and returns the grant.

* **Who.** Decided as `compute:providers:manage` on the provider account. `tenantId` must be the root tenant and the caller a member of it; anyone else is `ACCESS_DENIED`.

* **The reservation** must be on that account, `ready`, and bound to a pool that is offered to the grantee: otherwise `NOT_FOUND`, `INVALID_STATE` (not ready, gone or unbound) or `INVALID_INPUT` (the pool is not offered to the grantee).

* **Capacity.** The slots of every grant (other than cancelled ones) whose term overlaps, plus these, must fit the reservation's `total`: otherwise `CAPACITY_UNAVAILABLE` with `details { total, granted }`. Concurrent grants are decided one at a time under the reservation's row lock, so they never over-commit it.

* **Project.** A `projectId` must be a project of the grantee (`INVALID_INPUT`, `details.reason` `project-not-grantee`).

* **Term.** `startsAt` may be at most 5 minutes in the past; `endsAt` must be after it (`INVALID_INPUT`).

* **Idempotency.** A retry with the same `idempotencyKey` returns the grant; a different request under the key is `CONFLICT`.

* **Kind:** mutation

* **Authorization:** `compute:providers:manage` @ `compute-provider-account`

* **Module:** `vps`

* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                        |
| ---------------- | ------- | -------- | ------------------------------------------------------------------ |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `accountId`      | string  | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reservationId`  | string  | yes      | at most 300 characters; matches `^[A-Za-z0-9._/:-]{1,300}$`        |
| `grantTenantId`  | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |
| `projectId`      | string  |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `slots`          | integer | yes      | 1-100000                                                           |
| `startsAt`       | integer | yes      | at least 0                                                         |
| `endsAt`         | integer | yes      | at least 0                                                         |
| `idempotencyKey` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`         |

## get [#get]

`POST /api/compute/reservationGrants/get`

Returns one grant of the tenant. Root, acting in the root tenant, can read any grant. Decided as `compute:quotas:read` on the tenant. A grant of another tenant is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:quotas:read` @ `compute-tenant`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                         |
| ---------- | ------ | -------- | ------------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`          |
| `grantId`  | string | yes      | at most 59 characters; matches `^rgrant_[0-9a-hjkmnp-tv-z]{26,52}$` |

## list [#list]

`POST /api/compute/reservationGrants/list`

Returns the tenant's grants as `items`, newest first (at most 1000). With `all: true`, root (acting in the root tenant) lists every tenant's grants; anyone else asking for `all` is `ACCESS_DENIED`. A plain read decided as `compute:quotas:read` on the tenant (the grants are filtered to the tenant, not checked one by one).

* **Kind:** read
* **Authorization:** `compute:quotas:read` @ `compute-tenant`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                |
| ---------- | ------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `all`      | boolean |          | default `false`                                            |

## end [#end]

`POST /api/compute/reservationGrants/end`

Root only, decided as `compute:providers:manage` on the provider account. Ends the grant now: `endsAt` becomes the current time (never later than it was), so the hours already in the term stay committed and the rest are released. A grant that has not started yet is `cancelled` instead. Ending a grant that is already `ended` or `cancelled` returns it unchanged. A grant of another account is `NOT_FOUND`.

* **Kind:** mutation
* **Authorization:** `compute:providers:manage` @ `compute-provider-account`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                         |
| ----------- | ------ | -------- | ------------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`          |
| `accountId` | string | yes      | at most 58 characters; matches `^pacct_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `grantId`   | string | yes      | at most 59 characters; matches `^rgrant_[0-9a-hjkmnp-tv-z]{26,52}$` |


# reservations (/docs/reference/api/reservations)

> A reservation is a tenant's claim on devices of one pool for a time window. `create` first writes the request as a draft, Better IAM decides once on that draft with the request as attributes, and only an allowed draft is charged against quota and placed. A reservation holds its quota charge until it reaches a terminal state, and the `capacity.reservations` reconciler drives it through `placed`, `binding` and `active` to `released`. See [Reservations](/docs/guides/capacity/reservations).



A reservation is a tenant's claim on devices of one pool for a time window. `create` first writes the request as a draft, Better IAM decides once on that draft with the request as attributes, and only an allowed draft is charged against quota and placed. A reservation holds its quota charge until it reaches a terminal state, and the `capacity.reservations` reconciler drives it through `placed`, `binding` and `active` to `released`. See [Reservations](/docs/guides/capacity/reservations).

All 7 methods are served by @better-compute/server 0.3.0.

| Method                | Kind     | Authorization                                          | Phase | Status     |
| --------------------- | -------- | ------------------------------------------------------ | ----- | ---------- |
| [`create`](#create)   | mutation | `compute:reservations:create` @ `compute-reservation`  | v0.1  | registered |
| [`get`](#get)         | read     | `compute:reservations:read` @ `compute-reservation`    | v0.1  | registered |
| [`list`](#list)       | read     | `compute:reservations:read` @ `compute-tenant` + items | v0.1  | registered |
| [`release`](#release) | mutation | `compute:reservations:release` @ `compute-reservation` | v0.1  | registered |
| [`cancel`](#cancel)   | mutation | `compute:reservations:release` @ `compute-reservation` | v0.2  | registered |
| [`extend`](#extend)   | mutation | `compute:reservations:extend` @ `compute-reservation`  | v0.1  | registered |
| [`renew`](#renew)     | mutation | `compute:reservations:extend` @ `compute-reservation`  | v0.1  | registered |

## create [#create]

`POST /api/compute/reservations/create`

Books devices of one pool for a time window after one IAM decision on the draft; returns the reservation, `placed` or `pending`.

* **Idempotency.** The reservation id is derived from the tenant, the caller's identity and `idempotencyKey`. The same key with the same request returns the stored reservation without a second decision once it is past the draft stage, and decides the same draft again while it is still a draft. The same key with a different request is `CONFLICT`, with `details.reservationId`; a key whose request was refused stays `ACCESS_DENIED`. `wait` is not part of the compared request.
* **Decision.** `compute:reservations:create` is decided on the draft (`compute-reservation`), with attributes such as `gpuModel`, `gpuCount`, `durationHours` and `region`. When an agent in a delegated session is refused and IAM opens a confirmation request, the call fails with `CONFIRMATION_REQUIRED` and the draft waits in `awaiting-confirmation` for at most 30 minutes; a retry with the same key after approval continues it.
* **Errors.** `NOT_FOUND` when the pool is not owned by or offered to the tenant, or is archived. `INVALID_INPUT` when a device class or sharing mode does not match the pool's, `binding: 'node'` names a pool without `policy.tenantRoot`, `binding: 'kubernetes'` names a pool without `policy.kubernetes` or comes from a tenant other than the pool's owner (the kubelet gives bound GPUs to any pod of the cluster, so only the cluster's owner books into it), or a node binding asks for `resources`. `UNSUPPORTED` for any binding other than `hold`, `node` or `kubernetes`. `QUOTA_EXCEEDED` when the charge does not fit; the draft stays, so the same key can be retried. `SPEND_LIMIT_REACHED` when the pool or the tenant's offer asks for a budget check and the budget is spent.
* **Host resources.** `resources` (`vcpu`, `memoryGiB`, `hugepages1Gi`, per node for a gang) books host CPU, memory and 1 GiB hugepages on the same node as the devices, and the node enforces them. Without it, a request for whole GPUs books the pool's per-GPU share, stored on the reservation at creation. That share is `policy.perGpu` where it is set, or otherwise the smallest share any of the pool's GPU hosts can give (its allocatable CPUs or memory divided by its bookable GPUs). Requests for MIG instances or slots alone, and pools whose hosts report no CPU or memory rows, book none. The share is also the floor: `resources` below it (or `{}`) are raised to it, so a reservation's workload is always limited. The share rides on the GPUs, and needs no quota beyond theirs. Anything booked above it is charged to the tenant's `vcpu` and `memoryGiB` quota; 1 GiB hugepages count as memory. A missing quota row grants 0, so the excess is `QUOTA_EXCEEDED` unless the offer names these resources. IAM sees `vcpu` and `memoryGiB` as attributes (summed over a gang).
* **Placement.** A request that does not fit, and is not spot, may preempt spot work. Otherwise it is `pending` with an `Unschedulable` condition and retried every 30 seconds, or, with the pool-lease placer, `pending` with a `Queued` condition. `wait.timeoutMs` (at most 30 seconds) holds the answer while the reservation is queued; a timeout returns the reservation still queued, not an error.

Emits `reservation.placed`, `reservation.pending`, `reservation.queued` or `reservation.awaiting-confirmation`.

* **Kind:** mutation
* **Authorization:** `compute:reservations:create` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field                    | Type                                                          | Required | Constraints                                                       |
| ------------------------ | ------------------------------------------------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`               | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`                 | string                                                        | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `projectId`              | string                                                        |          | at most 56 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `devices`                | list of object                                                | yes      | 1-8 items                                                         |
| `devices[].class`        | one of `gpu`, `gpu.mig`, `gpu.slot`, `nic.vf`, `disk`, `node` | yes      |                                                                   |
| `devices[].count`        | integer                                                       | yes      | 1-64                                                              |
| `devices[].selector`     | object                                                        |          |                                                                   |
| `devices[].topology`     | one of `any`, `nvlink-all`                                    |          |                                                                   |
| `devices[].sharing`      | one of `exclusive`, `mig`, `timeslice`, `mps`                 |          |                                                                   |
| `topology`               | one of `any`, `nvlink-all`                                    |          |                                                                   |
| `start`                  | integer                                                       |          | 0-140737488355328                                                 |
| `duration`               | string,integer                                                | yes      |                                                                   |
| `binding`                | string                                                        |          | default `"hold"`                                                  |
| `resources`              | object                                                        |          |                                                                   |
| `resources.vcpu`         | integer                                                       |          | 0-4096                                                            |
| `resources.memoryGiB`    | integer                                                       |          | 0-65536                                                           |
| `resources.hugepages1Gi` | integer                                                       |          | 0-4096                                                            |
| `node`                   | object                                                        |          |                                                                   |
| `node.profileId`         | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:/-]{1,128}$`       |
| `node.profileVersion`    | integer                                                       |          | at least 1                                                        |
| `leaseSeconds`           | integer                                                       |          | 30-86400                                                          |
| `spot`                   | boolean                                                       |          |                                                                   |
| `idempotencyKey`         | string                                                        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `wait`                   | object                                                        |          |                                                                   |
| `wait.timeoutMs`         | integer                                                       | yes      | 0-30000                                                           |
| `gang`                   | object                                                        |          |                                                                   |
| `gang.size`              | integer                                                       | yes      | 2-64                                                              |
| `gang.topologyKey`       | string                                                        |          | at most 128 characters                                            |

## get [#get]

`POST /api/compute/reservations/get`

Returns one reservation of the tenant with its `operation` and, once a node reported its devices bound, its `binding`.

Drafts are internal: the call is `NOT_FOUND` for them, as for a reservation that does not exist in the tenant.

* **Kind:** read
* **Authorization:** `compute:reservations:read` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                      |
| --------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `reservationId` | string | yes      | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$` |

## list [#list]

`POST /api/compute/reservations/list`

Returns the tenant's reservations newest first, skipping drafts, filtered by `state` and `poolId`, one page at a time.

With `scope: 'mine'` (the default) only the caller's own reservations are listed. With `scope: 'all'` the tenant's reservations are loaded and each one is checked as `compute:reservations:read` on the reservation, so only the allowed ones are returned. `limit` is 1 to 200, default 50; `cursor` is the id of the last reservation of the previous page, and the answer carries the next `cursor` while more remain.

* **Kind:** read
* **Authorization:** `compute:reservations:read` @ `compute-tenant` + items
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type                                                                                                                                                   | Required | Constraints                                                       |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string                                                                                                                                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `scope`    | one of `mine`, `all`                                                                                                                                   |          | default `"mine"`                                                  |
| `state`    | one of `draft`, `denied`, `awaiting-confirmation`, `pending`, `placed`, `binding`, `active`, `releasing`, `released`, `failed`, `cancelled`, `expired` |          |                                                                   |
| `poolId`   | string                                                                                                                                                 |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `limit`    | integer                                                                                                                                                |          | 1-200                                                             |
| `cursor`   | string                                                                                                                                                 |          | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## release [#release]

`POST /api/compute/reservations/release`

Ends a reservation now: its bookings are released at `compute_now()`, its quota is refunded, and it becomes `released`; returns the reservation.

A `pending` reservation is `cancelled` instead. A reservation already in a terminal state is returned unchanged. A bound `binding: 'kubernetes'` reservation is `released` at once, and its node withdraws the devices from the kubelet. Its devices and host resources stay booked, and billed, until the node reports that no pod holds them (`AllocationReleased`) or the node is lost, and the reservation carries the `DevicesHeld` condition meanwhile. Until then nothing else can book them. If pods still hold them at the booking's end, the booking is carried on in 5-minute steps while the devices are free, unbilled, for at most 24 hours; the node is `KubernetesReleaseStuck` (NoSchedule) meanwhile. A booking made for those devices in advance takes them at its start. A request still in `draft` or `awaiting-confirmation` is `INVALID_STATE`. Freed capacity wakes the pool's pending reservations. Emits `reservation.released` or `reservation.cancelled`, with `reason: 'release'`.

* **Kind:** mutation
* **Authorization:** `compute:reservations:release` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                      |
| --------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `reservationId` | string | yes      | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$` |

## cancel [#cancel]

`POST /api/compute/reservations/cancel`

Withdraws a reservation that has not started (`pending` or `placed`): its bookings are released, its quota is refunded, and it becomes `cancelled`; returns the reservation.

A reservation already `cancelled` is returned unchanged. Any other state is `INVALID_STATE`; use `release` for a reservation that has started. Drafts are `NOT_FOUND`. It is decided as `compute:reservations:release`. Emits `reservation.cancelled`.

* **Kind:** mutation
* **Authorization:** `compute:reservations:release` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.2 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                      |
| --------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `reservationId` | string | yes      | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$` |

## extend [#extend]

`POST /api/compute/reservations/extend`

Moves the end of a `placed`, `binding` or `active` reservation later, all or nothing, after one IAM decision; returns the reservation and whether it was extended.

* Give exactly one of `until` (a new end, epoch milliseconds) or `by` (a duration); both or neither is `INVALID_INPUT`. Other states are `INVALID_STATE`.
* It is admitted like `create`: the pending change goes into the reservation's authz row with `durationHours` set to the total after the change, so a policy that caps duration also caps extensions. A refusal is `ACCESS_DENIED` and clears the pending change. `SPEND_LIMIT_REACHED` applies as for `create`.
* When another booking starts before the new end, nothing moves and the answer is `extended: false` with `availableUntil`, the latest end the bookings can reach.
* **Idempotency.** An `until` the reservation already reaches answers `extended: false` without a decision, so a retried `until` is safe. `idempotencyKey` does not replay: a repeated `by` extends again from the new end.
* While another extend of the same reservation is being decided, the call is `CONFLICT` with `details.reason: 'extend-in-flight'` and `retryAfterSeconds`.

Emits `reservation.extended` with the new `end`.

* **Kind:** mutation
* **Authorization:** `compute:reservations:extend` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field            | Type           | Required | Constraints                                                      |
| ---------------- | -------------- | -------- | ---------------------------------------------------------------- |
| `tenantId`       | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `reservationId`  | string         | yes      | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `until`          | integer        |          | 0-140737488355328                                                |
| `by`             | string,integer |          |                                                                  |
| `idempotencyKey` | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |

## renew [#renew]

`POST /api/compute/reservations/renew`

Moves the lease deadline of a renewable reservation to now plus its `leaseSeconds`, without changing its window; returns the reservation.

A reservation created without `leaseSeconds` is `INVALID_STATE`, as is one not in `placed`, `binding` or `active`. A lease that lapses without renewal releases the reservation with reason `lease-lapsed`. It is decided as `compute:reservations:extend`. Renewing emits no event.

* **Kind:** mutation
* **Authorization:** `compute:reservations:extend` @ `compute-reservation`
* **Module:** `capacity`
* **Phase:** v0.1 (registered)

**Input**

| Field           | Type   | Required | Constraints                                                      |
| --------------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`      | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `reservationId` | string | yes      | at most 56 characters; matches `^rsv_[0-9a-hjkmnp-tv-z]{26,52}$` |


# rollouts (/docs/reference/api/rollouts)

> Rollouts take a registered bcd release to the nodes of the tenant's pools in waves, within each pool's disruption budget. A controller settles nodes as they report the new version, pauses the rollout when a wave's failure ratio exceeds its limit, and soaks between waves. Every method decides `compute:fleet:admin` on the fleet scope. See [Rollouts](/docs/guides/rollouts).



Rollouts take a registered bcd release to the nodes of the tenant's pools in waves, within each pool's disruption budget. A controller settles nodes as they report the new version, pauses the rollout when a wave's failure ratio exceeds its limit, and soaks between waves. Every method decides `compute:fleet:admin` on the fleet scope. See [Rollouts](/docs/guides/rollouts).

All 6 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                           | Phase | Status     |
| ------------------- | -------- | --------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`list`](#list)     | read     | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`get`](#get)       | read     | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`pause`](#pause)   | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`resume`](#resume) | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |
| [`abort`](#abort)   | mutation | `compute:fleet:admin` @ `compute-fleet` | v0.4  | registered |

## create [#create]

`POST /api/compute/rollouts/create`

Plans a rollout of a registered release to every node that runs bcd in the tenant's pools, or in `poolIds`, and starts it; returns the rollout in `running`.

* **Plan.** Nodes are shuffled by a hash of the rollout id and assigned to `waves`, cumulative percentages that must increase and end at 100 (default 1, 10, 50, 100). The other gates default to `maxFailureRatio` 0.05, `soakSeconds` 600 and `upgradeTimeoutSeconds` 900. Nodes that already report the version start as `succeeded`.

* **Idempotency.** `idempotencyKey` is optional. With a key, the rollout id derives from the tenant, the caller and the key: a repeat for the same version returns the existing rollout, and another version under the key is `CONFLICT`.

* **Errors.** `NOT_FOUND` when the version is not registered or a listed pool is not an unarchived pool of the tenant. `INVALID_STATE` when no node in scope runs bcd yet. `CONFLICT` with `details.rolloutId` when a node in scope is in another running or paused rollout. `INVALID_INPUT` for gates out of range.

* **Events.** `rollout.started`; the controller later publishes `rollout.wave`, `rollout.paused` and `rollout.completed`.

* **Kind:** mutation

* **Authorization:** `compute:fleet:admin` @ `compute-fleet`

* **Module:** `rollout`

* **Phase:** v0.4 (registered)

**Input**

| Field                   | Type           | Required | Constraints                                                    |
| ----------------------- | -------------- | -------- | -------------------------------------------------------------- |
| `tenantId`              | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`     |
| `version`               | string         | yes      | matches `^v?\d+\.\d+\.\d+([-+][0-9A-Za-z.-]+)?$`               |
| `poolIds`               | list of string |          | 1-1000 items                                                   |
| `waves`                 | list of number |          | 1-10 items; default `[1,10,50,100]`                            |
| `maxFailureRatio`       | number         |          | 0-1; default `0.05`                                            |
| `soakSeconds`           | number         |          | 0-604800; default `600`                                        |
| `upgradeTimeoutSeconds` | number         |          | 60-86400; default `900`                                        |
| `idempotencyKey`        | string         |          | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$` |

## list [#list]

`POST /api/compute/rollouts/list`

Returns the tenant's 100 most recent rollouts, newest first, each with its node counts by state.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `rollout`
* **Phase:** v0.4 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## get [#get]

`POST /api/compute/rollouts/get`

Returns one rollout of the tenant with `nodeList`, each node's wave, state, previous version, timings and error, in rollout order.

A rollout of another tenant, or none, is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `rollout`
* **Phase:** v0.4 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                          |
| ----------- | ------ | -------- | -------------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `rolloutId` | string | yes      | at most 60 characters; matches `^rollout_[0-9a-hjkmnp-tv-z]{26,52}$` |

## pause [#pause]

`POST /api/compute/rollouts/pause`

Stops a running or paused rollout from starting more nodes and records `reason` (default "paused by an operator"); returns the rollout in `paused`.

Nodes already upgrading still settle. A completed or aborted rollout is `INVALID_STATE`. Publishes `rollout.paused`.

* **Kind:** mutation
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `rollout`
* **Phase:** v0.4 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                          |
| ----------- | ------ | -------- | -------------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `rolloutId` | string | yes      | at most 60 characters; matches `^rollout_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reason`    | string |          | at most 500 characters                                               |

## resume [#resume]

`POST /api/compute/rollouts/resume`

Sets a running or paused rollout back to `running`, clears its reason and wakes the controller; returns the rollout.

A rollout paused by the failure gate resumes the same way. A completed or aborted rollout is `INVALID_STATE`. Publishes `rollout.resumed`.

* **Kind:** mutation
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `rollout`
* **Phase:** v0.4 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                          |
| ----------- | ------ | -------- | -------------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `rolloutId` | string | yes      | at most 60 characters; matches `^rollout_[0-9a-hjkmnp-tv-z]{26,52}$` |

## abort [#abort]

`POST /api/compute/rollouts/abort`

Ends a running or paused rollout: nodes not yet started are skipped, and with `rollback: true` upgrades not yet confirmed go back to the version they ran before; returns the rollout in `aborted`.

* With `rollback`, nodes still `upgrading` whose previous version is a registered release move to `rolling-back`, and the controller settles them. Nodes that `succeeded` keep the release: once a node confirmed it, bcd refuses every older binary, the one in its other slot included. Undo a bad release by publishing a newer one (a higher sequence) and rolling it out. Without `rollback`, nodes still upgrading are marked `failed`.

* `reason` defaults to "aborted by an operator". A completed or aborted rollout is `INVALID_STATE`. Publishes `rollout.aborted`.

* **Kind:** mutation

* **Authorization:** `compute:fleet:admin` @ `compute-fleet`

* **Module:** `rollout`

* **Phase:** v0.4 (registered)

**Input**

| Field       | Type    | Required | Constraints                                                          |
| ----------- | ------- | -------- | -------------------------------------------------------------------- |
| `tenantId`  | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `rolloutId` | string  | yes      | at most 60 characters; matches `^rollout_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `reason`    | string  |          | at most 500 characters                                               |
| `rollback`  | boolean |          | default `false`                                                      |


# sandboxSnapshots (/docs/reference/api/sandbox-snapshots)

> Sandbox snapshots are their own resource, so a snapshot kept with `keep: true` outlives the sandbox it came from. `sandboxes.snapshot` takes them and `sandboxes.create` restores from them with `fromSnapshot`. These methods list, read and delete them. Snapshots stay private to their tenant and are never promoted to shared images. See [Sandboxes](/docs/guides/vps/sandboxes).



Sandbox snapshots are their own resource, so a snapshot kept with `keep: true` outlives the sandbox it came from. `sandboxes.snapshot` takes them and `sandboxes.create` restores from them with `fromSnapshot`. These methods list, read and delete them. Snapshots stay private to their tenant and are never promoted to shared images. See [Sandboxes](/docs/guides/vps/sandboxes).

All 3 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                           | Phase | Status     |
| ------------------- | -------- | ------------------------------------------------------- | ----- | ---------- |
| [`list`](#list)     | read     | `compute:sandboxes:read` @ `compute-tenant` + items     | v0.5  | registered |
| [`get`](#get)       | read     | `compute:sandboxes:read` @ `compute-sandbox-snapshot`   | v0.5  | registered |
| [`delete`](#delete) | mutation | `compute:sandboxes:delete` @ `compute-sandbox-snapshot` | v0.5  | registered |

## list [#list]

`POST /api/compute/sandboxSnapshots/list`

Returns a page of the tenant's snapshots, newest first, including kept snapshots of deleted sandboxes.

`scope` defaults to `mine`, the caller's own snapshots; `all` lists the tenant's. `sandboxId` keeps one sandbox's snapshots. `limit` is 1 to 200 (default 50), and `cursor`, the last snapshot id of the previous page, is set while more remain. Deleted snapshots are never listed. Decided as `compute:sandboxes:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field       | Type                 | Required | Constraints                                                       |
| ----------- | -------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`  | string               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `scope`     | one of `mine`, `all` |          | default `"mine"`                                                  |
| `sandboxId` | string               |          | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$`  |
| `limit`     | integer              |          | 1-200                                                             |
| `cursor`    | string               |          | at most 57 characters; matches `^snap_[0-9a-hjkmnp-tv-z]{26,52}$` |

## get [#get]

`POST /api/compute/sandboxSnapshots/get`

Returns one snapshot of the tenant with its state (`pending`, `ready` or `failed`), and once ready its sha256, size, runtime and CPU model.

A deleted snapshot, or one of another tenant, is `NOT_FOUND`. Decided as `compute:sandboxes:read` on the snapshot.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-sandbox-snapshot`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `snapshotId` | string | yes      | at most 57 characters; matches `^snap_[0-9a-hjkmnp-tv-z]{26,52}$` |

## delete [#delete]

`POST /api/compute/sandboxSnapshots/delete`

Deletes a snapshot at once for reads and restores; the stored object is removed by the next `vps.sandboxes` pass (after its upload settled, for a pending one), so no network call runs in the request.

Idempotent: deleting a deleted snapshot answers `{ deleted: true }` again; an unknown one is `NOT_FOUND`. Publishes `sandbox.snapshot.deleted`. Decided as `compute:sandboxes:delete` on the snapshot.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:delete` @ `compute-sandbox-snapshot`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `snapshotId` | string | yes      | at most 57 characters; matches `^snap_[0-9a-hjkmnp-tv-z]{26,52}$` |


# sandboxTemplates (/docs/reference/api/sandbox-templates)

> A sandbox template belongs to one tenant and fixes what its sandboxes run: a hosted pool, a flavor, an image, the pool's runtime, optional user data, TTL bounds and a warm pool. For each live template the `vps.sandboxes` reconciler keeps `warm.min` instances with `desiredState` `paused`, or `stopped` when the runtime cannot pause, creating at most 4 per template in one pass and deleting those beyond `warm.max`, newest first. Warm instances are owned by the template's author, placed and charged to quota when they are created, and a refill that fails is logged and tried again on a later pass. See [Sandboxes](/docs/guides/vps/sandboxes).



A sandbox template belongs to one tenant and fixes what its sandboxes run: a hosted pool, a flavor, an image, the pool's runtime, optional user data, TTL bounds and a warm pool. For each live template the `vps.sandboxes` reconciler keeps `warm.min` instances with `desiredState` `paused`, or `stopped` when the runtime cannot pause, creating at most 4 per template in one pass and deleting those beyond `warm.max`, newest first. Warm instances are owned by the template's author, placed and charged to quota when they are created, and a refill that fails is logged and tried again on a later pass. See [Sandboxes](/docs/guides/vps/sandboxes).

All 5 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                           | Phase | Status     |
| ------------------- | -------- | ------------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:sandboxes:manage` @ `compute-tenant`           | v0.5  | registered |
| [`list`](#list)     | read     | `compute:sandboxes:read` @ `compute-tenant` + items     | v0.5  | registered |
| [`get`](#get)       | read     | `compute:sandboxes:read` @ `compute-sandbox-template`   | v0.5  | registered |
| [`update`](#update) | mutation | `compute:sandboxes:manage` @ `compute-sandbox-template` | v0.5  | registered |
| [`delete`](#delete) | mutation | `compute:sandboxes:manage` @ `compute-sandbox-template` | v0.5  | registered |

## create [#create]

`POST /api/compute/sandboxTemplates/create`

Checks the pool, flavor, image and runtime once and stores a template under a name unique in the tenant; returns the template.

* **Checks.** The pool must be owned by or offered to the tenant and not archived (`NOT_FOUND`), and be a hosted `vps-host` pool: a provider pool is `INVALID_INPUT`. A missing or archived flavor and an image that does not resolve are `NOT_FOUND`; `INVALID_CONFIG` when `vps()` has no `images`. The pool's runtime must boot the image's format and pass through the flavor's GPUs (`INVALID_INPUT`), and `runtime`, when given, must equal the pool's runtime (`INVALID_INPUT`). The template stores the pool's runtime.
* **Lifetime.** `defaultTtlSeconds` defaults to 3,600 and `maxTtlSeconds` to 86,400 or the default TTL, whichever is larger; both lie between 60 seconds and 30 days, and a default above the maximum is `INVALID_INPUT`.
* **Warm pool.** `warm.min` and `warm.max` are 0 to 64; `max` defaults to `min`, and a `max` below `min` is `INVALID_INPUT`.
* **User data.** At most 32 KiB. A URI must be a `secret://` reference to a secret of the same tenant, otherwise `INVALID_INPUT`.
* **Volumes.** `volumes` lists up to 8 sandbox volumes (`sandboxVolumes`) that each sandbox of the template attaches:

  * each entry names a volume by `volumeId` or by `name` (one of them), with a `mountPath` and an optional `readOnly`;
  * each volume must be the tenant's, `ready` and in the template's pool (`NOT_FOUND`, `INVALID_STATE`, `INVALID_INPUT`), and named once;
  * a mount path is absolute, without `.` or `..`, never over a system directory (`/`, `/etc`, `/usr`, `/proc`, …), and never inside another entry's (`INVALID_INPUT`);
  * a template with volumes keeps no warm pool (`warm.max` must be 0), since a warm instance would hold the volumes;
  * every sandbox of the template reads the volume, and unless it is `readOnly` writes it, so each volume is also decided on the caller's own credential: `compute:sandboxes:manage` on `compute-sandbox-volume` for a read-write mount, `compute:sandboxes:read` for a read-only one (`ACCESS_DENIED` otherwise).

  Its sandboxes run on the volumes' host only. A read-write attachment needs the volume to itself, so a second sandbox of a template that attaches a volume read-write is `CONFLICT` (`details.reason: 'volume-in-use'`) while the first lives.

A live template with the same name is `CONFLICT`. The caller becomes the template's owner in the authorization projection. Publishes no event.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-tenant`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field                 | Type                               | Required | Constraints                                                             |
| --------------------- | ---------------------------------- | -------- | ----------------------------------------------------------------------- |
| `tenantId`            | string                             | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `name`                | string                             | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`            |
| `poolId`              | string                             | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$`       |
| `flavor`              | string                             | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`            |
| `imageId`             | string                             | yes      | at most 200 characters; matches `^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,199}$` |
| `runtime`             | string                             |          | at most 64 characters; matches `^[a-z][a-z0-9-]{0,63}$`                 |
| `userData`            | string                             |          | at most 32768 characters                                                |
| `defaultTtlSeconds`   | integer                            |          | 60-2592000; default `3600`                                              |
| `maxTtlSeconds`       | integer                            |          | 60-2592000; default `86400`                                             |
| `warm`                | object                             |          |                                                                         |
| `warm.min`            | integer                            | yes      | 0-64                                                                    |
| `warm.max`            | integer                            |          | 0-64                                                                    |
| `env`                 | map of string                      |          | at most 64 entries; keys match `^[A-Z_][A-Z0-9_]{0,63}$`                |
| `network`             | one of `egress`, `private`, `none` |          |                                                                         |
| `volumes`             | list of object                     |          | 0-8 items                                                               |
| `volumes[].volumeId`  | string                             |          | at most 56 characters; matches `^vol_[0-9a-hjkmnp-tv-z]{26,52}$`        |
| `volumes[].name`      | string                             |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`            |
| `volumes[].mountPath` | string                             | yes      | at most 255 characters; matches `^/[A-Za-z0-9._\-/]{1,254}$`            |
| `volumes[].readOnly`  | boolean                            |          | default `false`                                                         |

## list [#list]

`POST /api/compute/sandboxTemplates/list`

Returns every live template of the tenant, ordered by name, in one answer.

Each template is checked for `compute:sandboxes:read` on `compute-sandbox-template`; those the caller may not read are left out. Deleted templates are never listed.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                |
| ---------- | ------ | -------- | ---------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |

## get [#get]

`POST /api/compute/sandboxTemplates/get`

Returns one live template of the tenant.

A deleted template, or one of another tenant, is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-sandbox-template`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `templateId` | string | yes      | at most 57 characters; matches `^sbxt_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/sandboxTemplates/update`

Changes a template's flavor, image, user data, TTL bounds, warm pool or volumes; returns the template with its version raised.

The pool, name and runtime stay. A new `flavor` or `imageId` is checked against the pool as on `create`, and the pool must still run the template's runtime (`INVALID_INPUT`). Omitted fields keep their values, `userData: null` removes the user data, and `warm` replaces the warm pool, `max` again defaulting to `min`. A `defaultTtlSeconds` above `maxTtlSeconds` after the change is `INVALID_INPUT`. A deleted or missing template is `NOT_FOUND`. The secrets the template references are decided with the update; if another change added secret references while this one was decided, it is `CONFLICT` with `details.reason: 'secrets-changed'`: retry, and the new references are decided too. The volumes are decided the same way, each on its own resource as on `create`: a new `volumes` list, and the current one when the update changes what the guest runs (flavor, image, user data, env or network). If a volume name now points at another volume, or a mount became writable, since the decision, it is `CONFLICT` with `details.reason: 'volumes-changed'`: retry.

Sandboxes that exist keep their instances, and warm instances already created stay claimable with the shape they were created with; new sandboxes and warm refills use the new values. A lower `warm.max` makes the reconciler delete the excess. Publishes no event.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-sandbox-template`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field                 | Type                               | Required | Constraints                                                             |
| --------------------- | ---------------------------------- | -------- | ----------------------------------------------------------------------- |
| `tenantId`            | string                             | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`              |
| `templateId`          | string                             | yes      | at most 57 characters; matches `^sbxt_[0-9a-hjkmnp-tv-z]{26,52}$`       |
| `flavor`              | string                             |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`            |
| `imageId`             | string                             |          | at most 200 characters; matches `^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,199}$` |
| `userData`            | string,null                        |          | at most 32768 characters                                                |
| `defaultTtlSeconds`   | integer                            |          | 60-2592000                                                              |
| `maxTtlSeconds`       | integer                            |          | 60-2592000                                                              |
| `warm`                | object                             |          |                                                                         |
| `warm.min`            | integer                            | yes      | 0-64                                                                    |
| `warm.max`            | integer                            |          | 0-64                                                                    |
| `env`                 | map of string                      |          | at most 64 entries; keys match `^[A-Z_][A-Z0-9_]{0,63}$`                |
| `network`             | one of `egress`, `private`, `none` |          |                                                                         |
| `volumes`             | list of object                     |          | 0-8 items                                                               |
| `volumes[].volumeId`  | string                             |          | at most 56 characters; matches `^vol_[0-9a-hjkmnp-tv-z]{26,52}$`        |
| `volumes[].name`      | string                             |          | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`            |
| `volumes[].mountPath` | string                             | yes      | at most 255 characters; matches `^/[A-Za-z0-9._\-/]{1,254}$`            |
| `volumes[].readOnly`  | boolean                            |          | default `false`                                                         |

## delete [#delete]

`POST /api/compute/sandboxTemplates/delete`

Deletes a template so that `get`, `list`, `update` and `sandboxes.create` no longer find it; returns `deleted: true`.

The template leaves the authorization projection and its name is free for a new template. Its sandboxes live on until they are deleted or expire, and the `vps.sandboxes` reconciler deletes all of its warm instances. A template already deleted is `NOT_FOUND`. Publishes no event.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-sandbox-template`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `templateId` | string | yes      | at most 57 characters; matches `^sbxt_[0-9a-hjkmnp-tv-z]{26,52}$` |


# sandboxVolumes (/docs/reference/api/sandbox-volumes)

> Sandbox volumes are persistent disks on one host that a tenant's sandboxes attach across runs, so an agent keeps its memory, caches and checkouts. A volume is placed when it is created: its size is booked as `diskGiB` on one host of the pool, charged to the tenant's quota, and the host keeps it as a sparse raw file. A sandbox template names volumes with a mount path. Sandboxes of that template then run on the volume's host only, attach it as an extra virtio disk, and mount it by serial (`/dev/disk/by-id/virtio-<serial>`) from their user data. The first mount formats it as ext4, and no later one ever does. A volume takes any number of read-only attachments or one read-write attachment, never both. Volumes are metered as `vps.volume.gib.hours` from creation until the disk is released. See [Sandboxes](/docs/guides/vps/sandboxes).



Sandbox volumes are persistent disks on one host that a tenant's sandboxes attach across runs, so an agent keeps its memory, caches and checkouts. A volume is placed when it is created: its size is booked as `diskGiB` on one host of the pool, charged to the tenant's quota, and the host keeps it as a sparse raw file. A sandbox template names volumes with a mount path. Sandboxes of that template then run on the volume's host only, attach it as an extra virtio disk, and mount it by serial (`/dev/disk/by-id/virtio-<serial>`) from their user data. The first mount formats it as ext4, and no later one ever does. A volume takes any number of read-only attachments or one read-write attachment, never both. Volumes are metered as `vps.volume.gib.hours` from creation until the disk is released. See [Sandboxes](/docs/guides/vps/sandboxes).

All 5 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                         | Phase | Status     |
| ------------------- | -------- | ----------------------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:sandboxes:manage` @ `compute-sandbox-volume` | v1.0  | registered |
| [`list`](#list)     | read     | `compute:sandboxes:read` @ `compute-tenant`           | v1.0  | registered |
| [`get`](#get)       | read     | `compute:sandboxes:read` @ `compute-sandbox-volume`   | v1.0  | registered |
| [`resize`](#resize) | mutation | `compute:sandboxes:manage` @ `compute-sandbox-volume` | v1.0  | registered |
| [`delete`](#delete) | mutation | `compute:sandboxes:manage` @ `compute-sandbox-volume` | v1.0  | registered |

## create [#create]

`POST /api/compute/sandboxVolumes/create`

Books the volume's disk on one host of a hosted pool and puts it in that host's spec; returns the volume `ready`, with its `serial` and `nodeId`.

* `sizeGiB` runs from 1 to 16,384. The name is unique among the tenant's volumes that are not deleted (`CONFLICT` otherwise).
* `CAPACITY_UNAVAILABLE` when no host of the pool has the disk free (the reason is in the details), and `QUOTA_EXCEEDED` past the tenant's `diskGiB` quota.
* Volumes are sparse files in the host's data directory. A volume is booked only on a host whose volume store, as its bcd last reported it, can really hold it: available − reserved − outstanding (the unwritten space of its volumes), less what is booked there that bcd has not reported yet. It is never overcommitted, and a host that has not reported its store takes no volume. Otherwise it is `CAPACITY_UNAVAILABLE` with `details.reason: 'volume-storage'`.
* A provider pool is `INVALID_INPUT`: volumes live on hosted (`vps-host`) pools.
* Decided as `compute:sandboxes:manage` on the volume it would create (`compute-sandbox-volume`, under the pool), whose `sizeGiB` attribute is the requested size, so a policy can bound it. A refusal leaves nothing behind.

The volume id derives from the tenant, the caller and `idempotencyKey`. A retry with the same key returns the same volume; the same key with another pool, name or size is `CONFLICT`. Publishes `sandbox.volume.created` with the host.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-sandbox-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `poolId`         | string  | yes      | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `name`           | string  | yes      | at most 63 characters; matches `^[a-z0-9][a-z0-9._-]{0,62}$`      |
| `sizeGiB`        | integer | yes      | 1-16384                                                           |
| `idempotencyKey` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |

## list [#list]

`POST /api/compute/sandboxVolumes/list`

Returns the caller's own volumes, oldest first (`scope: 'mine'`, the default, as for snapshots), or with `scope: 'all'` every volume of the tenant. `poolId` narrows it to one pool, and `includeDeleted: true` adds deleted ones.

At most 500 volumes are returned. Decided as `compute:sandboxes:read` on the tenant.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-tenant`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type                 | Required | Constraints                                                       |
| ---------------- | -------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `scope`          | one of `mine`, `all` |          | default `"mine"`                                                  |
| `poolId`         | string               |          | at most 57 characters; matches `^pool_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `includeDeleted` | boolean              |          | default `false`                                                   |

## get [#get]

`POST /api/compute/sandboxVolumes/get`

Returns one volume of the tenant: its size, serial, host and state (`ready`, `deleting` or `deleted`).

A volume of another tenant is `NOT_FOUND`. Decided as `compute:sandboxes:read` on the volume.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-sandbox-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `volumeId` | string | yes      | at most 56 characters; matches `^vol_[0-9a-hjkmnp-tv-z]{26,52}$` |

## resize [#resize]

`POST /api/compute/sandboxVolumes/resize`

Grows a volume to `sizeGiB` on the host it lives on and returns it with the new size. A volume never shrinks.

* A size below the current one is `INVALID_INPUT`. The current size returns the volume unchanged.
* The growth must fit the host's volume store as on `create` (`CAPACITY_UNAVAILABLE`, `details.reason: 'volume-storage'` with `freeGiB`). It is then charged to the tenant's `diskGiB` quota (`QUOTA_EXCEEDED` past it), then booked on the volume's host. When the host has no room it is `CAPACITY_UNAVAILABLE`: the quota charge is refunded and nothing changes.
* Only a `ready` volume grows (`INVALID_STATE` otherwise).

The host grows the file as soon as its spec carries the new size, even while a running sandbox has the volume attached: the guest's disk grows live. The filesystem on it grows when the next sandbox that attaches the volume boots, because its user data runs `resize2fs` on every writable volume. Inside a running guest, run `resize2fs /dev/disk/by-id/virtio-<serial>` to use the space at once. Publishes `sandbox.volume.resized`. Decided as `compute:sandboxes:manage` on the volume, with the new size as its pending `sizeGiB` (as `sandboxes.extend` decides a lifetime), so a policy bounds the size it grants; afterwards the volume's `sizeGiB` is the new size. One resize of a volume is decided at a time: another one meanwhile is `CONFLICT` (`details.reason: 'resize-in-flight'`), and a stalled one is taken over after 60 s.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-sandbox-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                      |
| ---------- | ------- | -------- | ---------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `volumeId` | string  | yes      | at most 56 characters; matches `^vol_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `sizeGiB`  | integer | yes      | 1-16384                                                          |

## delete [#delete]

`POST /api/compute/sandboxVolumes/delete`

Takes the volume out of its host's spec, and the host discards its file; the volume is `deleting` until the host has applied a spec without it, then `deleted`, and its disk booking and quota come back.

* `CONFLICT` while a sandbox that is not deleted attaches it, or while a template names it: delete those sandboxes, and update or delete the template, first.
* Deleting a volume that is already `deleting` or `deleted` returns it unchanged.

Publishes `sandbox.volume.deleting`, then `sandbox.volume.deleted`. Decided as `compute:sandboxes:manage` on the volume.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:manage` @ `compute-sandbox-volume`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                      |
| ---------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `volumeId` | string | yes      | at most 56 characters; matches `^vol_[0-9a-hjkmnp-tv-z]{26,52}$` |


# sandboxes (/docs/reference/api/sandboxes)

> A sandbox is a hosted VPS instance with a template and a lifetime: `create` admits it after one IAM decision, then claims a warm instance of its template or creates one through the instance lifecycle, and the `vps.sandboxes` reconciler, which runs every 15 seconds, deletes it once `expiresAt` passes. A sandbox's `state` follows its instance: `starting`, `ready` while the instance runs, `failed` when it is in `error` or `lost`, `deleting` once deletion was asked, and `deleted` when the instance is gone. The instance belongs to the sandbox: `instances.start`, `stop`, `resize` and `delete` refuse it with `INVALID_STATE`, and `instances.get` still returns it. See [Sandboxes](/docs/guides/vps/sandboxes).



A sandbox is a hosted VPS instance with a template and a lifetime: `create` admits it after one IAM decision, then claims a warm instance of its template or creates one through the instance lifecycle, and the `vps.sandboxes` reconciler, which runs every 15 seconds, deletes it once `expiresAt` passes. A sandbox's `state` follows its instance: `starting`, `ready` while the instance runs, `failed` when it is in `error` or `lost`, `deleting` once deletion was asked, and `deleted` when the instance is gone. The instance belongs to the sandbox: `instances.start`, `stop`, `resize` and `delete` refuse it with `INVALID_STATE`, and `instances.get` still returns it. See [Sandboxes](/docs/guides/vps/sandboxes).

All 8 methods are served by @better-compute/server 0.3.0.

| Method                  | Kind     | Authorization                                       | Phase | Status     |
| ----------------------- | -------- | --------------------------------------------------- | ----- | ---------- |
| [`create`](#create)     | mutation | `compute:sandboxes:create` @ `compute-sandbox`      | v0.5  | registered |
| [`list`](#list)         | read     | `compute:sandboxes:read` @ `compute-tenant` + items | v0.5  | registered |
| [`get`](#get)           | read     | `compute:sandboxes:read` @ `compute-sandbox`        | v0.5  | registered |
| [`extend`](#extend)     | mutation | `compute:sandboxes:update` @ `compute-sandbox`      | v0.5  | registered |
| [`snapshot`](#snapshot) | mutation | `compute:sandboxes:update` @ `compute-sandbox`      | v0.5  | registered |
| [`exec`](#exec)         | mutation | `compute:sandboxes:exec` @ `compute-sandbox`        | v0.5  | registered |
| [`delete`](#delete)     | mutation | `compute:sandboxes:delete` @ `compute-sandbox`      | v0.5  | registered |
| [`update`](#update)     | mutation | `compute:sandboxes:update` @ `compute-sandbox`      | v1.0  | registered |

## create [#create]

`POST /api/compute/sandboxes/create`

Starts a sandbox from a template after one IAM decision on the draft, claiming a warm instance when the template holds one and creating a new instance otherwise; the lifetime starts at once.

* **Idempotency.** `idempotencyKey` is required. The sandbox id derives from the tenant, the caller's identity and the key: a retry of an admitted request returns the stored sandbox without a second decision, and a different request under the same key is `CONFLICT` with `details.sandboxId`, as is the key replayed by another agent or person. `wait` is not part of the compared request.
* **IAM.** `compute:sandboxes:create` is decided on `compute-sandbox/` plus the draft id, whose parent is the template's pool. Policies can test `templateId`, `vcpu`, `memoryGiB`, `gpus`, `ttlSeconds` and `warm` (true when the template keeps a warm pool). A refusal deletes the draft; a delegated session can get `CONFIRMATION_REQUIRED`. After an allow, the pool's budget check can answer `SPEND_LIMIT_REACHED` and keep the draft for a retry with the same key.
* **Start.** The oldest stopped or paused warm instance of the template is claimed: it changes owner to the caller, is set to run, and publishes `instance.starting`; the claim places nothing new. Without one, a new instance is created and placed as `instances.create` would, publishing `instance.created` and, once placed, `instance.scheduled`; when no host fits it waits in scheduling and the sandbox stays `starting`. `warm` in the answer tells which path was taken.
* **Restore.** `fromSnapshot` starts the sandbox from one of the tenant's own `ready` snapshots of this template, taken on its runtime (another tenant's is `NOT_FOUND`, one not ready is `INVALID_STATE`, one of another template is `INVALID_INPUT`). An agent restores only snapshots of sandboxes it created itself; any other is `NOT_FOUND` to it. A restore never claims a warm instance, and a snapshot that includes memory places only on nodes with the same `cpu.model`. Policies see it as the `fromSnapshot` attribute.
* **Admission.** For every caller, in the admission transaction: the template's flavor passes the flavor allowlists of the tenant's offer and of the quota rows it consumes (`QUOTA_EXCEEDED` `flavor-not-offered` or `flavor-not-allowed`), then the host's `vps({ admissionPolicy })` (method `sandboxes.create`), then `sandboxes` is charged, one per sandbox where a row exists on the pool or tenant-wide (`*`). The count rides on the sandbox's instance and comes back when that instance is deleted. A sandbox's instance is not counted as an `instances`.
* **Lifetime.** `ttlSeconds` defaults to the template's `defaultTtlSeconds` and may not exceed its `maxTtlSeconds` (`INVALID_INPUT`). `expiresAt` is set when the sandbox is admitted.
* **Errors.** `NOT_FOUND` for a template missing or deleted in the tenant. `INVALID_STATE` when the template's flavor has been archived or removed. On the new-instance path the template's pool, flavor and image are checked again, with the errors of `sandboxTemplates.create`; `QUOTA_EXCEEDED` when the shape does not fit the tenant's quota and `CAPACITY_UNAVAILABLE` when the pool has no free IP address, both keeping the draft.
* **Waiting.** `wait.timeoutMs` (at most 30 seconds) holds the answer until the sandbox leaves `starting`; a timeout returns it still `starting`, not an error.
* **Identity.** `name`, `metadata` and `labels` take the rules of `instances.create` (`better-compute-` and `bc-` keys reserved) and are part of the request. They stay on the sandbox, never reach its host, and `update` changes them later.

Publishes `sandbox.created` with `warm` and `expiresAt`. `sandbox.ready` follows from the reconciler once the instance first runs.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:create` @ `compute-sandbox`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field            | Type          | Required | Constraints                                                          |
| ---------------- | ------------- | -------- | -------------------------------------------------------------------- |
| `tenantId`       | string        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `templateId`     | string        | yes      | at most 57 characters; matches `^sbxt_[0-9a-hjkmnp-tv-z]{26,52}$`    |
| `ttlSeconds`     | integer       |          | 60-2592000                                                           |
| `fromSnapshot`   | string        |          | at most 57 characters; matches `^snap_[0-9a-hjkmnp-tv-z]{26,52}$`    |
| `name`           | string        |          | at most 128 characters                                               |
| `metadata`       | map of string |          | at most 64 entries; keys match `^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$` |
| `labels`         | map of string |          | at most 62 entries; keys match `^[a-z][a-z0-9_-]{0,62}$`             |
| `idempotencyKey` | string        | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `wait`           | object        |          |                                                                      |
| `wait.timeoutMs` | integer       | yes      | 0-30000                                                              |

## list [#list]

`POST /api/compute/sandboxes/list`

Returns a page of the tenant's sandboxes, newest first, with a `cursor` when more remain.

`scope` defaults to `mine`, the caller's own sandboxes, which needs no per-item check. `all` lists every sandbox of the tenant, each one checked for `compute:sandboxes:read`; those the caller may not read are left out, so a page can hold fewer than `limit` items. Deleted sandboxes are left out unless `deleted` is true, `templateId` keeps one template's sandboxes, and `labelSelector` (as `instances.list` takes it) keeps those whose labels match. `limit` is 1 to 200 (default 50); `cursor` is the last sandbox id of the previous page. Drafts are never listed. Each item carries `identityRebound` as `get` describes.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-tenant` + items
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field           | Type                 | Required | Constraints                                                       |
| --------------- | -------------------- | -------- | ----------------------------------------------------------------- |
| `tenantId`      | string               | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `scope`         | one of `mine`, `all` |          | default `"mine"`                                                  |
| `deleted`       | boolean              |          | default `false`                                                   |
| `templateId`    | string               |          | at most 57 characters; matches `^sbxt_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `labelSelector` | string               |          | at most 1024 characters                                           |
| `limit`         | integer              |          | 1-200                                                             |
| `cursor`        | string               |          | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$`  |

## get [#get]

`POST /api/compute/sandboxes/get`

Returns one sandbox of the tenant with its state, its instance's state, addresses and node, and its `expiresAt`.

`readyAt` is set once the reconciler saw the instance run, and `expiredAt` when the sandbox ended by reaching `expiresAt`. `snapshots` lists the sandbox's snapshots with their state. Drafts and sandboxes of other tenants are `NOT_FOUND`.

`identityRebound` appears only on a memory restore that was given its own guest identity. It is `false` until the host has installed that identity; until then the guest still answers to its source's identity, and `exec` is refused with `restore-shares-identity` while another guest shares it. It is `true` once the identity is installed, from which point `exec` signs for this sandbox alone. It is absent when there is nothing to install: the sandbox is not a memory restore, or exec is not configured.

* **Kind:** read
* **Authorization:** `compute:sandboxes:read` @ `compute-sandbox`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `sandboxId` | string | yes      | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$` |

## extend [#extend]

`POST /api/compute/sandboxes/extend`

Moves a sandbox's `expiresAt` later after one IAM decision, within its template's `maxTtlSeconds` counted from creation; returns the sandbox.

* Give exactly one of `until` (the new expiry, epoch milliseconds) or `by` (added to the current expiry: `30m`, `8h`, `7d`, or milliseconds, at most 30 days); both or neither is `INVALID_INPUT`. A lifetime past the template's `maxTtlSeconds` is `INVALID_INPUT`, and a sandbox whose deletion has started, by `delete` or by expiry, is `INVALID_STATE`.
* **IAM.** `compute:sandboxes:update` is decided on the sandbox with a pending `ttlSeconds` equal to the whole lifetime after the change, so a policy that caps `ttlSeconds` on `create` also caps extensions. A refusal is `ACCESS_DENIED` and clears the pending change.
* **Idempotency.** An `until` that is not later than the current expiry returns the sandbox unchanged without a decision, so a retried `until` is safe. There is no `idempotencyKey`: a repeated `by` extends again.
* While another extend of the same sandbox is being decided, the call is `CONFLICT` with `details.reason: 'extend-in-flight'` and `retryAfterSeconds`. A spent budget is `SPEND_LIMIT_REACHED`.

The stored `ttlSeconds` becomes the new whole lifetime. Publishes `sandbox.extended` with the new `expiresAt`.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:update` @ `compute-sandbox`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field       | Type           | Required | Constraints                                                      |
| ----------- | -------------- | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string         | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `sandboxId` | string         | yes      | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `until`     | integer        |          | 0-140737488355328                                                |
| `by`        | string,integer |          |                                                                  |

## snapshot [#snapshot]

`POST /api/compute/sandboxes/snapshot`

Saves a running or paused sandbox's state (disk, and memory unless `includeMemory` is false) as a tenant-private snapshot; returns it `pending` until the host's `instance.snapshot` command settles.

* **Where.** The deployment's snapshot store (`vps({ snapshots })`) presigns an https upload URL, and the host uploads straight to it. Without a store the call is `INVALID_CONFIG`. A sandbox that is not running or paused, or whose deletion has started, is `INVALID_STATE`.

* **Settling.** The `vps.sandboxes` reconciler, woken when the command settles, makes the snapshot `ready` (with its sha256, size, runtime and CPU model) and publishes `sandbox.snapshot.ready`, or makes it `failed` and publishes `sandbox.snapshot.failed`.

* **Idempotency.** With `idempotencyKey`, a retry returns the same snapshot; the key used for another sandbox is `CONFLICT`. Without one, every call takes a new snapshot.

* **Lifetime.** A snapshot is deleted with its sandbox unless `keep` is true. A tenant teardown deletes all of them, kept ones included. Snapshots are never promoted to shared images.

* **Kind:** mutation

* **Authorization:** `compute:sandboxes:update` @ `compute-sandbox`

* **Module:** `vps`

* **Phase:** v0.5 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                      |
| ---------------- | ------- | -------- | ---------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `sandboxId`      | string  | yes      | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `includeMemory`  | boolean |          | default `true`                                                   |
| `keep`           | boolean |          | default `false`                                                  |
| `idempotencyKey` | string  |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |

## exec [#exec]

`POST /api/compute/sandboxes/exec`

Opens SSH into a running sandbox: returns a single-use `ssh` tunnel ticket for its instance and a short OpenSSH user certificate from the deployment's sandbox SSH CA for the principal `<account>@<instanceId>`.

* **How.** A sandbox has no in-guest agent: the node's daemon dials the instance's own address, and the instance's sshd trusts the sandbox CA and accepts only its own principal, so a certificate for one sandbox opens no other. Open `GET` on the returned `path` with the ticket before `expiresAt`, then speak SSH over it. `better-compute sandbox exec` does both.
* **Host key.** The answer tells whether the guest can be pinned. `hostCertificate` is `valid` when the guest presents a host certificate from the sandbox CA, still good for at least a minute (then pin it with `@cert-authority` for `hostPrincipal` and `HostKeyAlias`). It is `expired` when that certificate ended, and `none` when the instance was never certified. `hostCertified` is true only when `valid`.
* **Certificate.** `publicKey` must be an Ed25519 or ECDSA P-256 key. `ttlSeconds` is 60 to 900 (default 600) and never past the caller's session. Guests receive no revocation list, so the short lifetime is the revocation control, and tunnel re-authorization closes the transport.
* **Errors.** `UNSUPPORTED` when the deployment has no sandbox SSH CA (`tunnels.sandboxSsh`). `ACCESS_DENIED` for an API key (reason `session-required`) or a session ending within a minute. `INVALID_STATE` when the sandbox is not running or is being deleted, and with `details.reason: 'restore-shares-identity'` while another running guest shares its SSH identity (a memory restore of the same snapshot, or the sandbox it was taken from): a certificate for one would open the others, so exec waits until only one of them runs. The node must be connected and advertise `tunnel.instance-dial`. Exec shares the per-identity tunnel rate limit with `tunnels.open` (`RATE_LIMITED`).

Hosted by the tunnels module. Publishes `tunnel.issued`.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:exec` @ `compute-sandbox`
* **Module:** `tunnels`
* **Phase:** v0.5 (registered)

**Input**

| Field        | Type    | Required | Constraints                                                |
| ------------ | ------- | -------- | ---------------------------------------------------------- |
| `tenantId`   | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `sandboxId`  | string  | yes      | at most 56 characters; matches `^sbx_[0-9a-z]{26,52}$`     |
| `publicKey`  | string  | yes      | at most 4096 characters                                    |
| `ttlSeconds` | integer |          | 60-900; default `600`                                      |

## delete [#delete]

`POST /api/compute/sandboxes/delete`

Asks for the sandbox's instance to be deleted and marks the sandbox `deleting`; returns the sandbox at once.

The instance's `desiredState` becomes `deleted` and the `vps.instances` reconciler releases its allocation, IP lease and quota as for any instance, publishing `instance.deleting` with the `sandboxId`. The call is idempotent: a sandbox already deleting is returned in its current state. Drafts and missing sandboxes are `NOT_FOUND`.

When the instance is gone, the `vps.sandboxes` reconciler marks the sandbox deleted, removes it from the authorization projection and publishes `sandbox.deleted` with `expired: false`. Expiry takes the same path after publishing `sandbox.expired`, and its `sandbox.deleted` carries `expired: true`.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:delete` @ `compute-sandbox`
* **Module:** `vps`
* **Phase:** v0.5 (registered)

**Input**

| Field       | Type   | Required | Constraints                                                      |
| ----------- | ------ | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `sandboxId` | string | yes      | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/sandboxes/update`

Renames a sandbox and replaces its metadata and labels; returns the sandbox.

At least one of `name`, `metadata` and `labels`, with the rules of `create`; `null` clears a field and `{}` clears metadata or labels, which are replaced as a whole. Decided as `compute:sandboxes:update` on the sandbox. Nothing reaches the host. An update that changes nothing returns the sandbox as it is, with no event; otherwise `sandbox.updated` is published with `changed`. A sandbox being deleted, or deleted, is `INVALID_STATE`; one of another tenant, or none, is `NOT_FOUND`.

* **Kind:** mutation
* **Authorization:** `compute:sandboxes:update` @ `compute-sandbox`
* **Module:** `vps`
* **Phase:** v1.0 (registered)

**Input**

| Field       | Type        | Required | Constraints                                                          |
| ----------- | ----------- | -------- | -------------------------------------------------------------------- |
| `tenantId`  | string      | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `sandboxId` | string      | yes      | at most 56 characters; matches `^sbx_[0-9a-hjkmnp-tv-z]{26,52}$`     |
| `name`      | string,null |          | at most 128 characters                                               |
| `metadata`  | object,null |          | at most 64 entries; keys match `^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$` |
| `labels`    | object,null |          | at most 62 entries; keys match `^[a-z][a-z0-9_-]{0,62}$`             |


# secrets (/docs/reference/api/secrets)

> Secrets are write-only: `put` hands a value to the configured secret store and returns its `secret://` reference and version, and no method ever reads a value back. Events and audit rows carry only the reference, name and version; values reach agents through the secret broker, never through the API. Methods are decided on `compute-tenant/{tenantId}`, or on `compute-fleet/global` for platform secrets with `scope: 'fleet'` in the root tenant. See [Security](/docs/operations/security) and [BMC credentials](/docs/guides/bare-metal/bmc).



Secrets are write-only: `put` hands a value to the configured secret store and returns its `secret://` reference and version, and no method ever reads a value back. Events and audit rows carry only the reference, name and version; values reach agents through the secret broker, never through the API. Methods are decided on `compute-tenant/{tenantId}`, or on `compute-fleet/global` for platform secrets with `scope: 'fleet'` in the root tenant. See [Security](/docs/operations/security) and [BMC credentials](/docs/guides/bare-metal/bmc).

All 2 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                                                             | Phase | Status     |
| ------------------- | -------- | ----------------------------------------------------------------------------------------- | ----- | ---------- |
| [`put`](#put)       | mutation | `compute:secrets:manage` @ `compute-tenant` or `compute:secrets:manage` @ `compute-fleet` | v0.2  | registered |
| [`delete`](#delete) | mutation | `compute:secrets:manage` @ `compute-tenant` or `compute:secrets:manage` @ `compute-fleet` | v0.2  | registered |

## put [#put]

`POST /api/compute/secrets/put`

Stores a secret value under a name and returns its reference and version; a second put with the same name writes a new version.

The value is at most 64 KiB. A retry whose `idempotencyKey` equals the key of the put that wrote the current version returns that version and writes nothing; the key is not scoped to the calling identity. `scope: 'fleet'` outside the root tenant is `INVALID_INPUT`. The call is `UNAVAILABLE` when no secret store is configured, and `INVALID_STATE` when the existing secret lives in another store than the configured one.

Publishes `security.secret.put` with the reference, name and version, never the value.

* **Kind:** mutation
* **Authorization:** `compute:secrets:manage` @ `compute-tenant` or `compute:secrets:manage` @ `compute-fleet`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type                     | Required | Constraints                                                          |
| ---------------- | ------------------------ | -------- | -------------------------------------------------------------------- |
| `tenantId`       | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `name`           | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$` |
| `value`          | string                   | yes      | 1-65536 characters                                                   |
| `scope`          | one of `tenant`, `fleet` |          | default `"tenant"`                                                   |
| `idempotencyKey` | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`       |

## delete [#delete]

`POST /api/compute/secrets/delete`

Deletes a secret's value from the store and its reference, returning `deleted: true`, or `deleted: false` when no such secret exists.

`scope: 'fleet'` outside the root tenant is `INVALID_INPUT`. The call is `UNAVAILABLE` when no secret store is configured, and `INVALID_STATE` when the secret lives in another store than the configured one. Publishes `security.secret.deleted`.

* **Kind:** mutation
* **Authorization:** `compute:secrets:manage` @ `compute-tenant` or `compute:secrets:manage` @ `compute-fleet`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type                     | Required | Constraints                                                          |
| ---------- | ------------------------ | -------- | -------------------------------------------------------------------- |
| `tenantId` | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`           |
| `name`     | string                   | yes      | at most 128 characters; matches `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$` |
| `scope`    | one of `tenant`, `fleet` |          | default `"tenant"`                                                   |


# siteTokens (/docs/reference/api/site-tokens)

> Site tokens are join tokens with the role `site`: they enroll `bcd site` agents on one site, which then contend for the site's boot lease. They share the join-token bearer format `bcjt_<id>.<secret>` and storage, so the secret is shown once and only its hash is kept. Both methods are decided as `compute:sites:manage` on `compute-site/{siteId}`. See [Sites](/docs/guides/bare-metal/sites).



Site tokens are join tokens with the role `site`: they enroll `bcd site` agents on one site, which then contend for the site's boot lease. They share the join-token bearer format `bcjt_<id>.<secret>` and storage, so the secret is shown once and only its hash is kept. Both methods are decided as `compute:sites:manage` on `compute-site/{siteId}`. See [Sites](/docs/guides/bare-metal/sites).

All 2 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                           | Phase | Status     |
| ------------------- | -------- | --------------------------------------- | ----- | ---------- |
| [`create`](#create) | mutation | `compute:sites:manage` @ `compute-site` | v0.2  | registered |
| [`revoke`](#revoke) | mutation | `compute:sites:manage` @ `compute-site` | v0.2  | registered |

## create [#create]

`POST /api/compute/siteTokens/create`

Mints a role-`site` join token for one site and returns its record with the bearer `token`, which is never shown again.

Only the site's owner tenant can mint tokens (the root tenant for platform sites); another tenant gets `NOT_FOUND`, and a deleted site is `INVALID_STATE`. `ttlSeconds` runs from 60 to 86,400, and `maxUses` from 1 to 16 (default 1).

The token id derives from the tenant, the calling identity and `idempotencyKey`. A retry with the same key returns the stored record without `token` and without a new IAM decision; the same key for a different site is `CONFLICT`. Publishes `security.site-token.created`.

* **Kind:** mutation
* **Authorization:** `compute:sites:manage` @ `compute-site`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `siteId`         | string  | yes      | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `ttlSeconds`     | integer | yes      | 60-86400                                                          |
| `maxUses`        | integer |          | 1-16; default `1`                                                 |
| `idempotencyKey` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$`    |

## revoke [#revoke]

`POST /api/compute/siteTokens/revoke`

Revokes a site token at once and returns its record with `revokedAt` set.

Revoking a token that is already revoked returns it unchanged and publishes nothing. A token that is not a site token of this tenant and site is `NOT_FOUND`. A first revocation publishes `security.site-token.revoked`.

* **Kind:** mutation
* **Authorization:** `compute:sites:manage` @ `compute-site`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `siteId`   | string | yes      | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `tokenId`  | string | yes      | at most 64 characters; matches `^jt_[0-9a-hjkmnp-tv-z]{26,52}$`   |


# sites (/docs/reference/api/sites)

> Sites are boot and failure domains: every pool and node belongs to one site. Platform sites (`shared`, `dedicated`) belong to the root tenant; a tenant's own hardware lives on its `custom` sites. A tenant sees the sites the authorization projection holds for it. See [Sites](/docs/guides/bare-metal/sites).



Sites are boot and failure domains: every pool and node belongs to one site. Platform sites (`shared`, `dedicated`) belong to the root tenant; a tenant's own hardware lives on its `custom` sites. A tenant sees the sites the authorization projection holds for it. See [Sites](/docs/guides/bare-metal/sites).

All 5 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                                                                         | Phase | Status     |
| ------------------- | -------- | ------------------------------------------------------------------------------------- | ----- | ---------- |
| [`list`](#list)     | read     | `compute:sites:read` @ `compute-tenant` + items                                       | v0.1  | registered |
| [`get`](#get)       | read     | `compute:sites:read` @ `compute-site`                                                 | v0.1  | registered |
| [`create`](#create) | mutation | `compute:sites:manage` @ `compute-fleet` or `compute:sites:manage` @ `compute-tenant` | v0.1  | registered |
| [`update`](#update) | mutation | `compute:sites:manage` @ `compute-site`                                               | v0.2  | registered |
| [`delete`](#delete) | mutation | `compute:sites:manage` @ `compute-site`                                               | v0.2  | registered |

## list [#list]

`POST /api/compute/sites/list`

Lists the sites visible to the tenant in id order, filtered by `kind` or `region`.

The call is decided once on `compute-tenant/{tenantId}`, then each site is checked for `compute:sites:read`; sites the caller may not read are left out, so a page can hold fewer than `limit` items. Pages hold up to 50 sites.

* **Kind:** read
* **Authorization:** `compute:sites:read` @ `compute-tenant` + items
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type                                   | Required | Constraints                                                |
| ---------- | -------------------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId` | string                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `kind`     | one of `shared`, `dedicated`, `custom` |          |                                                            |
| `region`   | string                                 |          | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`    |
| `limit`    | integer                                |          | 1-50; default `50`                                         |
| `cursor`   | string                                 |          | at most 256 characters; matches `^[A-Za-z0-9._:-]{1,256}$` |

## get [#get]

`POST /api/compute/sites/get`

Returns one site: its kind, region, IAM region and version.

A site the tenant cannot see is `NOT_FOUND`.

* **Kind:** read
* **Authorization:** `compute:sites:read` @ `compute-site`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field      | Type   | Required | Constraints                                                       |
| ---------- | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId` | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `siteId`   | string | yes      | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$` |

## create [#create]

`POST /api/compute/sites/create`

Creates a site and adds it to the authorization projection with its region and kind as IAM attributes.

A `custom` site is decided on `compute-tenant/{tenantId}`; a `shared` or `dedicated` site on `compute-fleet/global`, and its `tenantId` must be the root tenant, otherwise the call is `INVALID_INPUT`. `iamRegion` names the IAM region when it differs from `region`.

The site id derives from the tenant, the calling identity and `idempotencyKey`. A retry with the same key returns the site without a new IAM decision; the same key with a different kind, region or IAM region is `CONFLICT`. Publishes `site.created`.

* **Kind:** mutation
* **Authorization:** `compute:sites:manage` @ `compute-fleet` or `compute:sites:manage` @ `compute-tenant`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

| Field            | Type                                   | Required | Constraints                                                    |
| ---------------- | -------------------------------------- | -------- | -------------------------------------------------------------- |
| `tenantId`       | string                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`     |
| `kind`           | one of `shared`, `dedicated`, `custom` | yes      |                                                                |
| `region`         | string                                 | yes      | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`        |
| `iamRegion`      | string                                 |          | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`        |
| `idempotencyKey` | string                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._~:/+=-]{1,128}$` |

## update [#update]

`POST /api/compute/sites/update`

Changes a site's `region` or `iamRegion`, or clears `iamRegion` so IAM uses `region`, and returns the site.

Only the owner tenant can update a site; a deleted site is `NOT_FOUND`. Naming no change, or both `iamRegion` and `clearIamRegion`, is `INVALID_INPUT`, and a stale `version` is `CONFLICT`. An update that changes nothing returns the site unchanged. Otherwise the new attributes reach every tenant that sees the site, and `site.updated` is published.

* **Kind:** mutation
* **Authorization:** `compute:sites:manage` @ `compute-site`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field            | Type    | Required | Constraints                                                       |
| ---------------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId`       | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `siteId`         | string  | yes      | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `region`         | string  |          | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`           |
| `iamRegion`      | string  |          | at most 63 characters; matches `^[a-z][a-z0-9-]{0,62}$`           |
| `clearIamRegion` | boolean |          | default `false`                                                   |
| `version`        | integer |          | at least 1                                                        |

## delete [#delete]

`POST /api/compute/sites/delete`

Deletes a site that has no unarchived pool and no undeleted node.

Only the owner tenant can delete a site. A site with live pools or nodes is `INVALID_STATE`, and a stale `version` is `CONFLICT`. The call revokes the site's join and site tokens, expires its pending and approved join requests, drops its boot lease and removes it from the projection. The row is removed when nothing references it and tombstoned otherwise (`tombstoned: true`).

A retry after the delete returns `deleted: false`. Publishes `site.deleted`.

* **Kind:** mutation
* **Authorization:** `compute:sites:manage` @ `compute-site`
* **Module:** `server`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `siteId`   | string  | yes      | at most 64 characters; matches `^site_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `version`  | integer |          | at least 1                                                        |


# system (/docs/reference/api/system)

> The `system` group answers questions about the deployment itself. `whoami` and `version` are public and answer without a credential, which makes them a quick connectivity and credential check; `selfCheck` is a fleet administrator's health report and `workQueue` the work queue's backlog. See [Observability](/docs/operations/observability).



The `system` group answers questions about the deployment itself. `whoami` and `version` are public and answer without a credential, which makes them a quick connectivity and credential check; `selfCheck` is a fleet administrator's health report and `workQueue` the work queue's backlog. See [Observability](/docs/operations/observability).

All 4 methods are served by @better-compute/server 0.3.0.

| Method                    | Kind   | Authorization                           | Phase | Status     |
| ------------------------- | ------ | --------------------------------------- | ----- | ---------- |
| [`whoami`](#whoami)       | public | public                                  | v0.1  | registered |
| [`version`](#version)     | public | public                                  | v0.1  | registered |
| [`selfCheck`](#selfcheck) | read   | `compute:fleet:admin` @ `compute-fleet` | v0.1  | registered |
| [`workQueue`](#workqueue) | read   | `compute:fleet:admin` @ `compute-fleet` | v1.0  | registered |

## whoami [#whoami]

`POST /api/compute/system/whoami`

Reports who is calling: `authenticated: false` without a credential, or the authenticated identity.

Public; no IAM decision. When the call carries a credential, IAM authenticates it once and the answer holds the identity id and kind, the tenant, the session kind, whether the session used MFA, and the agent and delegation ids when present. The session id is never returned.

* **Kind:** public
* **Authorization:** public
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

Takes an empty object: `{}`.

## version [#version]

`POST /api/compute/system/version`

Returns the server's name, release version, API base path and the agent protocol versions it speaks.

Public; no IAM decision and no input. The JoinService public keys appear only in the `.well-known/better-compute` document, not in this answer.

* **Kind:** public
* **Authorization:** public
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

Takes an empty object: `{}`.

## selfCheck [#selfcheck]

`POST /api/compute/system/selfCheck`

Runs the deployment's read-only health checks and returns the findings, with `ok` false when any finding is an `error`.

Decided as `compute:fleet:admin` on `compute-fleet/global`, which resolves only in the root tenant. The report covers the method catalog, applied migrations, the `btree_gist` extension, `compute_now()` against the database clock, the operations checks (overdue work, command sweeper lag, usage dead letters and backlog, gateway headroom, stale heartbeats, daemon protocol skew, expiring node certificates) and every module's own `selfCheck`. A failing check becomes an `error` finding rather than a failed call.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `server`
* **Phase:** v0.1 (registered)

**Input**

Takes an empty object: `{}`.

## workQueue [#workqueue]

`POST /api/compute/system/workQueue`

Returns the work queue's backlog per kind: how many rows are queued, due, overdue and held by a worker, how many are backing off after failures, and when the next one is due, with how many workers are alive.

Decided as `compute:fleet:admin` on `compute-fleet/global`, which resolves only in the root tenant. `kinds` (1 to 100) answers those kinds in that order, a kind with no rows as zeros; without it, every kind with rows, sorted. A row is due once its `not_before` passed and no worker holds it, and overdue when that was more than `overdueAfterMs` ago (default 5 minutes, as `selfCheck`'s `workers.queue` check counts it). `nextDueAt` (epoch ms) is the earliest `not_before` of the rows no worker holds, absent when there is none; `at` is the database clock (`compute_now()`) the counts were taken at. It only reads, in one statement.

* **Kind:** read
* **Authorization:** `compute:fleet:admin` @ `compute-fleet`
* **Module:** `server`
* **Phase:** v1.0 (registered)

**Input**

| Field            | Type           | Required | Constraints                   |
| ---------------- | -------------- | -------- | ----------------------------- |
| `kinds`          | list of string |          | 1-100 items                   |
| `overdueAfterMs` | integer        |          | 0-604800000; default `300000` |


# tunnels (/docs/reference/api/tunnels)

> Interactive access to an instance or a node through its gateway and daemon: exec, SSH, TCP or the serial console of an instance, or the console of a node. The server's tunnels module hosts this group when the deployment configures `tunnels` with an exec ticket issuer. See [Tunnels](/docs/guides/access/tunnels).



Interactive access to an instance or a node through its gateway and daemon: exec, SSH, TCP or the serial console of an instance, or the console of a node. The server's tunnels module hosts this group when the deployment configures `tunnels` with an exec ticket issuer. See [Tunnels](/docs/guides/access/tunnels).

All 1 methods are served by @better-compute/server 0.3.0.

| Method          | Kind     | Authorization                                                                             | Phase | Status     |
| --------------- | -------- | ----------------------------------------------------------------------------------------- | ----- | ---------- |
| [`open`](#open) | mutation | `compute:instances:exec` @ `compute-instance` or `compute:nodes:console` @ `compute-node` | v0.3  | registered |

## open [#open]

`POST /api/compute/tunnels/open`

Issues a single-use exec ticket, valid for 60 seconds, for a tunnel to an instance or a node; returns the ticket, the tunnel id, the node and the `path` (`tunnels/<ticket>`, relative to the API base path) the caller then opens.

* Give exactly one of `instanceId` or `nodeId`. An instance takes `exec`, `ssh`, `tcp` (with a `port` from 1 to 65535) or `console` (its serial console: the daemon opens the domain's console from its runtime, never a host shell), decided with `compute:instances:exec` on `compute-instance/{instanceId}`. A node takes `console` only, decided with `compute:nodes:console` on `compute-node/{nodeId}`. Any other combination is `INVALID_INPUT`.

* Instance tunnels need a daemon that advertises `tunnel.instance-dial`: an older one would reach the host rather than the instance, so the call is `UNSUPPORTED` with `details.reason: 'daemon-capability'`; upgrade the daemon.

* A sandbox's instance, or a warm one waiting for a sandbox, takes no tunnels: `INVALID_STATE` with `details.reason: 'sandbox-managed'`; `sandboxes.exec` is its only way in.

* `NOT_FOUND` when the instance is not the tenant's or is still a draft, or the node is outside the tenant's view. `UNSUPPORTED` for a provider instance, which has no node agent to tunnel through.

* `INVALID_STATE` when the instance is not `running` or its node is not connected to a gateway.

* `RATE_LIMITED` (with `retryAfterMs`) beyond the per-identity limit of each API replica, 10 tunnels per minute by default.

* Emits `tunnel.issued`.

* **Kind:** mutation

* **Authorization:** `compute:instances:exec` @ `compute-instance` or `compute:nodes:console` @ `compute-node`

* **Module:** `tunnels`

* **Phase:** v0.3 (registered)

**Input**

| Field        | Type                                   | Required | Constraints                                                |
| ------------ | -------------------------------------- | -------- | ---------------------------------------------------------- |
| `tenantId`   | string                                 | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `instanceId` | string                                 |          | at most 57 characters; matches `^inst_[0-9a-z]{26,52}$`    |
| `nodeId`     | string                                 |          | at most 57 characters; matches `^node_[0-9a-z]{26,52}$`    |
| `op`         | one of `exec`, `console`, `ssh`, `tcp` | yes      |                                                            |
| `port`       | integer                                |          | 1-65535                                                    |


# usage (/docs/reference/api/usage)

> The hourly usage windows compute meters and hands to Better IAM billing, as the tenant can read them: one per hour and meter of every source (allocations of hosted capacity, provider instances, reservation grants, networks, snapshots, and pass-through provider costs). `list` and `export` decide `compute:usage:read` on `compute-tenant/{tenantId}`, or on `compute-project/{projectId}` when the call names a project, since windows carry their project as a tag. `platform`, `deadLetters`, `retry` and `void` are the root tenant's: the cross-tenant view and the corrections to what the usage sink sent. They decide `compute:usage:read` (`compute:usage:manage` for `retry` and `void`) on `compute-fleet/global`, so only root can call them. A voided window is not usage: `list` and `export` leave it out of rows and totals, and `platform` counts it apart. See [Billing and usage](/docs/guides/iam/billing-and-usage).



The hourly usage windows compute meters and hands to Better IAM billing, as the tenant can read them: one per hour and meter of every source (allocations of hosted capacity, provider instances, reservation grants, networks, snapshots, and pass-through provider costs). `list` and `export` decide `compute:usage:read` on `compute-tenant/{tenantId}`, or on `compute-project/{projectId}` when the call names a project, since windows carry their project as a tag. `platform`, `deadLetters`, `retry` and `void` are the root tenant's: the cross-tenant view and the corrections to what the usage sink sent. They decide `compute:usage:read` (`compute:usage:manage` for `retry` and `void`) on `compute-fleet/global`, so only root can call them. A voided window is not usage: `list` and `export` leave it out of rows and totals, and `platform` counts it apart. See [Billing and usage](/docs/guides/iam/billing-and-usage).

All 6 methods are served by @better-compute/server 0.3.0.

| Method                        | Kind     | Authorization                                                                       | Phase | Status     |
| ----------------------------- | -------- | ----------------------------------------------------------------------------------- | ----- | ---------- |
| [`list`](#list)               | read     | `compute:usage:read` @ `compute-tenant` or `compute:usage:read` @ `compute-project` | v0.1  | registered |
| [`export`](#export)           | read     | `compute:usage:read` @ `compute-tenant` or `compute:usage:read` @ `compute-project` | v0.2  | registered |
| [`platform`](#platform)       | read     | `compute:usage:read` @ `compute-fleet`                                              | v1.0  | registered |
| [`deadLetters`](#deadletters) | read     | `compute:usage:read` @ `compute-fleet`                                              | v1.0  | registered |
| [`retry`](#retry)             | mutation | `compute:usage:manage` @ `compute-fleet`                                            | v1.0  | registered |
| [`void`](#void)               | mutation | `compute:usage:manage` @ `compute-fleet`                                            | v1.0  | registered |

## list [#list]

`POST /api/compute/usage/list`

Lists the tenant's usage windows ordered by window start, optionally for one project, one meter and a time range; returns a page with a cursor.

* Each window carries its billing idempotency key (`cmp:<allocationId>:<windowStartMs>` for an allocation, which it also names as `allocationId`; `cmp:<kind>:<sourceId>:<windowStartMs>:<meter>` for an instance, grant, network or snapshot; `costx:<accountId>:<sourceRowId>` for a pass-through provider cost row), meter, quantity, bounds and tags, and, once the usage sink has handed it to IAM billing, `recordedAt` and the `outcome` (`recorded`, `duplicate`, `reattributed` or `dead-lettered`).

* `from` and `to` (epoch milliseconds) select windows that end after `from` and start before `to`; `to` not after `from` is `INVALID_INPUT`.

* `limit` is at most 500 (100 by default). A cursor this method did not return is `INVALID_INPUT`.

* Windows of every source appear, including the provider instance, grant, network and snapshot windows of the tenant's cloud VMs. A pass-through window's key contains the platform's provider account id (`costx:<accountId>:…`): tenants see that id, never the account itself. Voided windows are left out; `includeVoided: true` lists them with `voided: true` and `voidedAt`. The reason root gave when voiding is root's and is never shown to tenants.

* **Kind:** read

* **Authorization:** `compute:usage:read` @ `compute-tenant` or `compute:usage:read` @ `compute-project`

* **Module:** `server`

* **Phase:** v0.1 (registered)

**Input**

| Field           | Type    | Required | Constraints                                                      |
| --------------- | ------- | -------- | ---------------------------------------------------------------- |
| `tenantId`      | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `projectId`     | string  |          | at most 64 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `meter`         | string  |          | at most 128 characters; matches `^[a-z][a-z0-9_.-]{0,127}$`      |
| `from`          | integer |          | at least 0                                                       |
| `to`            | integer |          | at least 0                                                       |
| `limit`         | integer |          | 1-500; default `100`                                             |
| `cursor`        | string  |          | at most 256 characters; matches `^[A-Za-z0-9_-]{1,256}$`         |
| `includeVoided` | boolean |          |                                                                  |

## export [#export]

`POST /api/compute/usage/export`

Exports every usage window of a time range as CSV (the default) or JSON Lines, with the window count and the summed quantity per meter.

* `from` and `to` are required, and the range is at most 93 days; `to` not after `from`, or a longer range, is `INVALID_INPUT`.

* One export returns at most 50,000 windows; when more match, `truncated` is `true` and the range or meter should be narrowed.

* CSV has a header row and ISO 8601 timestamps; JSON Lines has one window object per line.

* Voided windows are in neither `data`, `count` nor `totals`; `voided` counts them.

* Cells a spreadsheet would read as a formula (starting with `=`, `+`, `-`, `@`, a tab or a carriage return) are prefixed with `'`; the same holds for the CSV of `platform`.

* **Kind:** read

* **Authorization:** `compute:usage:read` @ `compute-tenant` or `compute:usage:read` @ `compute-project`

* **Module:** `server`

* **Phase:** v0.2 (registered)

**Input**

| Field       | Type                  | Required | Constraints                                                      |
| ----------- | --------------------- | -------- | ---------------------------------------------------------------- |
| `tenantId`  | string                | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`       |
| `projectId` | string                |          | at most 64 characters; matches `^prj_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `meter`     | string                |          | at most 128 characters; matches `^[a-z][a-z0-9_.-]{0,127}$`      |
| `from`      | integer               | yes      | at least 0                                                       |
| `to`        | integer               | yes      | at least 0                                                       |
| `format`    | one of `csv`, `jsonl` |          |                                                                  |

## platform [#platform]

`POST /api/compute/usage/platform`

Totals the usage windows of every tenant over a time range, grouped by any of `tenant`, `pool`, `meter`, `source` (the window's source kind: allocation, instance, grant, network, snapshot or cost) and `provider`. Root only: the decision is `compute:usage:read` on `compute-fleet/global`, which resolves only in the root tenant, so a tenant administrator gets `ACCESS_DENIED`.

* `groupBy` is required and lists each key at most once; an empty list gives one row for everything. `tenantId` narrows the view to one tenant.

* Each row has `keys` (the values of the `groupBy` keys; `''` where a window has no `pool` or `provider` tag), the summed `quantity`, the number of `windows`, and `outcomes`, the windows per outcome (`pending` for a window not yet handed to IAM billing).

* `from` and `to` are required and select windows by their start, `from` inclusive and `to` exclusive, so adjacent ranges never count a window twice. The range is at most 93 days; `to` not after `from`, or a longer range, is `INVALID_INPUT`.

* Voided windows are in no row; `voided` counts them for the same range and tenant.

* One call returns at most 50,000 groups; when more match, `truncated` is `true` and the grouping or range should be narrowed.

* `format: 'csv'` answers the same groups as `data` (a header row, the keys, `quantity`, `windows`, and `outcomes` as JSON) with `rows` empty; `count` is the number of groups either way.

* **Kind:** read

* **Authorization:** `compute:usage:read` @ `compute-fleet`

* **Module:** `server`

* **Phase:** v1.0 (registered)

**Input**

| Field      | Type                                                           | Required | Constraints                                                |
| ---------- | -------------------------------------------------------------- | -------- | ---------------------------------------------------------- |
| `from`     | integer                                                        | yes      | at least 0                                                 |
| `to`       | integer                                                        | yes      | at least 0                                                 |
| `groupBy`  | list of one of `tenant`, `pool`, `meter`, `source`, `provider` | yes      | 0-5 items; no repeats                                      |
| `tenantId` | string                                                         |          | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$` |
| `format`   | one of `json`, `csv`                                           |          |                                                            |

## deadLetters [#deadletters]

`POST /api/compute/usage/deadLetters`

Root only. Lists the open dead letters of the usage sink (events Better IAM refused for good), ordered by idempotency key; returns a page with a cursor. Decides `compute:usage:read` on `compute-fleet/global`.

* Each item has `idempotencyKey`, `tenantId`, `meter`, `quantity`, `windowStart`, the better-iam `error` message and a `reason`: `meter-missing`, `period-closed`, `identity`, `invalid` or `other`. A letter disappears once a retry is recorded.

* A pass-through (`cost`) window's letter also has `cost`, its reported amount in currency units (the window's `cost_micros` / 10^6), so root can raise a Better IAM invoice item for a `period-closed` one. Rated usage has no `cost`.

* `reason` filters the list. `limit` is at most 500 (100 by default); a cursor this method did not return is `INVALID_INPUT`.

* **Kind:** read

* **Authorization:** `compute:usage:read` @ `compute-fleet`

* **Module:** `server`

* **Phase:** v1.0 (registered)

**Input**

| Field    | Type                                                                    | Required | Constraints                                              |
| -------- | ----------------------------------------------------------------------- | -------- | -------------------------------------------------------- |
| `reason` | one of `meter-missing`, `period-closed`, `identity`, `invalid`, `other` |          |                                                          |
| `limit`  | integer                                                                 |          | 1-500; default `100`                                     |
| `cursor` | string                                                                  |          | at most 256 characters; matches `^[A-Za-z0-9_-]{1,256}$` |

## retry [#retry]

`POST /api/compute/usage/retry`

Root only. Clears the outcome of dead-lettered windows so the next usage flush sends them again. Decides `compute:usage:manage` on `compute-fleet/global`.

* Give `idempotencyKeys` (1 to 1000 open dead letters) or a `reason`, not both. A key without an open dead letter is `NOT_FOUND` and nothing is retried.

* `period-closed` letters are refused with `INVALID_STATE`: retrying would only fail again, and the usage needs a root invoice item in Better IAM (`billing.createInvoiceItem`).

* A key whose window retention already pruned is `NOT_FOUND` ("window pruned; record it in Better IAM by hand").

* Answers `retried`, the windows put back to pending (their claim is released too). A voided window stays voided. The letter stays open until a flush records the window.

* **Kind:** mutation

* **Authorization:** `compute:usage:manage` @ `compute-fleet`

* **Module:** `server`

* **Phase:** v1.0 (registered)

**Input**

| Field             | Type                                                                    | Required | Constraints              |
| ----------------- | ----------------------------------------------------------------------- | -------- | ------------------------ |
| `idempotencyKeys` | list of string                                                          |          | 1-1000 items; no repeats |
| `reason`          | one of `meter-missing`, `period-closed`, `identity`, `invalid`, `other` |          |                          |

## void [#void]

`POST /api/compute/usage/void`

Root only. Voids usage windows no flush has sent, and dead-lettered ones, so no flush ever sends them: sets `voided_at` and `void_reason`. Decides `compute:usage:manage` on `compute-fleet/global`.

* `idempotencyKeys` (1 to 1000) and a `reason` are required.

* Voidable: a pending window, and a dead-lettered one (Better IAM refused it; nothing was billed). Voiding any window resolves its open dead letters.

* A window Better IAM recorded (outcome `recorded`, `duplicate` or `reattributed`) gives `INVALID_STATE` with the hint to issue a credit note in Better IAM. A window a usage flush claimed within the last 10 minutes is being sent and gives `INVALID_STATE` ("being sent; retry shortly"); a claim older than that belongs to a dead sender and does not block. An unknown key is `NOT_FOUND`. The call is all or nothing.

* A window voided before is left as it is and counted in `alreadyVoided`.

* Voided windows are left out of the tenants' `usage.list` and `usage.export`, and of `usage.platform` rows (which counts them apart).

* **Kind:** mutation

* **Authorization:** `compute:usage:manage` @ `compute-fleet`

* **Module:** `server`

* **Phase:** v1.0 (registered)

**Input**

| Field             | Type           | Required | Constraints                                           |
| ----------------- | -------------- | -------- | ----------------------------------------------------- |
| `idempotencyKeys` | list of string | yes      | 1-1000 items; no repeats                              |
| `reason`          | string         | yes      | 1-256 characters; matches `^[^\u0000-\u001f\u007f]+$` |


# webhooks (/docs/reference/api/webhooks)

> Per-tenant subscriptions that POST every matching committed event to an HTTPS endpoint, signed with a secret only the tenant and Better Compute know, with at-least-once delivery, retries with backoff, dead letters and redelivery. The events module hosts this group, and every method decides `compute:webhooks:manage` on `compute-tenant/{tenantId}`. See [Webhooks](/docs/guides/events/webhooks).



Per-tenant subscriptions that POST every matching committed event to an HTTPS endpoint, signed with a secret only the tenant and Better Compute know, with at-least-once delivery, retries with backoff, dead letters and redelivery. The events module hosts this group, and every method decides `compute:webhooks:manage` on `compute-tenant/{tenantId}`. See [Webhooks](/docs/guides/events/webhooks).

All 10 methods are served by @better-compute/server 0.3.0.

| Method                          | Kind     | Authorization                                | Phase | Status     |
| ------------------------------- | -------- | -------------------------------------------- | ----- | ---------- |
| [`create`](#create)             | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`list`](#list)                 | read     | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`get`](#get)                   | read     | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`update`](#update)             | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`rotateSecret`](#rotatesecret) | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`delete`](#delete)             | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`ping`](#ping)                 | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`deliveries`](#deliveries)     | read     | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`redeliver`](#redeliver)       | mutation | `compute:webhooks:manage` @ `compute-tenant` | v0.3  | registered |
| [`eventTypes`](#eventtypes)     | read     | `compute:webhooks:manage`                    | —     | registered |

## create [#create]

`POST /api/compute/webhooks/create`

Creates an active webhook for a URL and 1 to 20 event type patterns; returns the webhook and its signing secret, which is shown this once.

* A pattern is an exact event type, a family such as `reservation.*`, or `*`; a pattern that matches nothing in the event catalog is `INVALID_INPUT` (see `eventTypes`), unless the deployment is configured to accept any pattern.

* The URL must be absolute, use `https`, carry no credentials and not point at a private, loopback or other non-public address; otherwise `INVALID_INPUT`.

* A tenant has at most 20 webhooks by default; one more is `QUOTA_EXCEEDED`.

* The webhook receives events committed after it was created; earlier history is not replayed. The method takes no idempotency key, so a retry creates a second webhook.

* **Kind:** mutation

* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`

* **Module:** `events`

* **Phase:** v0.3 (registered)

**Input**

| Field         | Type           | Required | Constraints                        |
| ------------- | -------------- | -------- | ---------------------------------- |
| `tenantId`    | string         | yes      | matches `^[A-Za-z0-9._:-]{1,128}$` |
| `url`         | string         | yes      |                                    |
| `events`      | list of string | yes      | 1-20 items                         |
| `description` | string         |          | at most 256 characters             |

## list [#list]

`POST /api/compute/webhooks/list`

Lists the tenant's webhooks, oldest first, without their secrets.

* **Kind:** read
* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`
* **Module:** `events`
* **Phase:** v0.3 (registered)

**Input**

| Field      | Type   | Required | Constraints                        |
| ---------- | ------ | -------- | ---------------------------------- |
| `tenantId` | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$` |

## get [#get]

`POST /api/compute/webhooks/get`

Returns one webhook of the tenant by id, without its secret.

`NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** read
* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`
* **Module:** `events`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type   | Required | Constraints                               |
| ----------- | ------ | -------- | ----------------------------------------- |
| `tenantId`  | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId` | string | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |

## update [#update]

`POST /api/compute/webhooks/update`

Changes a webhook's URL, event patterns, description or `active` flag and increments its version; returns the webhook.

* URL and patterns are checked as in `create` (`INVALID_INPUT`).

* Setting `active` to `false` pauses the webhook and cancels its pending deliveries.

* `NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** mutation

* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`

* **Module:** `events`

* **Phase:** v0.3 (registered)

**Input**

| Field         | Type           | Required | Constraints                               |
| ------------- | -------------- | -------- | ----------------------------------------- |
| `tenantId`    | string         | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId`   | string         | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `url`         | string         |          |                                           |
| `events`      | list of string |          | 1-20 items                                |
| `description` | string         |          | at most 256 characters                    |
| `active`      | boolean        |          |                                           |

## rotateSecret [#rotatesecret]

`POST /api/compute/webhooks/rotateSecret`

Replaces a webhook's signing secret with a new one; returns the webhook and the new secret, shown this once.

* With `keepPreviousSeconds` (0 to 86,400, default 0), the old secret keeps signing deliveries alongside the new one until then, so receivers can switch without rejecting any; with 0 the old secret stops at once.

* `NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** mutation

* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`

* **Module:** `events`

* **Phase:** v0.3 (registered)

**Input**

| Field                 | Type    | Required | Constraints                               |
| --------------------- | ------- | -------- | ----------------------------------------- |
| `tenantId`            | string  | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId`           | string  | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `keepPreviousSeconds` | integer |          | 0-86400                                   |

## delete [#delete]

`POST /api/compute/webhooks/delete`

Deletes a webhook together with its delivery queue and history; returns `deleted: true`.

`NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** mutation
* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`
* **Module:** `events`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type   | Required | Constraints                               |
| ----------- | ------ | -------- | ----------------------------------------- |
| `tenantId`  | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId` | string | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |

## ping [#ping]

`POST /api/compute/webhooks/ping`

Queues a `webhook.ping` delivery to the webhook, due now; returns the delivery.

The ping is a delivery only: it is not written to the outbox, so `events.list` does not show it. A ping to a paused webhook is cancelled when the delivery worker picks it up. `NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** mutation
* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`
* **Module:** `events`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type   | Required | Constraints                               |
| ----------- | ------ | -------- | ----------------------------------------- |
| `tenantId`  | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId` | string | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |

## deliveries [#deliveries]

`POST /api/compute/webhooks/deliveries`

Lists a webhook's deliveries newest first, optionally filtered by `state` (`pending`, `delivered`, `dead` or `cancelled`), with attempts, the next attempt time and the last status or error.

`limit` is at most 200 (50 by default). `NOT_FOUND` when the webhook is not the tenant's.

* **Kind:** read
* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`
* **Module:** `events`
* **Phase:** v0.3 (registered)

**Input**

| Field       | Type                                               | Required | Constraints                               |
| ----------- | -------------------------------------------------- | -------- | ----------------------------------------- |
| `tenantId`  | string                                             | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `webhookId` | string                                             | yes      | matches `^whk_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `state`     | one of `pending`, `delivered`, `dead`, `cancelled` |          |                                           |
| `limit`     | integer                                            |          | 1-200                                     |

## redeliver [#redeliver]

`POST /api/compute/webhooks/redeliver`

Puts a delivery back in the queue, due now, with its attempt count reset; returns the delivery.

* Meant for `dead`, `cancelled` or `delivered` deliveries, for example once a failing endpoint is fixed.

* The delivery keeps its id, so the `X-Better-Compute-Delivery` header carries the same value as before.

* `NOT_FOUND` when the delivery is not the tenant's or its webhook is paused.

* **Kind:** mutation

* **Authorization:** `compute:webhooks:manage` @ `compute-tenant`

* **Module:** `events`

* **Phase:** v0.3 (registered)

**Input**

| Field        | Type   | Required | Constraints                               |
| ------------ | ------ | -------- | ----------------------------------------- |
| `tenantId`   | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `deliveryId` | string | yes      | matches `^dlv_[0-9a-hjkmnp-tv-z]{26,52}$` |

## eventTypes [#eventtypes]

`POST /api/compute/webhooks/eventTypes`

Returns the event catalog that webhook patterns are checked against: the built-in event families plus any plugin families the deployment configured, each with its type, subject, publishing module and description.

Decided like the rest of the group, with `compute:webhooks:manage` on `compute-tenant/{tenantId}`. The method comes from the events module and is not in core's API catalog, so it has no phase.

* **Kind:** read
* **Authorization:** `compute:webhooks:manage`
* **Module:** `events`
* **Phase:** not in PLAN.md §9.2 (registered)

**Input**

| Field      | Type   | Required | Constraints                        |
| ---------- | ------ | -------- | ---------------------------------- |
| `tenantId` | string | yes      | matches `^[A-Za-z0-9._:-]{1,128}$` |


# workflows (/docs/reference/api/workflows)

> The workflows group reads the provisioning workflows of a node and lets an operator abort, retry, pause or resume one. Workflows are started by other modules, such as metal for each workflow-driven provision state, not through this group. Every method is decided on the node (`compute-node`): reads as `compute:workflows:read`, the others as `compute:workflows:manage`. See [Provisioning workflows](/docs/guides/bare-metal/workflows).



The workflows group reads the provisioning workflows of a node and lets an operator abort, retry, pause or resume one. Workflows are started by other modules, such as metal for each workflow-driven provision state, not through this group. Every method is decided on the node (`compute-node`): reads as `compute:workflows:read`, the others as `compute:workflows:manage`. See [Provisioning workflows](/docs/guides/bare-metal/workflows).

All 6 methods are served by @better-compute/server 0.3.0.

| Method              | Kind     | Authorization                               | Phase | Status     |
| ------------------- | -------- | ------------------------------------------- | ----- | ---------- |
| [`get`](#get)       | read     | `compute:workflows:read` @ `compute-node`   | v0.2  | registered |
| [`list`](#list)     | read     | `compute:workflows:read` @ `compute-node`   | v0.2  | registered |
| [`abort`](#abort)   | mutation | `compute:workflows:manage` @ `compute-node` | v0.2  | registered |
| [`retry`](#retry)   | mutation | `compute:workflows:manage` @ `compute-node` | v0.2  | registered |
| [`pause`](#pause)   | mutation | `compute:workflows:manage` @ `compute-node` | v0.2  | registered |
| [`resume`](#resume) | mutation | `compute:workflows:manage` @ `compute-node` | v0.2  | registered |

## get [#get]

`POST /api/compute/workflows/get`

Returns one workflow of the node with all its steps, including each step's phase, state, attempt, command, deadline, result and error.

`NOT_FOUND` unless the workflow belongs to the tenant and the node.

* **Kind:** read
* **Authorization:** `compute:workflows:read` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`     | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `workflowId` | string | yes      | at most 55 characters; matches `^wf_[0-9a-hjkmnp-tv-z]{26,52}$`   |

## list [#list]

`POST /api/compute/workflows/list`

Returns a page of the node's workflows with their steps, newest first.

`limit` defaults to 20 and is at most 100; pass the returned `cursor` to read the next page.

* **Kind:** read
* **Authorization:** `compute:workflows:read` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field      | Type    | Required | Constraints                                                       |
| ---------- | ------- | -------- | ----------------------------------------------------------------- |
| `tenantId` | string  | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`   | string  | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `limit`    | integer |          | 1-100; default `20`                                               |
| `cursor`   | string  |          | at most 256 characters                                            |

## abort [#abort]

`POST /api/compute/workflows/abort`

Cancels a workflow: pending steps are cancelled, the in-flight step's command is cancelled, and the cleanup phase runs before the workflow ends `cancelled`.

* A `pending` or `failed` workflow is cancelled at once, with no cleanup, and publishes `workflow.cancelled`.
* Aborting a workflow that is already cancelled or cancelling does nothing; aborting a succeeded one is `INVALID_STATE`.
* The optional `reason` (at most 512 characters) is recorded in the workflow's error.

Aborting a workflow that metal owns fails the node; steer metal nodes with `metal.abort` and `metal.recover` instead.

* **Kind:** mutation
* **Authorization:** `compute:workflows:manage` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`     | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `workflowId` | string | yes      | at most 55 characters; matches `^wf_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `reason`     | string |          | at most 512 characters                                            |

## retry [#retry]

`POST /api/compute/workflows/retry`

Runs a failed workflow again from `fromStep`, by default the first main step that did not succeed; that step and every later main step return to `pending` with the next attempt number, and on-failure and cleanup steps are re-armed.

* `INVALID_STATE` unless the workflow is `failed`.
* `INVALID_INPUT` when `fromStep` is not a main step, or when every main step succeeded.
* `CONFLICT` when the node has another active workflow.

Publishes `workflow.retried` with the `fromStep`.

* **Kind:** mutation
* **Authorization:** `compute:workflows:manage` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`     | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `workflowId` | string | yes      | at most 55 characters; matches `^wf_[0-9a-hjkmnp-tv-z]{26,52}$`   |
| `fromStep`   | string |          | at most 256 characters                                            |

## pause [#pause]

`POST /api/compute/workflows/pause`

Pauses a running workflow: the step in flight finishes, but no further step starts until `resume`.

`INVALID_STATE` when the workflow is not `running`, or when it is in its cleanup phase after a failure or abort. Publishes `workflow.paused`.

* **Kind:** mutation
* **Authorization:** `compute:workflows:manage` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`     | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `workflowId` | string | yes      | at most 55 characters; matches `^wf_[0-9a-hjkmnp-tv-z]{26,52}$`   |

## resume [#resume]

`POST /api/compute/workflows/resume`

Puts a paused workflow back to `running` and queues it to continue with its next step.

`INVALID_STATE` when the workflow is not `paused`. Publishes `workflow.resumed`.

* **Kind:** mutation
* **Authorization:** `compute:workflows:manage` @ `compute-node`
* **Module:** `provision`
* **Phase:** v0.2 (registered)

**Input**

| Field        | Type   | Required | Constraints                                                       |
| ------------ | ------ | -------- | ----------------------------------------------------------------- |
| `tenantId`   | string | yes      | at most 128 characters; matches `^[A-Za-z0-9._:-]{1,128}$`        |
| `nodeId`     | string | yes      | at most 57 characters; matches `^node_[0-9a-hjkmnp-tv-z]{26,52}$` |
| `workflowId` | string | yes      | at most 55 characters; matches `^wf_[0-9a-hjkmnp-tv-z]{26,52}$`   |